Jul 28 04:11:39.516103 ip-10-0-129-187 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 04:11:39.825960 ip-10-0-129-187 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 04:11:39.825960 ip-10-0-129-187 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 04:11:39.825960 ip-10-0-129-187 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 04:11:39.825960 ip-10-0-129-187 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 04:11:39.825960 ip-10-0-129-187 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 04:11:39.826780 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.826659    2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 04:11:39.831976 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.831953    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 04:11:39.831976 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.831973    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 04:11:39.831976 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.831980    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 04:11:39.831976 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.831985    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 04:11:39.832242 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.831990    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 04:11:39.832242 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.831995    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 04:11:39.832242 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.831998    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 04:11:39.832242 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832002    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 04:11:39.832242 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832007    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 04:11:39.832242 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832011    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 04:11:39.832242 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832015    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 04:11:39.832242 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832018    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 04:11:39.832242 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832022    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 04:11:39.832242 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832026    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 04:11:39.832242 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832030    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 04:11:39.832242 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832034    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 04:11:39.832242 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832039    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 04:11:39.832242 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832044    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 04:11:39.832242 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832048    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 04:11:39.832242 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832052    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 04:11:39.832242 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832056    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 04:11:39.832242 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832060    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 04:11:39.832242 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832064    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 04:11:39.832242 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832068    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 04:11:39.833156 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832073    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 04:11:39.833156 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832077    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 04:11:39.833156 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832081    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 04:11:39.833156 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832085    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 04:11:39.833156 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832089    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 04:11:39.833156 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832093    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 04:11:39.833156 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832098    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 04:11:39.833156 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832103    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 04:11:39.833156 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832107    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 04:11:39.833156 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832111    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 04:11:39.833156 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832115    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 04:11:39.833156 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832119    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 04:11:39.833156 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832124    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 04:11:39.833156 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832131    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 04:11:39.833156 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832136    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 04:11:39.833156 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832141    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 04:11:39.833156 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832146    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 04:11:39.833156 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832150    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 04:11:39.833156 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832154    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 04:11:39.833156 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832159    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 04:11:39.834038 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832163    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 04:11:39.834038 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832167    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 04:11:39.834038 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832172    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 04:11:39.834038 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832176    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 04:11:39.834038 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832184    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 04:11:39.834038 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832189    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 04:11:39.834038 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832196    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 04:11:39.834038 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832201    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 04:11:39.834038 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832206    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 04:11:39.834038 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832210    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 04:11:39.834038 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832215    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 04:11:39.834038 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832220    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 04:11:39.834038 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832224    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 04:11:39.834038 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832229    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 04:11:39.834038 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832234    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 04:11:39.834038 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832237    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 04:11:39.834038 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832242    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 04:11:39.834038 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832246    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 04:11:39.834038 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832252    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 04:11:39.834038 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832256    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 04:11:39.834933 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832261    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 04:11:39.834933 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832265    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 04:11:39.834933 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832270    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 04:11:39.834933 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832274    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 04:11:39.834933 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832279    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 04:11:39.834933 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832283    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 04:11:39.834933 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832288    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 04:11:39.834933 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832292    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 04:11:39.834933 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832297    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 04:11:39.834933 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832301    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 04:11:39.834933 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832306    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 04:11:39.834933 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832310    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 04:11:39.834933 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832314    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 04:11:39.834933 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832319    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 04:11:39.834933 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832323    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 04:11:39.834933 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832328    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 04:11:39.834933 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832332    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 04:11:39.834933 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832336    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 04:11:39.834933 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832341    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 04:11:39.834933 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832346    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 04:11:39.835807 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832350    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 04:11:39.835807 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832355    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 04:11:39.835807 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832359    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 04:11:39.835807 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832363    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 04:11:39.835807 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832370    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 04:11:39.835807 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832377    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 04:11:39.835807 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832381    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 04:11:39.835807 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832388    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 04:11:39.835807 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832392    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 04:11:39.835807 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832397    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 04:11:39.835807 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832402    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 04:11:39.835807 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832409    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 04:11:39.835807 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832415    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 04:11:39.835807 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832421    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 04:11:39.835807 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.832426    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 04:11:39.835807 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833625    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 04:11:39.835807 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833636    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 04:11:39.835807 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833643    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 04:11:39.836604 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833648    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 04:11:39.836604 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833654    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 04:11:39.836604 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833658    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 04:11:39.836604 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833664    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 04:11:39.836604 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833669    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 04:11:39.836604 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833675    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 04:11:39.836604 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833679    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 04:11:39.836604 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833685    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 04:11:39.836604 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833689    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 04:11:39.836604 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833694    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 04:11:39.836604 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833698    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 04:11:39.836604 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833703    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 04:11:39.836604 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833707    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 04:11:39.836604 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833711    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 04:11:39.836604 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833715    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 04:11:39.836604 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833720    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 04:11:39.836604 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833724    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 04:11:39.836604 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833729    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 04:11:39.836604 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833733    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 04:11:39.836604 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833737    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 04:11:39.837309 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833743    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 04:11:39.837309 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833748    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 04:11:39.837309 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833752    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 04:11:39.837309 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833757    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 04:11:39.837309 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833762    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 04:11:39.837309 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833766    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 04:11:39.837309 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833771    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 04:11:39.837309 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833776    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 04:11:39.837309 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833781    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 04:11:39.837309 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833786    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 04:11:39.837309 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833791    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 04:11:39.837309 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833795    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 04:11:39.837309 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833800    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 04:11:39.837309 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833804    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 04:11:39.837309 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833808    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 04:11:39.837309 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833812    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 04:11:39.837309 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833817    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 04:11:39.837309 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833822    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 04:11:39.837309 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833826    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 04:11:39.837309 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833830    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 04:11:39.837920 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833854    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 04:11:39.837920 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833859    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 04:11:39.837920 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833863    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 04:11:39.837920 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833869    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 04:11:39.837920 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833873    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 04:11:39.837920 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833878    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 04:11:39.837920 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833882    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 04:11:39.837920 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833887    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 04:11:39.837920 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833892    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 04:11:39.837920 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833897    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 04:11:39.837920 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833901    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 04:11:39.837920 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833905    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 04:11:39.837920 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833910    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 04:11:39.837920 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833914    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 04:11:39.837920 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833919    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 04:11:39.837920 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833923    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 04:11:39.837920 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833927    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 04:11:39.837920 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833932    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 04:11:39.837920 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833936    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 04:11:39.837920 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833941    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 04:11:39.838657 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833946    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 04:11:39.838657 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833952    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 04:11:39.838657 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833956    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 04:11:39.838657 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833961    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 04:11:39.838657 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833966    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 04:11:39.838657 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833970    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 04:11:39.838657 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833975    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 04:11:39.838657 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833980    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 04:11:39.838657 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833984    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 04:11:39.838657 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833989    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 04:11:39.838657 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833993    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 04:11:39.838657 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.833997    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 04:11:39.838657 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834002    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 04:11:39.838657 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834008    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 04:11:39.838657 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834012    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 04:11:39.838657 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834017    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 04:11:39.838657 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834021    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 04:11:39.838657 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834026    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 04:11:39.838657 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834036    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 04:11:39.838657 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834042    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 04:11:39.839272 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834047    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 04:11:39.839272 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834052    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 04:11:39.839272 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834057    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 04:11:39.839272 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834063    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 04:11:39.839272 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834070    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 04:11:39.839272 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834078    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 04:11:39.839272 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834083    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 04:11:39.839272 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834088    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 04:11:39.839272 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834093    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 04:11:39.839272 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834098    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 04:11:39.839272 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834103    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 04:11:39.839272 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834108    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 04:11:39.839272 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834113    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 04:11:39.839272 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834118    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 04:11:39.839272 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834123    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 04:11:39.839272 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834128    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 04:11:39.839272 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834734    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 04:11:39.839272 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834743    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 04:11:39.839272 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834748    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 04:11:39.839761 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834753    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 04:11:39.839761 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834757    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 04:11:39.839761 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834762    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 04:11:39.839761 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834766    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 04:11:39.839761 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834770    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 04:11:39.839761 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834774    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 04:11:39.839761 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834779    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 04:11:39.839761 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834783    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 04:11:39.839761 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834789    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 04:11:39.839761 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834795    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 04:11:39.839761 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834801    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 04:11:39.839761 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834806    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 04:11:39.839761 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834810    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 04:11:39.839761 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834815    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 04:11:39.839761 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834819    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 04:11:39.839761 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834823    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 04:11:39.839761 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834828    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 04:11:39.839761 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834832    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 04:11:39.839761 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834854    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 04:11:39.839761 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834863    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 04:11:39.840379 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834868    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 04:11:39.840379 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834872    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 04:11:39.840379 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834877    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 04:11:39.840379 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834881    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 04:11:39.840379 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834885    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 04:11:39.840379 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834889    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 04:11:39.840379 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834894    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 04:11:39.840379 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834899    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 04:11:39.840379 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834904    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 04:11:39.840379 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834908    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 04:11:39.840379 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834914    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 04:11:39.840379 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834919    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 04:11:39.840379 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834923    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 04:11:39.840379 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834927    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 04:11:39.840379 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834931    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 04:11:39.840379 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834937    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 04:11:39.840379 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834942    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 04:11:39.840379 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834946    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 04:11:39.840379 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834951    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 04:11:39.840379 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834956    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 04:11:39.840918 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834960    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 04:11:39.840918 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834964    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 04:11:39.840918 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834968    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 04:11:39.840918 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834972    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 04:11:39.840918 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834977    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 04:11:39.840918 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834981    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 04:11:39.840918 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834986    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 04:11:39.840918 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834990    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 04:11:39.840918 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834994    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 04:11:39.840918 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.834999    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 04:11:39.840918 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.835003    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 04:11:39.840918 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.835009    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 04:11:39.840918 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.835013    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 04:11:39.840918 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.835018    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 04:11:39.840918 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.835022    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 04:11:39.840918 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.835027    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 04:11:39.840918 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.835031    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 04:11:39.840918 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.835036    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 04:11:39.840918 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.835040    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 04:11:39.840918 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.835044    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 04:11:39.840918 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.835049    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 04:11:39.841555 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.835054    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 04:11:39.841555 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.835058    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 04:11:39.841555 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.835062    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 04:11:39.841555 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.835066    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 04:11:39.841555 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.835071    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 04:11:39.841555 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.835075    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 04:11:39.841555 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.835080    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 04:11:39.841555 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.835084    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 04:11:39.841555 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.835088    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 04:11:39.841555 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.835092    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 04:11:39.841555 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.835097    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 04:11:39.841555 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.835101    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 04:11:39.841555 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.835106    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 04:11:39.841555 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.835112    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 04:11:39.841555 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.835116    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 04:11:39.841555 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.835121    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 04:11:39.841555 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.835128    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 04:11:39.841555 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.835133    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 04:11:39.841555 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.835137    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 04:11:39.842037 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.835141    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 04:11:39.842037 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.835146    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 04:11:39.842037 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.835150    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 04:11:39.842037 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.835156    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 04:11:39.842037 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.835161    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 04:11:39.842037 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.835165    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 04:11:39.842037 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.835170    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 04:11:39.842037 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.835174    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 04:11:39.842037 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.835178    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 04:11:39.842037 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.835182    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 04:11:39.842037 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.835187    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 04:11:39.842037 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.835191    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 04:11:39.842037 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.835196    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 04:11:39.842037 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.835202    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 04:11:39.842037 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.835207    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 04:11:39.842037 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.835213    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 04:11:39.842037 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835332    2566 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 04:11:39.842037 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835355    2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 04:11:39.842037 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835366    2566 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 04:11:39.842037 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835373    2566 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 04:11:39.842037 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835380    2566 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 04:11:39.842552 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835386    2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 04:11:39.842552 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835393    2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 04:11:39.842552 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835400    2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 04:11:39.842552 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835406    2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 04:11:39.842552 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835411    2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 04:11:39.842552 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835417    2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 04:11:39.842552 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835422    2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 04:11:39.842552 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835427    2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 04:11:39.842552 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835432    2566 flags.go:64] FLAG: --cgroup-root=""
Jul 28 04:11:39.842552 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835438    2566 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 04:11:39.842552 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835443    2566 flags.go:64] FLAG: --client-ca-file=""
Jul 28 04:11:39.842552 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835448    2566 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 04:11:39.842552 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835453    2566 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 04:11:39.842552 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835459    2566 flags.go:64] FLAG: --cluster-domain=""
Jul 28 04:11:39.842552 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835467    2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 04:11:39.842552 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835473    2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 04:11:39.842552 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835478    2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 04:11:39.842552 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835484    2566 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 04:11:39.842552 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835490    2566 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 04:11:39.842552 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835495    2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 04:11:39.842552 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835501    2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 04:11:39.842552 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835506    2566 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 04:11:39.842552 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835512    2566 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 04:11:39.842552 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835517    2566 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 04:11:39.843238 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835522    2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 04:11:39.843238 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835528    2566 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 04:11:39.843238 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835534    2566 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 04:11:39.843238 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835541    2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 04:11:39.843238 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835546    2566 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 04:11:39.843238 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835551    2566 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 04:11:39.843238 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835557    2566 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 04:11:39.843238 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835562    2566 flags.go:64] FLAG: --enable-server="true"
Jul 28 04:11:39.843238 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835568    2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 04:11:39.843238 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835580    2566 flags.go:64] FLAG: --event-burst="100"
Jul 28 04:11:39.843238 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835586    2566 flags.go:64] FLAG: --event-qps="50"
Jul 28 04:11:39.843238 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835591    2566 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 04:11:39.843238 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835596    2566 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 04:11:39.843238 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835602    2566 flags.go:64] FLAG: --eviction-hard=""
Jul 28 04:11:39.843238 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835609    2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 04:11:39.843238 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835614    2566 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 04:11:39.843238 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835620    2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 04:11:39.843238 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835625    2566 flags.go:64] FLAG: --eviction-soft=""
Jul 28 04:11:39.843238 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835630    2566 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 04:11:39.843238 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835635    2566 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 04:11:39.843238 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835640    2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 04:11:39.843238 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835645    2566 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 04:11:39.843238 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835652    2566 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 04:11:39.843238 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835657    2566 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 04:11:39.843238 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835662    2566 flags.go:64] FLAG: --feature-gates=""
Jul 28 04:11:39.843938 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835676    2566 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 04:11:39.843938 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835681    2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 04:11:39.843938 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835687    2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 04:11:39.843938 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835692    2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 04:11:39.843938 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835698    2566 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 04:11:39.843938 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835703    2566 flags.go:64] FLAG: --help="false"
Jul 28 04:11:39.843938 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835708    2566 flags.go:64] FLAG: --hostname-override="ip-10-0-129-187.ec2.internal"
Jul 28 04:11:39.843938 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835714    2566 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 04:11:39.843938 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835719    2566 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 04:11:39.843938 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835725    2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 04:11:39.843938 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835732    2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 04:11:39.843938 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835739    2566 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 04:11:39.843938 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835744    2566 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 04:11:39.843938 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835749    2566 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 04:11:39.843938 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835754    2566 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 04:11:39.843938 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835759    2566 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 04:11:39.843938 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835764    2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 04:11:39.843938 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835770    2566 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 04:11:39.843938 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835775    2566 flags.go:64] FLAG: --kube-reserved=""
Jul 28 04:11:39.843938 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835781    2566 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 04:11:39.843938 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835786    2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 04:11:39.843938 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835791    2566 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 04:11:39.843938 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835796    2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 04:11:39.843938 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835801    2566 flags.go:64] FLAG: --lock-file=""
Jul 28 04:11:39.844532 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835805    2566 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 04:11:39.844532 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835810    2566 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 04:11:39.844532 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835815    2566 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 04:11:39.844532 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835824    2566 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 04:11:39.844532 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835829    2566 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 04:11:39.844532 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835850    2566 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 04:11:39.844532 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835856    2566 flags.go:64] FLAG: --logging-format="text"
Jul 28 04:11:39.844532 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835861    2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 04:11:39.844532 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835867    2566 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 04:11:39.844532 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835872    2566 flags.go:64] FLAG: --manifest-url=""
Jul 28 04:11:39.844532 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835877    2566 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 04:11:39.844532 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835884    2566 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 04:11:39.844532 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835889    2566 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 04:11:39.844532 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835897    2566 flags.go:64] FLAG: --max-pods="110"
Jul 28 04:11:39.844532 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835901    2566 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 04:11:39.844532 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835907    2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 04:11:39.844532 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835912    2566 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 04:11:39.844532 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835917    2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 04:11:39.844532 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835923    2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 04:11:39.844532 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835929    2566 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 04:11:39.844532 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835934    2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 04:11:39.844532 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835949    2566 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 04:11:39.844532 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835955    2566 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 04:11:39.844532 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835960    2566 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 04:11:39.844532 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835966    2566 flags.go:64] FLAG: --pod-cidr=""
Jul 28 04:11:39.845583 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835971    2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 04:11:39.845583 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835980    2566 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 04:11:39.845583 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835985    2566 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 04:11:39.845583 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835991    2566 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 04:11:39.845583 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.835996    2566 flags.go:64] FLAG: --port="10250"
Jul 28 04:11:39.845583 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.836001    2566 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 04:11:39.845583 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.836007    2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0f94d253d51f8b9a4"
Jul 28 04:11:39.845583 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.836012    2566 flags.go:64] FLAG: --qos-reserved=""
Jul 28 04:11:39.845583 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.836017    2566 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 04:11:39.845583 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.836022    2566 flags.go:64] FLAG: --register-node="true"
Jul 28 04:11:39.845583 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.836027    2566 flags.go:64] FLAG: --register-with-taints=""
Jul 28 04:11:39.845583 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.836034    2566 flags.go:64] FLAG: --registry-burst="10"
Jul 28 04:11:39.845583 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.836039    2566 flags.go:64] FLAG: --registry-qps="5"
Jul 28 04:11:39.845583 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.836044    2566 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 04:11:39.845583 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.836049    2566 flags.go:64] FLAG: --reserved-memory=""
Jul 28 04:11:39.845583 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.836056    2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 04:11:39.845583 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.836061    2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 04:11:39.845583 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.836066    2566 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 04:11:39.845583 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.836071    2566 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 04:11:39.845583 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.836076    2566 flags.go:64] FLAG: --runonce="false"
Jul 28 04:11:39.845583 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.836082    2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 04:11:39.845583 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.836087    2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 04:11:39.845583 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.836092    2566 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 04:11:39.845583 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.836097    2566 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 04:11:39.845583 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.836102    2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 04:11:39.846377 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.836107    2566 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 04:11:39.846377 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.836113    2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 04:11:39.846377 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.836119    2566 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 04:11:39.846377 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.836126    2566 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 04:11:39.846377 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.836147    2566 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 04:11:39.846377 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.836154    2566 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 04:11:39.846377 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.836159    2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 04:11:39.846377 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.836165    2566 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 04:11:39.846377 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.836170    2566 flags.go:64] FLAG: --system-cgroups=""
Jul 28 04:11:39.846377 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.836175    2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 04:11:39.846377 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.836185    2566 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 04:11:39.846377 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.836190    2566 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 04:11:39.846377 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.836195    2566 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 04:11:39.846377 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.836202    2566 flags.go:64] FLAG: --tls-min-version=""
Jul 28 04:11:39.846377 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.836207    2566 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 04:11:39.846377 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.836212    2566 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 04:11:39.846377 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.836217    2566 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 04:11:39.846377 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.836222    2566 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 04:11:39.846377 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.836227    2566 flags.go:64] FLAG: --v="2"
Jul 28 04:11:39.846377 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.836234    2566 flags.go:64] FLAG: --version="false"
Jul 28 04:11:39.846377 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.836242    2566 flags.go:64] FLAG: --vmodule=""
Jul 28 04:11:39.846377 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.836248    2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 04:11:39.846377 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.836254    2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 04:11:39.846377 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836435    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 04:11:39.846377 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836443    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 04:11:39.847016 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836448    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 04:11:39.847016 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836453    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 04:11:39.847016 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836457    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 04:11:39.847016 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836463    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 04:11:39.847016 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836468    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 04:11:39.847016 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836472    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 04:11:39.847016 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836477    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 04:11:39.847016 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836481    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 04:11:39.847016 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836485    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 04:11:39.847016 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836491    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 04:11:39.847016 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836498    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 04:11:39.847016 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836502    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 04:11:39.847016 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836507    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 04:11:39.847016 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836511    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 04:11:39.847016 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836516    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 04:11:39.847016 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836521    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 04:11:39.847016 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836525    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 04:11:39.847016 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836529    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 04:11:39.847016 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836534    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 04:11:39.847016 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836538    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 04:11:39.847560 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836543    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 04:11:39.847560 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836548    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 04:11:39.847560 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836552    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 04:11:39.847560 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836556    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 04:11:39.847560 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836561    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 04:11:39.847560 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836565    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 04:11:39.847560 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836569    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 04:11:39.847560 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836576    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 04:11:39.847560 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836583    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 04:11:39.847560 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836587    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 04:11:39.847560 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836592    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 04:11:39.847560 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836597    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 04:11:39.847560 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836601    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 04:11:39.847560 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836605    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 04:11:39.847560 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836610    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 04:11:39.847560 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836614    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 04:11:39.847560 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836618    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 04:11:39.847560 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836623    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 04:11:39.847560 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836627    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 04:11:39.848074 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836632    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 04:11:39.848074 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836636    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 04:11:39.848074 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836640    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 04:11:39.848074 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836648    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 04:11:39.848074 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836652    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 04:11:39.848074 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836657    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 04:11:39.848074 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836661    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 04:11:39.848074 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836666    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 04:11:39.848074 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836670    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 04:11:39.848074 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836674    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 04:11:39.848074 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836679    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 04:11:39.848074 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836683    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 04:11:39.848074 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836689    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 04:11:39.848074 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836694    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 04:11:39.848074 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836699    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 04:11:39.848074 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836703    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 04:11:39.848074 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836707    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 04:11:39.848074 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836712    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 04:11:39.848074 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836716    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 04:11:39.848074 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836722    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 04:11:39.848575 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836726    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 04:11:39.848575 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836731    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 04:11:39.848575 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836735    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 04:11:39.848575 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836740    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 04:11:39.848575 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836744    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 04:11:39.848575 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836748    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 04:11:39.848575 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836752    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 04:11:39.848575 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836756    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 04:11:39.848575 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836760    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 04:11:39.848575 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836765    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 04:11:39.848575 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836770    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 04:11:39.848575 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836774    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 04:11:39.848575 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836779    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 04:11:39.848575 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836786    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 04:11:39.848575 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836792    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 04:11:39.848575 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836802    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 04:11:39.848575 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836808    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 04:11:39.848575 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836813    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 04:11:39.848575 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836818    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 04:11:39.848575 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836823    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 04:11:39.849154 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836828    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 04:11:39.849154 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836833    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 04:11:39.849154 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836855    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 04:11:39.849154 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836860    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 04:11:39.849154 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836865    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 04:11:39.849154 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836869    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 04:11:39.849154 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836874    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 04:11:39.849154 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836880    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 04:11:39.849154 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836887    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 04:11:39.849154 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836892    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 04:11:39.849154 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836897    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 04:11:39.849154 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836903    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 04:11:39.849154 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836908    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 04:11:39.849154 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836913    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 04:11:39.849154 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836917    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 04:11:39.849154 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836922    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 04:11:39.849154 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836926    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 04:11:39.849154 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.836931    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 04:11:39.849603 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.837426    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 04:11:39.849603 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.846507    2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 04:11:39.849603 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.846523    2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 04:11:39.849603 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.846982    2566 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 04:11:39.849603 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.846994    2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 04:11:39.849603 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847053    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 04:11:39.849603 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847059    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 04:11:39.849603 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847062    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 04:11:39.849603 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847066    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 04:11:39.849603 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847068    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 04:11:39.849603 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847071    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 04:11:39.849603 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847074    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 04:11:39.849603 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847077    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 04:11:39.849603 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847079    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 04:11:39.849603 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847082    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 04:11:39.850053 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847085    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 04:11:39.850053 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847087    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 04:11:39.850053 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847090    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 04:11:39.850053 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847093    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 04:11:39.850053 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847095    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 04:11:39.850053 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847098    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 04:11:39.850053 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847102    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 04:11:39.850053 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847105    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 04:11:39.850053 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847107    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 04:11:39.850053 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847111    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 04:11:39.850053 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847113    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 04:11:39.850053 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847116    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 04:11:39.850053 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847118    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 04:11:39.850053 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847121    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 04:11:39.850053 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847123    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 04:11:39.850053 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847126    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 04:11:39.850053 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847129    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 04:11:39.850053 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847131    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 04:11:39.850053 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847134    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 04:11:39.850053 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847137    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 04:11:39.850592 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847142    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 04:11:39.850592 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847146    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 04:11:39.850592 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847149    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 04:11:39.850592 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847152    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 04:11:39.850592 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847155    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 04:11:39.850592 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847157    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 04:11:39.850592 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847160    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 04:11:39.850592 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847163    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 04:11:39.850592 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847165    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 04:11:39.850592 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847168    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 04:11:39.850592 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847171    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 04:11:39.850592 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847173    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 04:11:39.850592 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847176    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 04:11:39.850592 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847178    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 04:11:39.850592 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847181    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 04:11:39.850592 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847184    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 04:11:39.850592 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847188    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 04:11:39.850592 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847191    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 04:11:39.850592 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847193    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 04:11:39.851096 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847196    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 04:11:39.851096 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847199    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 04:11:39.851096 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847202    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 04:11:39.851096 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847204    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 04:11:39.851096 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847207    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 04:11:39.851096 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847209    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 04:11:39.851096 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847212    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 04:11:39.851096 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847215    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 04:11:39.851096 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847217    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 04:11:39.851096 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847220    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 04:11:39.851096 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847222    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 04:11:39.851096 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847225    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 04:11:39.851096 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847228    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 04:11:39.851096 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847230    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 04:11:39.851096 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847233    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 04:11:39.851096 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847236    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 04:11:39.851096 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847238    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 04:11:39.851096 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847241    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 04:11:39.851096 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847243    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 04:11:39.851096 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847246    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 04:11:39.851956 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847249    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 04:11:39.851956 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847251    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 04:11:39.851956 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847254    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 04:11:39.851956 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847258    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 04:11:39.851956 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847261    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 04:11:39.851956 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847264    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 04:11:39.851956 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847266    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 04:11:39.851956 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847269    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 04:11:39.851956 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847272    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 04:11:39.851956 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847275    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 04:11:39.851956 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847279    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 04:11:39.851956 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847281    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 04:11:39.851956 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847284    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 04:11:39.851956 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847287    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 04:11:39.851956 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847289    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 04:11:39.851956 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847292    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 04:11:39.851956 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847294    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 04:11:39.851956 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847297    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 04:11:39.851956 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847300    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 04:11:39.852616 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847302    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 04:11:39.852616 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847305    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 04:11:39.852616 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847307    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 04:11:39.852616 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847311    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 04:11:39.852616 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847315    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 04:11:39.852616 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847318    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 04:11:39.852616 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847321    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 04:11:39.852616 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847324    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 04:11:39.852616 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847326    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 04:11:39.852616 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847329    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 04:11:39.852616 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847332    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 04:11:39.852616 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.847337    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 04:11:39.852616 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847450    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 04:11:39.852616 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847455    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 04:11:39.852616 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847459    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 04:11:39.853033 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847462    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 04:11:39.853033 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847465    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 04:11:39.853033 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847467    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 04:11:39.853033 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847470    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 04:11:39.853033 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847473    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 04:11:39.853033 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847475    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 04:11:39.853033 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847478    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 04:11:39.853033 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847481    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 04:11:39.853033 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847484    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 04:11:39.853033 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847487    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 04:11:39.853033 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847490    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 04:11:39.853033 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847493    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 04:11:39.853033 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847495    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 04:11:39.853033 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847498    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 04:11:39.853033 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847500    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 04:11:39.853033 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847503    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 04:11:39.853033 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847505    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 04:11:39.853033 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847508    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 04:11:39.853033 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847510    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 04:11:39.853033 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847514    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 04:11:39.853646 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847518    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 04:11:39.853646 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847521    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 04:11:39.853646 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847524    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 04:11:39.853646 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847528    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 04:11:39.853646 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847532    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 04:11:39.853646 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847535    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 04:11:39.853646 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847538    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 04:11:39.853646 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847541    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 04:11:39.853646 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847543    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 04:11:39.853646 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847546    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 04:11:39.853646 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847549    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 04:11:39.853646 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847552    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 04:11:39.853646 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847555    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 04:11:39.853646 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847557    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 04:11:39.853646 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847559    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 04:11:39.853646 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847562    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 04:11:39.853646 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847565    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 04:11:39.853646 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847568    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 04:11:39.853646 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847571    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 04:11:39.854354 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847574    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 04:11:39.854354 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847576    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 04:11:39.854354 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847579    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 04:11:39.854354 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847582    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 04:11:39.854354 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847585    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 04:11:39.854354 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847588    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 04:11:39.854354 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847590    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 04:11:39.854354 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847592    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 04:11:39.854354 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847595    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 04:11:39.854354 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847597    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 04:11:39.854354 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847600    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 04:11:39.854354 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847602    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 04:11:39.854354 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847605    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 04:11:39.854354 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847608    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 04:11:39.854354 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847610    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 04:11:39.854354 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847613    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 04:11:39.854354 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847615    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 04:11:39.854354 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847618    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 04:11:39.854354 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847620    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 04:11:39.854354 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847623    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 04:11:39.854953 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847626    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 04:11:39.854953 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847628    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 04:11:39.854953 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847630    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 04:11:39.854953 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847633    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 04:11:39.854953 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847636    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 04:11:39.854953 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847638    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 04:11:39.854953 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847641    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 04:11:39.854953 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847643    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 04:11:39.854953 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847646    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 04:11:39.854953 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847648    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 04:11:39.854953 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847651    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 04:11:39.854953 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847653    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 04:11:39.854953 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847657    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 04:11:39.854953 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847660    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 04:11:39.854953 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847663    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 04:11:39.854953 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847673    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 04:11:39.854953 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847677    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 04:11:39.854953 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847681    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 04:11:39.854953 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847684    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 04:11:39.855453 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847686    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 04:11:39.855453 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847689    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 04:11:39.855453 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847692    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 04:11:39.855453 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847695    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 04:11:39.855453 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847697    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 04:11:39.855453 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847700    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 04:11:39.855453 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847703    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 04:11:39.855453 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847706    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 04:11:39.855453 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847708    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 04:11:39.855453 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847711    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 04:11:39.855453 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847714    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 04:11:39.855453 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847716    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 04:11:39.855453 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847719    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 04:11:39.855453 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847721    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 04:11:39.855453 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847724    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 04:11:39.855453 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847726    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 04:11:39.855453 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847729    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 04:11:39.855453 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:39.847732    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 04:11:39.856006 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.847737    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 04:11:39.856006 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.847891    2566 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 04:11:39.856006 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.849822    2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 04:11:39.856006 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.851166    2566 server.go:1019] "Starting client certificate rotation"
Jul 28 04:11:39.856006 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.851277    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 04:11:39.856006 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.851328    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 04:11:39.856006 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.853416    2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 04:11:39.856006 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.854602    2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 04:11:39.863464 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.863448    2566 log.go:25] "Validated CRI v1 runtime API"
Jul 28 04:11:39.866850 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.866824    2566 log.go:25] "Validated CRI v1 image API"
Jul 28 04:11:39.867735 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.867718    2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 04:11:39.872153 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.872131    2566 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 f728d804-d40b-451e-a180-46bde21b14f6:/dev/nvme0n1p3 f88dfcb6-40da-44df-bf12-7e10272c49dd:/dev/nvme0n1p4]
Jul 28 04:11:39.872241 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.872159    2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 04:11:39.877186 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.877073    2566 manager.go:216] Machine: {Timestamp:2026-07-28 04:11:39.876035387 +0000 UTC m=+0.266211400 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096105 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21459b925e450c4f7fcc3705558f8a SystemUUID:ec21459b-925e-450c-4f7f-cc3705558f8a BootID:48f1f308-fbeb-49a6-8c25-f64e1973dcf0 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f9:b4:64:d5:3b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f9:b4:64:d5:3b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:da:8a:67:cb:54:9f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 04:11:39.877186 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.877174    2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 04:11:39.877337 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.877282    2566 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 04:11:39.879014 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.878961    2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 04:11:39.879486 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.879020    2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-187.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 04:11:39.880161 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.880135    2566 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 04:11:39.880161 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.880156    2566 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 04:11:39.880161 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.880166    2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 04:11:39.880358 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.880188    2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 04:11:39.880394 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.880371    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 04:11:39.881005 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.880993    2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 04:11:39.883646 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.883634    2566 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 04:11:39.883809 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.883799    2566 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 04:11:39.883858 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.883830    2566 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 04:11:39.883858 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.883855    2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 04:11:39.883919 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.883871    2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 04:11:39.883919 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.883880    2566 kubelet.go:398] "Adding apiserver pod source"
Jul 28 04:11:39.883919 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.883888    2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 04:11:39.884241 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.884230    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 04:11:39.884280 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.884248    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 04:11:39.886025 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.886006    2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 04:11:39.887805 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.887791    2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 04:11:39.887866 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.887822    2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 04:11:39.889310 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.889297    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 04:11:39.889310 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.889316    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 04:11:39.889400 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.889322    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 04:11:39.889400 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.889327    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 04:11:39.889400 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.889337    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 04:11:39.889400 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.889343    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 04:11:39.889400 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.889352    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 04:11:39.889400 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.889357    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 04:11:39.889400 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.889364    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 04:11:39.889400 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.889370    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 04:11:39.889400 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.889378    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 04:11:39.889400 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.889383    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 04:11:39.890456 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.890443    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 04:11:39.890490 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.890459    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 04:11:39.893553 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.893529    2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-187.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 04:11:39.893630 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:39.893564    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-187.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 04:11:39.893630 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:39.893569    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 04:11:39.893858 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.893830    2566 server.go:1268] "Started kubelet"
Jul 28 04:11:39.893959 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.893901    2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 04:11:39.894008 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.893933    2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 04:11:39.894079 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.894061    2566 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 04:11:39.894725 ip-10-0-129-187 systemd[1]: Started Kubernetes Kubelet.
Jul 28 04:11:39.895209 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.895194    2566 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 04:11:39.895502 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.895487    2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 04:11:39.899471 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:39.899449    2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 04:11:39.901683 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.901665    2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 04:11:39.901683 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.901675    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 04:11:39.901898 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:39.901822    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-187.ec2.internal\" not found"
Jul 28 04:11:39.902395 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.902376    2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 04:11:39.902395 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.902377    2566 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 04:11:39.902512 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.902406    2566 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 04:11:39.902512 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:39.901580    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-187.ec2.internal.18c658c3b5f8675d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-187.ec2.internal,UID:ip-10-0-129-187.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-187.ec2.internal,},FirstTimestamp:2026-07-28 04:11:39.893802845 +0000 UTC m=+0.283978857,LastTimestamp:2026-07-28 04:11:39.893802845 +0000 UTC m=+0.283978857,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-187.ec2.internal,}"
Jul 28 04:11:39.902512 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.902474    2566 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 04:11:39.902512 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.902484    2566 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 04:11:39.902645 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.902548    2566 factory.go:55] Registering systemd factory
Jul 28 04:11:39.902645 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.902562    2566 factory.go:223] Registration of the systemd container factory successfully
Jul 28 04:11:39.902784 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.902771    2566 factory.go:153] Registering CRI-O factory
Jul 28 04:11:39.902784 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.902782    2566 factory.go:223] Registration of the crio container factory successfully
Jul 28 04:11:39.902913 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.902863    2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 04:11:39.902913 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.902886    2566 factory.go:103] Registering Raw factory
Jul 28 04:11:39.902913 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.902899    2566 manager.go:1196] Started watching for new ooms in manager
Jul 28 04:11:39.903452 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.903436    2566 manager.go:349] Starting recovery of all containers
Jul 28 04:11:39.905322 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:39.905294    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 04:11:39.905430 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:39.905379    2566 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-187.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 04:11:39.914496 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.914478    2566 manager.go:354] Recovery completed
Jul 28 04:11:39.918448 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.918433    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 04:11:39.919358 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.919342    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rdbpw"
Jul 28 04:11:39.920104 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.920086    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-187.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 04:11:39.920165 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.920116    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-187.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 04:11:39.920165 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.920126    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-187.ec2.internal" event="NodeHasSufficientPID"
Jul 28 04:11:39.920581 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.920565    2566 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 04:11:39.920581 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.920580    2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 04:11:39.920732 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.920599    2566 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 04:11:39.920732 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.920580    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 04:11:39.921261 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.921246    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-187.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 04:11:39.921309 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.921274    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-187.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 04:11:39.921309 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.921283    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-187.ec2.internal" event="NodeHasSufficientPID"
Jul 28 04:11:39.922054 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:39.921979    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-187.ec2.internal.18c658c3b789b8e9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-187.ec2.internal,UID:ip-10-0-129-187.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-187.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-187.ec2.internal,},FirstTimestamp:2026-07-28 04:11:39.920103657 +0000 UTC m=+0.310279668,LastTimestamp:2026-07-28 04:11:39.920103657 +0000 UTC m=+0.310279668,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-187.ec2.internal,}"
Jul 28 04:11:39.923283 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.923267    2566 policy_none.go:49] "None policy: Start"
Jul 28 04:11:39.923361 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.923287    2566 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 04:11:39.923361 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.923301    2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 04:11:39.925345 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.925332    2566 policy_none.go:47] "Start"
Jul 28 04:11:39.926536 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.926391    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rdbpw"
Jul 28 04:11:39.931706 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:39.931637    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-187.ec2.internal.18c658c3b789facd  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-187.ec2.internal,UID:ip-10-0-129-187.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-187.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-187.ec2.internal,},FirstTimestamp:2026-07-28 04:11:39.920120525 +0000 UTC m=+0.310296537,LastTimestamp:2026-07-28 04:11:39.920120525 +0000 UTC m=+0.310296537,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-187.ec2.internal,}"
Jul 28 04:11:39.961431 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.961414    2566 manager.go:337] "Starting Device Plugin manager"
Jul 28 04:11:39.961546 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:39.961453    2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 04:11:39.961546 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.961466    2566 server.go:87] "Starting device plugin registration server"
Jul 28 04:11:39.961730 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.961717    2566 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 04:11:39.961782 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.961731    2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 04:11:39.961890 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.961832    2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 04:11:39.961954 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.961935    2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 04:11:39.961954 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:39.961945    2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 04:11:39.963131 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:39.963114    2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 04:11:39.963208 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:39.963157    2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-187.ec2.internal\" not found"
Jul 28 04:11:40.062073 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.062042    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 04:11:40.064356 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.062982    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-187.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 04:11:40.064356 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.063012    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-187.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 04:11:40.064356 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.063023    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-187.ec2.internal" event="NodeHasSufficientPID"
Jul 28 04:11:40.064356 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.063053    2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-187.ec2.internal"
Jul 28 04:11:40.070820 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.070805    2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-187.ec2.internal"
Jul 28 04:11:40.070901 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:40.070827    2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-187.ec2.internal\": node \"ip-10-0-129-187.ec2.internal\" not found"
Jul 28 04:11:40.080494 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.080445    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 04:11:40.081681 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.081664    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 04:11:40.081781 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.081713    2566 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 04:11:40.081781 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.081744    2566 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 04:11:40.081901 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:40.081780    2566 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 04:11:40.083381 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.083362    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 04:11:40.085863 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:40.085831    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-187.ec2.internal\" not found"
Jul 28 04:11:40.182465 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.182430    2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-187.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-187.ec2.internal"]
Jul 28 04:11:40.182609 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.182506    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 04:11:40.184243 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.184227    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-187.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 04:11:40.184347 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.184257    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-187.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 04:11:40.184347 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.184267    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-187.ec2.internal" event="NodeHasSufficientPID"
Jul 28 04:11:40.186374 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:40.186359    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-187.ec2.internal\" not found"
Jul 28 04:11:40.186667 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.186656    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 04:11:40.186791 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.186777    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-187.ec2.internal"
Jul 28 04:11:40.186834 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.186818    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 04:11:40.187474 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.187459    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-187.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 04:11:40.187551 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.187488    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-187.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 04:11:40.187551 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.187498    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-187.ec2.internal" event="NodeHasSufficientPID"
Jul 28 04:11:40.187551 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.187459    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-187.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 04:11:40.187694 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.187576    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-187.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 04:11:40.187694 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.187594    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-187.ec2.internal" event="NodeHasSufficientPID"
Jul 28 04:11:40.189829 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.189812    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-187.ec2.internal"
Jul 28 04:11:40.189924 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.189853    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 04:11:40.190702 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.190688    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-187.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 04:11:40.190781 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.190713    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-187.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 04:11:40.190781 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.190737    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-187.ec2.internal" event="NodeHasSufficientPID"
Jul 28 04:11:40.215485 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:40.215467    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-187.ec2.internal\" not found" node="ip-10-0-129-187.ec2.internal"
Jul 28 04:11:40.219865 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:40.219833    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-187.ec2.internal\" not found" node="ip-10-0-129-187.ec2.internal"
Jul 28 04:11:40.286457 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:40.286424    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-187.ec2.internal\" not found"
Jul 28 04:11:40.303830 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.303802    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c9650e72cdb5110eb610920a5f32b652-config\") pod \"kube-apiserver-proxy-ip-10-0-129-187.ec2.internal\" (UID: \"c9650e72cdb5110eb610920a5f32b652\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-187.ec2.internal"
Jul 28 04:11:40.303951 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.303841    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bb2b2cec0e2ae932bf2eeb6eb1815049-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-187.ec2.internal\" (UID: \"bb2b2cec0e2ae932bf2eeb6eb1815049\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-187.ec2.internal"
Jul 28 04:11:40.303951 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.303891    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb2b2cec0e2ae932bf2eeb6eb1815049-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-187.ec2.internal\" (UID: \"bb2b2cec0e2ae932bf2eeb6eb1815049\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-187.ec2.internal"
Jul 28 04:11:40.386919 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:40.386807    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-187.ec2.internal\" not found"
Jul 28 04:11:40.404419 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.404387    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c9650e72cdb5110eb610920a5f32b652-config\") pod \"kube-apiserver-proxy-ip-10-0-129-187.ec2.internal\" (UID: \"c9650e72cdb5110eb610920a5f32b652\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-187.ec2.internal"
Jul 28 04:11:40.404533 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.404436    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bb2b2cec0e2ae932bf2eeb6eb1815049-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-187.ec2.internal\" (UID: \"bb2b2cec0e2ae932bf2eeb6eb1815049\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-187.ec2.internal"
Jul 28 04:11:40.404533 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.404461    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb2b2cec0e2ae932bf2eeb6eb1815049-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-187.ec2.internal\" (UID: \"bb2b2cec0e2ae932bf2eeb6eb1815049\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-187.ec2.internal"
Jul 28 04:11:40.404533 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.404496    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb2b2cec0e2ae932bf2eeb6eb1815049-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-187.ec2.internal\" (UID: \"bb2b2cec0e2ae932bf2eeb6eb1815049\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-187.ec2.internal"
Jul 28 04:11:40.404533 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.404498    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c9650e72cdb5110eb610920a5f32b652-config\") pod \"kube-apiserver-proxy-ip-10-0-129-187.ec2.internal\" (UID: \"c9650e72cdb5110eb610920a5f32b652\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-187.ec2.internal"
Jul 28 04:11:40.404690 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.404498    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bb2b2cec0e2ae932bf2eeb6eb1815049-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-187.ec2.internal\" (UID: \"bb2b2cec0e2ae932bf2eeb6eb1815049\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-187.ec2.internal"
Jul 28 04:11:40.488965 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:40.488913    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-187.ec2.internal\" not found"
Jul 28 04:11:40.518161 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.518137    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-187.ec2.internal"
Jul 28 04:11:40.524560 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.523572    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-187.ec2.internal"
Jul 28 04:11:40.589929 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:40.589890    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-187.ec2.internal\" not found"
Jul 28 04:11:40.690997 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:40.690901    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-187.ec2.internal\" not found"
Jul 28 04:11:40.751768 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.751740    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 04:11:40.792016 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:40.791969    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-187.ec2.internal\" not found"
Jul 28 04:11:40.853789 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.853764    2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 04:11:40.854248 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.853926    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 04:11:40.854248 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.853926    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 04:11:40.893214 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:40.893168    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-187.ec2.internal\" not found"
Jul 28 04:11:40.925171 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.925145    2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 04:11:40.932069 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.932036    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 04:06:40 +0000 UTC" deadline="2028-04-25 15:44:29.505032704 +0000 UTC"
Jul 28 04:11:40.932069 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.932063    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15299h32m48.57297225s"
Jul 28 04:11:40.935450 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.935435    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 04:11:40.936387 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.936369    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-187.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 04:11:40.936509 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.936404    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-187.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 04:11:40.936509 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:40.936420    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-187.ec2.internal" event="NodeHasSufficientPID"
Jul 28 04:11:40.993832 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:40.993770    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-187.ec2.internal\" not found"
Jul 28 04:11:41.093913 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:41.093874    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-187.ec2.internal\" not found"
Jul 28 04:11:41.167544 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:41.167503    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9650e72cdb5110eb610920a5f32b652.slice/crio-6917da4f38cc9bdabff1625c7e03af3073766d288d134b0dbe53ab0c84af6d44 WatchSource:0}: Error finding container 6917da4f38cc9bdabff1625c7e03af3073766d288d134b0dbe53ab0c84af6d44: Status 404 returned error can't find the container with id 6917da4f38cc9bdabff1625c7e03af3073766d288d134b0dbe53ab0c84af6d44
Jul 28 04:11:41.174194 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.174177    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 04:11:41.181298 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:41.181276    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb2b2cec0e2ae932bf2eeb6eb1815049.slice/crio-a1608ae3cf3a270e5cda6bc340456761bb614104f8023111364517f91550eb3f WatchSource:0}: Error finding container a1608ae3cf3a270e5cda6bc340456761bb614104f8023111364517f91550eb3f: Status 404 returned error can't find the container with id a1608ae3cf3a270e5cda6bc340456761bb614104f8023111364517f91550eb3f
Jul 28 04:11:41.194636 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:41.194615    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-187.ec2.internal\" not found"
Jul 28 04:11:41.227174 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.227146    2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 04:11:41.302769 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.302679    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-187.ec2.internal"
Jul 28 04:11:41.315620 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.315596    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 04:11:41.316491 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.316477    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-187.ec2.internal"
Jul 28 04:11:41.325126 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.325107    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 04:11:41.885140 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.885096    2566 apiserver.go:52] "Watching apiserver"
Jul 28 04:11:41.895471 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.895140    2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 04:11:41.896821 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.896351    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-z2gn6","kube-system/konnectivity-agent-vfg8j","kube-system/kube-apiserver-proxy-ip-10-0-129-187.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7v87k","openshift-cluster-node-tuning-operator/tuned-4scq5","openshift-dns/node-resolver-tlsvx","openshift-image-registry/node-ca-t6nfz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-187.ec2.internal","openshift-multus/multus-additional-cni-plugins-fk4hq","openshift-multus/multus-jxpf8","openshift-multus/network-metrics-daemon-l6p44","openshift-network-diagnostics/network-check-target-qgqqn","openshift-network-operator/iptables-alerter-l2b28","openshift-ovn-kubernetes/ovnkube-node-8xswd"]
Jul 28 04:11:41.899496 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.899215    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t6nfz"
Jul 28 04:11:41.901805 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.901773    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l2b28"
Jul 28 04:11:41.903017 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.902316    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 04:11:41.903017 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.902496    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bzjvz\""
Jul 28 04:11:41.903017 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.902511    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 04:11:41.903017 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.902646    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 04:11:41.903017 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.902964    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 04:11:41.906200 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.904484    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 04:11:41.906200 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.904746    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-cbcbp\""
Jul 28 04:11:41.906200 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.904975    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 04:11:41.906200 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.905366    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 04:11:41.906200 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.905618    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:41.907786 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.907295    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7v87k"
Jul 28 04:11:41.909163 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.908099    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 04:11:41.909163 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.908489    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 04:11:41.909734 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.909528    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8dr2z\""
Jul 28 04:11:41.909734 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.909715    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 04:11:41.911138 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.910123    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 04:11:41.911138 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.910278    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-279wd\""
Jul 28 04:11:41.911138 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.910326    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 04:11:41.911138 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.910489    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 04:11:41.911138 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.910588    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 04:11:41.911138 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.910741    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 04:11:41.911138 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.911041    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 04:11:41.911138 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.911071    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4scq5"
Jul 28 04:11:41.914401 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.912820    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8cf586fe-2e9b-474b-8fa6-3ca6317e2b31-host\") pod \"node-ca-t6nfz\" (UID: \"8cf586fe-2e9b-474b-8fa6-3ca6317e2b31\") " pod="openshift-image-registry/node-ca-t6nfz"
Jul 28 04:11:41.914401 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.912853    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/abf055f7-9730-480a-897d-6fa35cd77b61-host-kubelet\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:41.914401 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.912896    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abf055f7-9730-480a-897d-6fa35cd77b61-var-lib-openvswitch\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:41.914401 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.912920    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/abf055f7-9730-480a-897d-6fa35cd77b61-run-ovn\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:41.914401 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.912988    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/abf055f7-9730-480a-897d-6fa35cd77b61-host-run-netns\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:41.914401 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.913012    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/abf055f7-9730-480a-897d-6fa35cd77b61-log-socket\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:41.914401 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.913045    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8cf586fe-2e9b-474b-8fa6-3ca6317e2b31-serviceca\") pod \"node-ca-t6nfz\" (UID: \"8cf586fe-2e9b-474b-8fa6-3ca6317e2b31\") " pod="openshift-image-registry/node-ca-t6nfz"
Jul 28 04:11:41.914401 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.913070    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxcng\" (UniqueName: \"kubernetes.io/projected/8cf586fe-2e9b-474b-8fa6-3ca6317e2b31-kube-api-access-nxcng\") pod \"node-ca-t6nfz\" (UID: \"8cf586fe-2e9b-474b-8fa6-3ca6317e2b31\") " pod="openshift-image-registry/node-ca-t6nfz"
Jul 28 04:11:41.914401 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.913096    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a6783c1-279b-45e7-acf2-b56e2e9f1908-host-slash\") pod \"iptables-alerter-l2b28\" (UID: \"1a6783c1-279b-45e7-acf2-b56e2e9f1908\") " pod="openshift-network-operator/iptables-alerter-l2b28"
Jul 28 04:11:41.914401 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.913121    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1a6783c1-279b-45e7-acf2-b56e2e9f1908-iptables-alerter-script\") pod \"iptables-alerter-l2b28\" (UID: \"1a6783c1-279b-45e7-acf2-b56e2e9f1908\") " pod="openshift-network-operator/iptables-alerter-l2b28"
Jul 28 04:11:41.914401 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.913146    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/abf055f7-9730-480a-897d-6fa35cd77b61-host-cni-netd\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:41.914401 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.913181    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/abf055f7-9730-480a-897d-6fa35cd77b61-ovnkube-config\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:41.914401 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.913204    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/abf055f7-9730-480a-897d-6fa35cd77b61-env-overrides\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:41.914401 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.913229    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abf055f7-9730-480a-897d-6fa35cd77b61-etc-openvswitch\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:41.914401 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.913266    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abf055f7-9730-480a-897d-6fa35cd77b61-run-openvswitch\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:41.914401 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.913293    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abf055f7-9730-480a-897d-6fa35cd77b61-host-run-ovn-kubernetes\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:41.915296 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.913335    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abf055f7-9730-480a-897d-6fa35cd77b61-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:41.915296 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.913364    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/abf055f7-9730-480a-897d-6fa35cd77b61-ovn-node-metrics-cert\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:41.915296 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.913389    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsswm\" (UniqueName: \"kubernetes.io/projected/abf055f7-9730-480a-897d-6fa35cd77b61-kube-api-access-qsswm\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:41.915296 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.913420    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx25s\" (UniqueName: \"kubernetes.io/projected/1a6783c1-279b-45e7-acf2-b56e2e9f1908-kube-api-access-rx25s\") pod \"iptables-alerter-l2b28\" (UID: \"1a6783c1-279b-45e7-acf2-b56e2e9f1908\") " pod="openshift-network-operator/iptables-alerter-l2b28"
Jul 28 04:11:41.915296 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.913443    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/abf055f7-9730-480a-897d-6fa35cd77b61-host-slash\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:41.915296 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.913467    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/abf055f7-9730-480a-897d-6fa35cd77b61-node-log\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:41.915296 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.913490    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/abf055f7-9730-480a-897d-6fa35cd77b61-host-cni-bin\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:41.915296 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.913533    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/abf055f7-9730-480a-897d-6fa35cd77b61-systemd-units\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:41.915296 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.913564    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/abf055f7-9730-480a-897d-6fa35cd77b61-run-systemd\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:41.915296 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.913612    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/abf055f7-9730-480a-897d-6fa35cd77b61-ovnkube-script-lib\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:41.915296 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.913991    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jftcw\""
Jul 28 04:11:41.915296 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.914018    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 04:11:41.915296 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.914216    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tlsvx"
Jul 28 04:11:41.915296 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.914224    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 04:11:41.916057 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.916012    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fk4hq"
Jul 28 04:11:41.919074 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.917448    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bzh7h\""
Jul 28 04:11:41.919074 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.917633    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 04:11:41.919074 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.918124    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 04:11:41.919074 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.918602    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 04:11:41.919312 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.919220    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2gn6"
Jul 28 04:11:41.919662 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.919358    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mjpwv\""
Jul 28 04:11:41.919662 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:41.919611    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2gn6" podUID="accccd5b-603e-49d5-b9fc-b328dcb7e959"
Jul 28 04:11:41.923224 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.922181    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vfg8j"
Jul 28 04:11:41.924254 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.923463    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 04:11:41.924254 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.923833    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 04:11:41.924254 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.923909    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 04:11:41.924254 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.924096    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 04:11:41.928398 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.924867    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 04:11:41.928398 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.925467    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:41.928398 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.925641    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 04:11:41.928398 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.925847    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-kzlf2\""
Jul 28 04:11:41.928398 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.926081    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 04:11:41.928398 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.928389    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 04:11:41.928809 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.928789    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-xmg7q\""
Jul 28 04:11:41.929116 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.929099    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6p44"
Jul 28 04:11:41.929201 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:41.929165    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6p44" podUID="e6107883-62a6-4a82-9e04-ed46057a6b0e"
Jul 28 04:11:41.930612 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.930503    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qgqqn"
Jul 28 04:11:41.930612 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:41.930569    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qgqqn" podUID="33a30d7b-76be-42e9-ae95-6779c5770b15"
Jul 28 04:11:41.937355 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.937333    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-cv9st"
Jul 28 04:11:41.945296 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:41.944939    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cv9st"
Jul 28 04:11:42.004726 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.004706    2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 04:11:42.014608 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.014582    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8cf586fe-2e9b-474b-8fa6-3ca6317e2b31-host\") pod \"node-ca-t6nfz\" (UID: \"8cf586fe-2e9b-474b-8fa6-3ca6317e2b31\") " pod="openshift-image-registry/node-ca-t6nfz"
Jul 28 04:11:42.014724 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.014624    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abf055f7-9730-480a-897d-6fa35cd77b61-var-lib-openvswitch\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:42.014724 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.014654    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/abf055f7-9730-480a-897d-6fa35cd77b61-run-ovn\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:42.014724 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.014686    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptvxq\" (UniqueName: \"kubernetes.io/projected/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-kube-api-access-ptvxq\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.014724 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.014714    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27f106d2-f008-49e7-8928-623a0f10c6c9-var-lib-kubelet\") pod \"tuned-4scq5\" (UID: \"27f106d2-f008-49e7-8928-623a0f10c6c9\") " pod="openshift-cluster-node-tuning-operator/tuned-4scq5"
Jul 28 04:11:42.014963 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.014741    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/abf055f7-9730-480a-897d-6fa35cd77b61-host-run-netns\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:42.014963 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.014747    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abf055f7-9730-480a-897d-6fa35cd77b61-var-lib-openvswitch\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:42.014963 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.014767    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmnvx\" (UniqueName: \"kubernetes.io/projected/33a30d7b-76be-42e9-ae95-6779c5770b15-kube-api-access-zmnvx\") pod \"network-check-target-qgqqn\" (UID: \"33a30d7b-76be-42e9-ae95-6779c5770b15\") " pod="openshift-network-diagnostics/network-check-target-qgqqn"
Jul 28 04:11:42.014963 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.014794    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5b11f875-be7a-4fe1-8ca4-40bb7725889b-etc-selinux\") pod \"aws-ebs-csi-driver-node-7v87k\" (UID: \"5b11f875-be7a-4fe1-8ca4-40bb7725889b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7v87k"
Jul 28 04:11:42.014963 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.014813    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8cf586fe-2e9b-474b-8fa6-3ca6317e2b31-host\") pod \"node-ca-t6nfz\" (UID: \"8cf586fe-2e9b-474b-8fa6-3ca6317e2b31\") " pod="openshift-image-registry/node-ca-t6nfz"
Jul 28 04:11:42.014963 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.014818    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8cf586fe-2e9b-474b-8fa6-3ca6317e2b31-serviceca\") pod \"node-ca-t6nfz\" (UID: \"8cf586fe-2e9b-474b-8fa6-3ca6317e2b31\") " pod="openshift-image-registry/node-ca-t6nfz"
Jul 28 04:11:42.014963 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.014862    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxcng\" (UniqueName: \"kubernetes.io/projected/8cf586fe-2e9b-474b-8fa6-3ca6317e2b31-kube-api-access-nxcng\") pod \"node-ca-t6nfz\" (UID: \"8cf586fe-2e9b-474b-8fa6-3ca6317e2b31\") " pod="openshift-image-registry/node-ca-t6nfz"
Jul 28 04:11:42.014963 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.014895    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a6783c1-279b-45e7-acf2-b56e2e9f1908-host-slash\") pod \"iptables-alerter-l2b28\" (UID: \"1a6783c1-279b-45e7-acf2-b56e2e9f1908\") " pod="openshift-network-operator/iptables-alerter-l2b28"
Jul 28 04:11:42.014963 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.014943    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5b11f875-be7a-4fe1-8ca4-40bb7725889b-socket-dir\") pod \"aws-ebs-csi-driver-node-7v87k\" (UID: \"5b11f875-be7a-4fe1-8ca4-40bb7725889b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7v87k"
Jul 28 04:11:42.015402 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.014973    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-system-cni-dir\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.015402 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.015000    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/27f106d2-f008-49e7-8928-623a0f10c6c9-etc-tuned\") pod \"tuned-4scq5\" (UID: \"27f106d2-f008-49e7-8928-623a0f10c6c9\") " pod="openshift-cluster-node-tuning-operator/tuned-4scq5"
Jul 28 04:11:42.015402 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.015030    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1a6783c1-279b-45e7-acf2-b56e2e9f1908-iptables-alerter-script\") pod \"iptables-alerter-l2b28\" (UID: \"1a6783c1-279b-45e7-acf2-b56e2e9f1908\") " pod="openshift-network-operator/iptables-alerter-l2b28"
Jul 28 04:11:42.015402 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.015058    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5b11f875-be7a-4fe1-8ca4-40bb7725889b-registration-dir\") pod \"aws-ebs-csi-driver-node-7v87k\" (UID: \"5b11f875-be7a-4fe1-8ca4-40bb7725889b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7v87k"
Jul 28 04:11:42.015402 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.015087    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-os-release\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.015402 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.015115    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-host-run-netns\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.015402 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.015141    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27f106d2-f008-49e7-8928-623a0f10c6c9-sys\") pod \"tuned-4scq5\" (UID: \"27f106d2-f008-49e7-8928-623a0f10c6c9\") " pod="openshift-cluster-node-tuning-operator/tuned-4scq5"
Jul 28 04:11:42.015402 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.015171    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/068d70ee-9efb-49a5-94cb-2df7591a4652-tmp-dir\") pod \"node-resolver-tlsvx\" (UID: \"068d70ee-9efb-49a5-94cb-2df7591a4652\") " pod="openshift-dns/node-resolver-tlsvx"
Jul 28 04:11:42.015402 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.015201    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5b11f875-be7a-4fe1-8ca4-40bb7725889b-device-dir\") pod \"aws-ebs-csi-driver-node-7v87k\" (UID: \"5b11f875-be7a-4fe1-8ca4-40bb7725889b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7v87k"
Jul 28 04:11:42.015402 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.015228    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5b11f875-be7a-4fe1-8ca4-40bb7725889b-sys-fs\") pod \"aws-ebs-csi-driver-node-7v87k\" (UID: \"5b11f875-be7a-4fe1-8ca4-40bb7725889b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7v87k"
Jul 28 04:11:42.015402 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.015257    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/accccd5b-603e-49d5-b9fc-b328dcb7e959-original-pull-secret\") pod \"global-pull-secret-syncer-z2gn6\" (UID: \"accccd5b-603e-49d5-b9fc-b328dcb7e959\") " pod="kube-system/global-pull-secret-syncer-z2gn6"
Jul 28 04:11:42.015402 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.015284    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-multus-conf-dir\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.015402 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.015310    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/27f106d2-f008-49e7-8928-623a0f10c6c9-etc-sysctl-conf\") pod \"tuned-4scq5\" (UID: \"27f106d2-f008-49e7-8928-623a0f10c6c9\") " pod="openshift-cluster-node-tuning-operator/tuned-4scq5"
Jul 28 04:11:42.015402 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.015337    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/068d70ee-9efb-49a5-94cb-2df7591a4652-hosts-file\") pod \"node-resolver-tlsvx\" (UID: \"068d70ee-9efb-49a5-94cb-2df7591a4652\") " pod="openshift-dns/node-resolver-tlsvx"
Jul 28 04:11:42.015402 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.015370    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abf055f7-9730-480a-897d-6fa35cd77b61-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:42.015402 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.015400    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/93788c64-611b-40ff-be6c-8dcea894c08d-cni-binary-copy\") pod \"multus-additional-cni-plugins-fk4hq\" (UID: \"93788c64-611b-40ff-be6c-8dcea894c08d\") " pod="openshift-multus/multus-additional-cni-plugins-fk4hq"
Jul 28 04:11:42.016308 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.015432    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8cf586fe-2e9b-474b-8fa6-3ca6317e2b31-serviceca\") pod \"node-ca-t6nfz\" (UID: \"8cf586fe-2e9b-474b-8fa6-3ca6317e2b31\") " pod="openshift-image-registry/node-ca-t6nfz"
Jul 28 04:11:42.016308 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.015444    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-etc-kubernetes\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.016308 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.015475    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/27f106d2-f008-49e7-8928-623a0f10c6c9-etc-systemd\") pod \"tuned-4scq5\" (UID: \"27f106d2-f008-49e7-8928-623a0f10c6c9\") " pod="openshift-cluster-node-tuning-operator/tuned-4scq5"
Jul 28 04:11:42.016308 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.015504    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/abf055f7-9730-480a-897d-6fa35cd77b61-host-slash\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:42.016308 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.015525    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/abf055f7-9730-480a-897d-6fa35cd77b61-run-ovn\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:42.016308 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.015531    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/abf055f7-9730-480a-897d-6fa35cd77b61-node-log\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:42.016308 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.015932    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/accccd5b-603e-49d5-b9fc-b328dcb7e959-dbus\") pod \"global-pull-secret-syncer-z2gn6\" (UID: \"accccd5b-603e-49d5-b9fc-b328dcb7e959\") " pod="kube-system/global-pull-secret-syncer-z2gn6"
Jul 28 04:11:42.016308 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.015977    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-multus-cni-dir\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.016308 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.016009    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-host-var-lib-cni-bin\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.016308 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.016040    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n528z\" (UniqueName: \"kubernetes.io/projected/e6107883-62a6-4a82-9e04-ed46057a6b0e-kube-api-access-n528z\") pod \"network-metrics-daemon-l6p44\" (UID: \"e6107883-62a6-4a82-9e04-ed46057a6b0e\") " pod="openshift-multus/network-metrics-daemon-l6p44"
Jul 28 04:11:42.016308 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.016069    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/27f106d2-f008-49e7-8928-623a0f10c6c9-run\") pod \"tuned-4scq5\" (UID: \"27f106d2-f008-49e7-8928-623a0f10c6c9\") " pod="openshift-cluster-node-tuning-operator/tuned-4scq5"
Jul 28 04:11:42.016308 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.016095    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-cnibin\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.016308 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.016123    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-host-run-k8s-cni-cncf-io\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.016308 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.016138    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/abf055f7-9730-480a-897d-6fa35cd77b61-host-run-netns\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:42.016308 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.016182    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-host-run-multus-certs\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.016308 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.016204    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abf055f7-9730-480a-897d-6fa35cd77b61-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:42.016308 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.016210    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27f106d2-f008-49e7-8928-623a0f10c6c9-tmp\") pod \"tuned-4scq5\" (UID: \"27f106d2-f008-49e7-8928-623a0f10c6c9\") " pod="openshift-cluster-node-tuning-operator/tuned-4scq5"
Jul 28 04:11:42.017195 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.016246    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/abf055f7-9730-480a-897d-6fa35cd77b61-host-kubelet\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:42.017195 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.016269    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a6783c1-279b-45e7-acf2-b56e2e9f1908-host-slash\") pod \"iptables-alerter-l2b28\" (UID: \"1a6783c1-279b-45e7-acf2-b56e2e9f1908\") " pod="openshift-network-operator/iptables-alerter-l2b28"
Jul 28 04:11:42.017195 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.016326    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93788c64-611b-40ff-be6c-8dcea894c08d-system-cni-dir\") pod \"multus-additional-cni-plugins-fk4hq\" (UID: \"93788c64-611b-40ff-be6c-8dcea894c08d\") " pod="openshift-multus/multus-additional-cni-plugins-fk4hq"
Jul 28 04:11:42.017195 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.016354    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/93788c64-611b-40ff-be6c-8dcea894c08d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fk4hq\" (UID: \"93788c64-611b-40ff-be6c-8dcea894c08d\") " pod="openshift-multus/multus-additional-cni-plugins-fk4hq"
Jul 28 04:11:42.017195 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.016381    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6mtg\" (UniqueName: \"kubernetes.io/projected/93788c64-611b-40ff-be6c-8dcea894c08d-kube-api-access-g6mtg\") pod \"multus-additional-cni-plugins-fk4hq\" (UID: \"93788c64-611b-40ff-be6c-8dcea894c08d\") " pod="openshift-multus/multus-additional-cni-plugins-fk4hq"
Jul 28 04:11:42.017195 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.016409    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/accccd5b-603e-49d5-b9fc-b328dcb7e959-kubelet-config\") pod \"global-pull-secret-syncer-z2gn6\" (UID: \"accccd5b-603e-49d5-b9fc-b328dcb7e959\") " pod="kube-system/global-pull-secret-syncer-z2gn6"
Jul 28 04:11:42.017195 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.016434    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-host-var-lib-cni-multus\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.017195 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.016462    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-host-var-lib-kubelet\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.017195 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.016487    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-hostroot\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.017195 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.016514    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/abf055f7-9730-480a-897d-6fa35cd77b61-log-socket\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:42.017195 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.016539    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2c4c0738-fb7c-4610-aab6-7cd762ebaf83-agent-certs\") pod \"konnectivity-agent-vfg8j\" (UID: \"2c4c0738-fb7c-4610-aab6-7cd762ebaf83\") " pod="kube-system/konnectivity-agent-vfg8j"
Jul 28 04:11:42.017195 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.016619    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-multus-daemon-config\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.017195 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.016680    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/abf055f7-9730-480a-897d-6fa35cd77b61-host-slash\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:42.017195 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.016681    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/abf055f7-9730-480a-897d-6fa35cd77b61-host-kubelet\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:42.017195 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.016707    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/abf055f7-9730-480a-897d-6fa35cd77b61-log-socket\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:42.017195 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.016722    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/abf055f7-9730-480a-897d-6fa35cd77b61-node-log\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:42.017195 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.016760    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/27f106d2-f008-49e7-8928-623a0f10c6c9-etc-modprobe-d\") pod \"tuned-4scq5\" (UID: \"27f106d2-f008-49e7-8928-623a0f10c6c9\") " pod="openshift-cluster-node-tuning-operator/tuned-4scq5"
Jul 28 04:11:42.017896 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.016793    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27f106d2-f008-49e7-8928-623a0f10c6c9-etc-kubernetes\") pod \"tuned-4scq5\" (UID: \"27f106d2-f008-49e7-8928-623a0f10c6c9\") " pod="openshift-cluster-node-tuning-operator/tuned-4scq5"
Jul 28 04:11:42.017896 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.016821    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/93788c64-611b-40ff-be6c-8dcea894c08d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fk4hq\" (UID: \"93788c64-611b-40ff-be6c-8dcea894c08d\") " pod="openshift-multus/multus-additional-cni-plugins-fk4hq"
Jul 28 04:11:42.017896 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.016830    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1a6783c1-279b-45e7-acf2-b56e2e9f1908-iptables-alerter-script\") pod \"iptables-alerter-l2b28\" (UID: \"1a6783c1-279b-45e7-acf2-b56e2e9f1908\") " pod="openshift-network-operator/iptables-alerter-l2b28"
Jul 28 04:11:42.017896 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.016878    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjphl\" (UniqueName: \"kubernetes.io/projected/068d70ee-9efb-49a5-94cb-2df7591a4652-kube-api-access-rjphl\") pod \"node-resolver-tlsvx\" (UID: \"068d70ee-9efb-49a5-94cb-2df7591a4652\") " pod="openshift-dns/node-resolver-tlsvx"
Jul 28 04:11:42.017896 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.016972    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/abf055f7-9730-480a-897d-6fa35cd77b61-host-cni-netd\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:42.017896 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.017012    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6107883-62a6-4a82-9e04-ed46057a6b0e-metrics-certs\") pod \"network-metrics-daemon-l6p44\" (UID: \"e6107883-62a6-4a82-9e04-ed46057a6b0e\") " pod="openshift-multus/network-metrics-daemon-l6p44"
Jul 28 04:11:42.017896 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.017038    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/abf055f7-9730-480a-897d-6fa35cd77b61-host-cni-netd\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:42.017896 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.017086    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/27f106d2-f008-49e7-8928-623a0f10c6c9-lib-modules\") pod \"tuned-4scq5\" (UID: \"27f106d2-f008-49e7-8928-623a0f10c6c9\") " pod="openshift-cluster-node-tuning-operator/tuned-4scq5"
Jul 28 04:11:42.017896 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.017136    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/27f106d2-f008-49e7-8928-623a0f10c6c9-host\") pod \"tuned-4scq5\" (UID: \"27f106d2-f008-49e7-8928-623a0f10c6c9\") " pod="openshift-cluster-node-tuning-operator/tuned-4scq5"
Jul 28 04:11:42.017896 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.017165    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddl8c\" (UniqueName: \"kubernetes.io/projected/27f106d2-f008-49e7-8928-623a0f10c6c9-kube-api-access-ddl8c\") pod \"tuned-4scq5\" (UID: \"27f106d2-f008-49e7-8928-623a0f10c6c9\") " pod="openshift-cluster-node-tuning-operator/tuned-4scq5"
Jul 28 04:11:42.017896 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.017205    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/abf055f7-9730-480a-897d-6fa35cd77b61-ovnkube-config\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:42.017896 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.017234    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/abf055f7-9730-480a-897d-6fa35cd77b61-env-overrides\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:42.017896 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.017332    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbsfz\" (UniqueName: \"kubernetes.io/projected/5b11f875-be7a-4fe1-8ca4-40bb7725889b-kube-api-access-bbsfz\") pod \"aws-ebs-csi-driver-node-7v87k\" (UID: \"5b11f875-be7a-4fe1-8ca4-40bb7725889b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7v87k"
Jul 28 04:11:42.017896 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.017350    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-cni-binary-copy\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.017896 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.017370    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-multus-socket-dir-parent\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.017896 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.017387    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abf055f7-9730-480a-897d-6fa35cd77b61-etc-openvswitch\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:42.017896 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.017402    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abf055f7-9730-480a-897d-6fa35cd77b61-run-openvswitch\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:42.018522 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.017427    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abf055f7-9730-480a-897d-6fa35cd77b61-host-run-ovn-kubernetes\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:42.018522 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.017471    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/abf055f7-9730-480a-897d-6fa35cd77b61-ovn-node-metrics-cert\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:42.018522 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.017487    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsswm\" (UniqueName: \"kubernetes.io/projected/abf055f7-9730-480a-897d-6fa35cd77b61-kube-api-access-qsswm\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:42.018522 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.017503    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/93788c64-611b-40ff-be6c-8dcea894c08d-cnibin\") pod \"multus-additional-cni-plugins-fk4hq\" (UID: \"93788c64-611b-40ff-be6c-8dcea894c08d\") " pod="openshift-multus/multus-additional-cni-plugins-fk4hq"
Jul 28 04:11:42.018522 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.017519    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/93788c64-611b-40ff-be6c-8dcea894c08d-os-release\") pod \"multus-additional-cni-plugins-fk4hq\" (UID: \"93788c64-611b-40ff-be6c-8dcea894c08d\") " pod="openshift-multus/multus-additional-cni-plugins-fk4hq"
Jul 28 04:11:42.018522 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.017537    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx25s\" (UniqueName: \"kubernetes.io/projected/1a6783c1-279b-45e7-acf2-b56e2e9f1908-kube-api-access-rx25s\") pod \"iptables-alerter-l2b28\" (UID: \"1a6783c1-279b-45e7-acf2-b56e2e9f1908\") " pod="openshift-network-operator/iptables-alerter-l2b28"
Jul 28 04:11:42.018522 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.017553    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/abf055f7-9730-480a-897d-6fa35cd77b61-host-cni-bin\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:42.018522 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.017570    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/93788c64-611b-40ff-be6c-8dcea894c08d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fk4hq\" (UID: \"93788c64-611b-40ff-be6c-8dcea894c08d\") " pod="openshift-multus/multus-additional-cni-plugins-fk4hq"
Jul 28 04:11:42.018522 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.017586    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/27f106d2-f008-49e7-8928-623a0f10c6c9-etc-sysconfig\") pod \"tuned-4scq5\" (UID: \"27f106d2-f008-49e7-8928-623a0f10c6c9\") " pod="openshift-cluster-node-tuning-operator/tuned-4scq5"
Jul 28 04:11:42.018522 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.017627    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/27f106d2-f008-49e7-8928-623a0f10c6c9-etc-sysctl-d\") pod \"tuned-4scq5\" (UID: \"27f106d2-f008-49e7-8928-623a0f10c6c9\") " pod="openshift-cluster-node-tuning-operator/tuned-4scq5"
Jul 28 04:11:42.018522 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.017645    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/abf055f7-9730-480a-897d-6fa35cd77b61-systemd-units\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:42.018522 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.017660    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/abf055f7-9730-480a-897d-6fa35cd77b61-run-systemd\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:42.018522 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.017689    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/abf055f7-9730-480a-897d-6fa35cd77b61-ovnkube-script-lib\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:42.018522 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.017713    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5b11f875-be7a-4fe1-8ca4-40bb7725889b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7v87k\" (UID: \"5b11f875-be7a-4fe1-8ca4-40bb7725889b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7v87k"
Jul 28 04:11:42.018522 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.017730    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2c4c0738-fb7c-4610-aab6-7cd762ebaf83-konnectivity-ca\") pod \"konnectivity-agent-vfg8j\" (UID: \"2c4c0738-fb7c-4610-aab6-7cd762ebaf83\") " pod="kube-system/konnectivity-agent-vfg8j"
Jul 28 04:11:42.018522 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.017751    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/abf055f7-9730-480a-897d-6fa35cd77b61-ovnkube-config\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:42.018522 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.017790    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abf055f7-9730-480a-897d-6fa35cd77b61-etc-openvswitch\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:42.019396 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.017813    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/abf055f7-9730-480a-897d-6fa35cd77b61-run-openvswitch\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:42.019396 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.017811    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/abf055f7-9730-480a-897d-6fa35cd77b61-host-run-ovn-kubernetes\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:42.019396 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.018161    2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 04:11:42.019396 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.018191    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/abf055f7-9730-480a-897d-6fa35cd77b61-env-overrides\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:42.019396 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.018254    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/abf055f7-9730-480a-897d-6fa35cd77b61-run-systemd\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:42.019396 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.018295    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/abf055f7-9730-480a-897d-6fa35cd77b61-systemd-units\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:42.019396 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.018535    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/abf055f7-9730-480a-897d-6fa35cd77b61-host-cni-bin\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:42.019396 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.018767    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/abf055f7-9730-480a-897d-6fa35cd77b61-ovnkube-script-lib\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:42.023770 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.023712    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/abf055f7-9730-480a-897d-6fa35cd77b61-ovn-node-metrics-cert\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:42.025041 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.024934    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxcng\" (UniqueName: \"kubernetes.io/projected/8cf586fe-2e9b-474b-8fa6-3ca6317e2b31-kube-api-access-nxcng\") pod \"node-ca-t6nfz\" (UID: \"8cf586fe-2e9b-474b-8fa6-3ca6317e2b31\") " pod="openshift-image-registry/node-ca-t6nfz"
Jul 28 04:11:42.026598 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.026349    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsswm\" (UniqueName: \"kubernetes.io/projected/abf055f7-9730-480a-897d-6fa35cd77b61-kube-api-access-qsswm\") pod \"ovnkube-node-8xswd\" (UID: \"abf055f7-9730-480a-897d-6fa35cd77b61\") " pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:42.027715 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.027690    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx25s\" (UniqueName: \"kubernetes.io/projected/1a6783c1-279b-45e7-acf2-b56e2e9f1908-kube-api-access-rx25s\") pod \"iptables-alerter-l2b28\" (UID: \"1a6783c1-279b-45e7-acf2-b56e2e9f1908\") " pod="openshift-network-operator/iptables-alerter-l2b28"
Jul 28 04:11:42.088756 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.088703    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-187.ec2.internal" event={"ID":"bb2b2cec0e2ae932bf2eeb6eb1815049","Type":"ContainerStarted","Data":"a1608ae3cf3a270e5cda6bc340456761bb614104f8023111364517f91550eb3f"}
Jul 28 04:11:42.090215 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.090179    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-187.ec2.internal" event={"ID":"c9650e72cdb5110eb610920a5f32b652","Type":"ContainerStarted","Data":"6917da4f38cc9bdabff1625c7e03af3073766d288d134b0dbe53ab0c84af6d44"}
Jul 28 04:11:42.118783 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.118749    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-hostroot\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.118783 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.118792    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2c4c0738-fb7c-4610-aab6-7cd762ebaf83-agent-certs\") pod \"konnectivity-agent-vfg8j\" (UID: \"2c4c0738-fb7c-4610-aab6-7cd762ebaf83\") " pod="kube-system/konnectivity-agent-vfg8j"
Jul 28 04:11:42.119024 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.118820    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-multus-daemon-config\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.119024 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.118846    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/27f106d2-f008-49e7-8928-623a0f10c6c9-etc-modprobe-d\") pod \"tuned-4scq5\" (UID: \"27f106d2-f008-49e7-8928-623a0f10c6c9\") " pod="openshift-cluster-node-tuning-operator/tuned-4scq5"
Jul 28 04:11:42.119024 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.118869    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-hostroot\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.119024 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.118884    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27f106d2-f008-49e7-8928-623a0f10c6c9-etc-kubernetes\") pod \"tuned-4scq5\" (UID: \"27f106d2-f008-49e7-8928-623a0f10c6c9\") " pod="openshift-cluster-node-tuning-operator/tuned-4scq5"
Jul 28 04:11:42.119024 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.118910    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/93788c64-611b-40ff-be6c-8dcea894c08d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fk4hq\" (UID: \"93788c64-611b-40ff-be6c-8dcea894c08d\") " pod="openshift-multus/multus-additional-cni-plugins-fk4hq"
Jul 28 04:11:42.119024 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.118936    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjphl\" (UniqueName: \"kubernetes.io/projected/068d70ee-9efb-49a5-94cb-2df7591a4652-kube-api-access-rjphl\") pod \"node-resolver-tlsvx\" (UID: \"068d70ee-9efb-49a5-94cb-2df7591a4652\") " pod="openshift-dns/node-resolver-tlsvx"
Jul 28 04:11:42.119024 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.118961    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6107883-62a6-4a82-9e04-ed46057a6b0e-metrics-certs\") pod \"network-metrics-daemon-l6p44\" (UID: \"e6107883-62a6-4a82-9e04-ed46057a6b0e\") " pod="openshift-multus/network-metrics-daemon-l6p44"
Jul 28 04:11:42.119024 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.118989    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/27f106d2-f008-49e7-8928-623a0f10c6c9-lib-modules\") pod \"tuned-4scq5\" (UID: \"27f106d2-f008-49e7-8928-623a0f10c6c9\") " pod="openshift-cluster-node-tuning-operator/tuned-4scq5"
Jul 28 04:11:42.119024 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.119010    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/27f106d2-f008-49e7-8928-623a0f10c6c9-host\") pod \"tuned-4scq5\" (UID: \"27f106d2-f008-49e7-8928-623a0f10c6c9\") " pod="openshift-cluster-node-tuning-operator/tuned-4scq5"
Jul 28 04:11:42.119523 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.119032    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddl8c\" (UniqueName: \"kubernetes.io/projected/27f106d2-f008-49e7-8928-623a0f10c6c9-kube-api-access-ddl8c\") pod \"tuned-4scq5\" (UID: \"27f106d2-f008-49e7-8928-623a0f10c6c9\") " pod="openshift-cluster-node-tuning-operator/tuned-4scq5"
Jul 28 04:11:42.119523 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.119057    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbsfz\" (UniqueName: \"kubernetes.io/projected/5b11f875-be7a-4fe1-8ca4-40bb7725889b-kube-api-access-bbsfz\") pod \"aws-ebs-csi-driver-node-7v87k\" (UID: \"5b11f875-be7a-4fe1-8ca4-40bb7725889b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7v87k"
Jul 28 04:11:42.119523 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.119081    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-cni-binary-copy\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.119523 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.119104    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-multus-socket-dir-parent\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.119523 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.119137    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/93788c64-611b-40ff-be6c-8dcea894c08d-cnibin\") pod \"multus-additional-cni-plugins-fk4hq\" (UID: \"93788c64-611b-40ff-be6c-8dcea894c08d\") " pod="openshift-multus/multus-additional-cni-plugins-fk4hq"
Jul 28 04:11:42.119523 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.119161    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/93788c64-611b-40ff-be6c-8dcea894c08d-os-release\") pod \"multus-additional-cni-plugins-fk4hq\" (UID: \"93788c64-611b-40ff-be6c-8dcea894c08d\") " pod="openshift-multus/multus-additional-cni-plugins-fk4hq"
Jul 28 04:11:42.119523 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.119190    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/93788c64-611b-40ff-be6c-8dcea894c08d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fk4hq\" (UID: \"93788c64-611b-40ff-be6c-8dcea894c08d\") " pod="openshift-multus/multus-additional-cni-plugins-fk4hq"
Jul 28 04:11:42.119523 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.119215    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/27f106d2-f008-49e7-8928-623a0f10c6c9-etc-sysconfig\") pod \"tuned-4scq5\" (UID: \"27f106d2-f008-49e7-8928-623a0f10c6c9\") " pod="openshift-cluster-node-tuning-operator/tuned-4scq5"
Jul 28 04:11:42.119523 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.119240    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/27f106d2-f008-49e7-8928-623a0f10c6c9-etc-sysctl-d\") pod \"tuned-4scq5\" (UID: \"27f106d2-f008-49e7-8928-623a0f10c6c9\") " pod="openshift-cluster-node-tuning-operator/tuned-4scq5"
Jul 28 04:11:42.119523 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.119267    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5b11f875-be7a-4fe1-8ca4-40bb7725889b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7v87k\" (UID: \"5b11f875-be7a-4fe1-8ca4-40bb7725889b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7v87k"
Jul 28 04:11:42.119523 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.119292    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2c4c0738-fb7c-4610-aab6-7cd762ebaf83-konnectivity-ca\") pod \"konnectivity-agent-vfg8j\" (UID: \"2c4c0738-fb7c-4610-aab6-7cd762ebaf83\") " pod="kube-system/konnectivity-agent-vfg8j"
Jul 28 04:11:42.119523 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.119318    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptvxq\" (UniqueName: \"kubernetes.io/projected/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-kube-api-access-ptvxq\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.119523 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.119345    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27f106d2-f008-49e7-8928-623a0f10c6c9-var-lib-kubelet\") pod \"tuned-4scq5\" (UID: \"27f106d2-f008-49e7-8928-623a0f10c6c9\") " pod="openshift-cluster-node-tuning-operator/tuned-4scq5"
Jul 28 04:11:42.119523 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.119376    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmnvx\" (UniqueName: \"kubernetes.io/projected/33a30d7b-76be-42e9-ae95-6779c5770b15-kube-api-access-zmnvx\") pod \"network-check-target-qgqqn\" (UID: \"33a30d7b-76be-42e9-ae95-6779c5770b15\") " pod="openshift-network-diagnostics/network-check-target-qgqqn"
Jul 28 04:11:42.119523 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.119402    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5b11f875-be7a-4fe1-8ca4-40bb7725889b-etc-selinux\") pod \"aws-ebs-csi-driver-node-7v87k\" (UID: \"5b11f875-be7a-4fe1-8ca4-40bb7725889b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7v87k"
Jul 28 04:11:42.119523 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.119433    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5b11f875-be7a-4fe1-8ca4-40bb7725889b-socket-dir\") pod \"aws-ebs-csi-driver-node-7v87k\" (UID: \"5b11f875-be7a-4fe1-8ca4-40bb7725889b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7v87k"
Jul 28 04:11:42.119523 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.119459    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-system-cni-dir\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.120385 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.119484    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/27f106d2-f008-49e7-8928-623a0f10c6c9-etc-tuned\") pod \"tuned-4scq5\" (UID: \"27f106d2-f008-49e7-8928-623a0f10c6c9\") " pod="openshift-cluster-node-tuning-operator/tuned-4scq5"
Jul 28 04:11:42.120385 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.119512    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5b11f875-be7a-4fe1-8ca4-40bb7725889b-registration-dir\") pod \"aws-ebs-csi-driver-node-7v87k\" (UID: \"5b11f875-be7a-4fe1-8ca4-40bb7725889b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7v87k"
Jul 28 04:11:42.120385 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.119511    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-multus-daemon-config\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.120385 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.119536    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-os-release\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.120385 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.119561    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-host-run-netns\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.120385 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.119566    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/93788c64-611b-40ff-be6c-8dcea894c08d-cnibin\") pod \"multus-additional-cni-plugins-fk4hq\" (UID: \"93788c64-611b-40ff-be6c-8dcea894c08d\") " pod="openshift-multus/multus-additional-cni-plugins-fk4hq"
Jul 28 04:11:42.120385 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.119582    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27f106d2-f008-49e7-8928-623a0f10c6c9-sys\") pod \"tuned-4scq5\" (UID: \"27f106d2-f008-49e7-8928-623a0f10c6c9\") " pod="openshift-cluster-node-tuning-operator/tuned-4scq5"
Jul 28 04:11:42.120385 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.119606    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/068d70ee-9efb-49a5-94cb-2df7591a4652-tmp-dir\") pod \"node-resolver-tlsvx\" (UID: \"068d70ee-9efb-49a5-94cb-2df7591a4652\") " pod="openshift-dns/node-resolver-tlsvx"
Jul 28 04:11:42.120385 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.119673    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/27f106d2-f008-49e7-8928-623a0f10c6c9-lib-modules\") pod \"tuned-4scq5\" (UID: \"27f106d2-f008-49e7-8928-623a0f10c6c9\") " pod="openshift-cluster-node-tuning-operator/tuned-4scq5"
Jul 28 04:11:42.120385 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.119672    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27f106d2-f008-49e7-8928-623a0f10c6c9-etc-kubernetes\") pod \"tuned-4scq5\" (UID: \"27f106d2-f008-49e7-8928-623a0f10c6c9\") " pod="openshift-cluster-node-tuning-operator/tuned-4scq5"
Jul 28 04:11:42.120385 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.119776    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5b11f875-be7a-4fe1-8ca4-40bb7725889b-etc-selinux\") pod \"aws-ebs-csi-driver-node-7v87k\" (UID: \"5b11f875-be7a-4fe1-8ca4-40bb7725889b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7v87k"
Jul 28 04:11:42.120385 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.119792    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5b11f875-be7a-4fe1-8ca4-40bb7725889b-socket-dir\") pod \"aws-ebs-csi-driver-node-7v87k\" (UID: \"5b11f875-be7a-4fe1-8ca4-40bb7725889b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7v87k"
Jul 28 04:11:42.120385 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.119938    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/27f106d2-f008-49e7-8928-623a0f10c6c9-etc-modprobe-d\") pod \"tuned-4scq5\" (UID: \"27f106d2-f008-49e7-8928-623a0f10c6c9\") " pod="openshift-cluster-node-tuning-operator/tuned-4scq5"
Jul 28 04:11:42.120385 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.119954    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/93788c64-611b-40ff-be6c-8dcea894c08d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fk4hq\" (UID: \"93788c64-611b-40ff-be6c-8dcea894c08d\") " pod="openshift-multus/multus-additional-cni-plugins-fk4hq"
Jul 28 04:11:42.120385 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.119976    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5b11f875-be7a-4fe1-8ca4-40bb7725889b-device-dir\") pod \"aws-ebs-csi-driver-node-7v87k\" (UID: \"5b11f875-be7a-4fe1-8ca4-40bb7725889b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7v87k"
Jul 28 04:11:42.120385 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.120006    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5b11f875-be7a-4fe1-8ca4-40bb7725889b-sys-fs\") pod \"aws-ebs-csi-driver-node-7v87k\" (UID: \"5b11f875-be7a-4fe1-8ca4-40bb7725889b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7v87k"
Jul 28 04:11:42.120385 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.120027    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/accccd5b-603e-49d5-b9fc-b328dcb7e959-original-pull-secret\") pod \"global-pull-secret-syncer-z2gn6\" (UID: \"accccd5b-603e-49d5-b9fc-b328dcb7e959\") " pod="kube-system/global-pull-secret-syncer-z2gn6"
Jul 28 04:11:42.120385 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.120044    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-multus-conf-dir\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.121396 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.120059    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/27f106d2-f008-49e7-8928-623a0f10c6c9-etc-sysctl-conf\") pod \"tuned-4scq5\" (UID: \"27f106d2-f008-49e7-8928-623a0f10c6c9\") " pod="openshift-cluster-node-tuning-operator/tuned-4scq5"
Jul 28 04:11:42.121396 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.120076    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/068d70ee-9efb-49a5-94cb-2df7591a4652-hosts-file\") pod \"node-resolver-tlsvx\" (UID: \"068d70ee-9efb-49a5-94cb-2df7591a4652\") " pod="openshift-dns/node-resolver-tlsvx"
Jul 28 04:11:42.121396 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.120108    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/93788c64-611b-40ff-be6c-8dcea894c08d-cni-binary-copy\") pod \"multus-additional-cni-plugins-fk4hq\" (UID: \"93788c64-611b-40ff-be6c-8dcea894c08d\") " pod="openshift-multus/multus-additional-cni-plugins-fk4hq"
Jul 28 04:11:42.121396 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.120125    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-etc-kubernetes\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.121396 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.120143    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/27f106d2-f008-49e7-8928-623a0f10c6c9-etc-systemd\") pod \"tuned-4scq5\" (UID: \"27f106d2-f008-49e7-8928-623a0f10c6c9\") " pod="openshift-cluster-node-tuning-operator/tuned-4scq5"
Jul 28 04:11:42.121396 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.120171    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/accccd5b-603e-49d5-b9fc-b328dcb7e959-dbus\") pod \"global-pull-secret-syncer-z2gn6\" (UID: \"accccd5b-603e-49d5-b9fc-b328dcb7e959\") " pod="kube-system/global-pull-secret-syncer-z2gn6"
Jul 28 04:11:42.121396 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.120194    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-multus-cni-dir\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.121396 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.120211    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-host-var-lib-cni-bin\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.121396 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.120228    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n528z\" (UniqueName: \"kubernetes.io/projected/e6107883-62a6-4a82-9e04-ed46057a6b0e-kube-api-access-n528z\") pod \"network-metrics-daemon-l6p44\" (UID: \"e6107883-62a6-4a82-9e04-ed46057a6b0e\") " pod="openshift-multus/network-metrics-daemon-l6p44"
Jul 28 04:11:42.121396 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.120252    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/27f106d2-f008-49e7-8928-623a0f10c6c9-run\") pod \"tuned-4scq5\" (UID: \"27f106d2-f008-49e7-8928-623a0f10c6c9\") " pod="openshift-cluster-node-tuning-operator/tuned-4scq5"
Jul 28 04:11:42.121396 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.120277    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-cnibin\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.121396 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.120293    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-host-run-k8s-cni-cncf-io\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.121396 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.120312    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-host-run-multus-certs\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.121396 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.120328    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27f106d2-f008-49e7-8928-623a0f10c6c9-tmp\") pod \"tuned-4scq5\" (UID: \"27f106d2-f008-49e7-8928-623a0f10c6c9\") " pod="openshift-cluster-node-tuning-operator/tuned-4scq5"
Jul 28 04:11:42.121396 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.120346    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93788c64-611b-40ff-be6c-8dcea894c08d-system-cni-dir\") pod \"multus-additional-cni-plugins-fk4hq\" (UID: \"93788c64-611b-40ff-be6c-8dcea894c08d\") " pod="openshift-multus/multus-additional-cni-plugins-fk4hq"
Jul 28 04:11:42.121396 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.120371    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/93788c64-611b-40ff-be6c-8dcea894c08d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fk4hq\" (UID: \"93788c64-611b-40ff-be6c-8dcea894c08d\") " pod="openshift-multus/multus-additional-cni-plugins-fk4hq"
Jul 28 04:11:42.121396 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.120390    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6mtg\" (UniqueName: \"kubernetes.io/projected/93788c64-611b-40ff-be6c-8dcea894c08d-kube-api-access-g6mtg\") pod \"multus-additional-cni-plugins-fk4hq\" (UID: \"93788c64-611b-40ff-be6c-8dcea894c08d\") " pod="openshift-multus/multus-additional-cni-plugins-fk4hq"
Jul 28 04:11:42.122259 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.120406    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/accccd5b-603e-49d5-b9fc-b328dcb7e959-kubelet-config\") pod \"global-pull-secret-syncer-z2gn6\" (UID: \"accccd5b-603e-49d5-b9fc-b328dcb7e959\") " pod="kube-system/global-pull-secret-syncer-z2gn6"
Jul 28 04:11:42.122259 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.120423    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-host-var-lib-cni-multus\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.122259 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.120445    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-host-var-lib-kubelet\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.122259 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.120505    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-host-var-lib-kubelet\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.122259 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.120520    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/93788c64-611b-40ff-be6c-8dcea894c08d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fk4hq\" (UID: \"93788c64-611b-40ff-be6c-8dcea894c08d\") " pod="openshift-multus/multus-additional-cni-plugins-fk4hq"
Jul 28 04:11:42.122259 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.120590    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-cni-binary-copy\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.122259 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.120601    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/27f106d2-f008-49e7-8928-623a0f10c6c9-etc-sysconfig\") pod \"tuned-4scq5\" (UID: \"27f106d2-f008-49e7-8928-623a0f10c6c9\") " pod="openshift-cluster-node-tuning-operator/tuned-4scq5"
Jul 28 04:11:42.122259 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.120666    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5b11f875-be7a-4fe1-8ca4-40bb7725889b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7v87k\" (UID: \"5b11f875-be7a-4fe1-8ca4-40bb7725889b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7v87k"
Jul 28 04:11:42.122259 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.120667    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-multus-socket-dir-parent\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.122259 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.120726    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/27f106d2-f008-49e7-8928-623a0f10c6c9-host\") pod \"tuned-4scq5\" (UID: \"27f106d2-f008-49e7-8928-623a0f10c6c9\") " pod="openshift-cluster-node-tuning-operator/tuned-4scq5"
Jul 28 04:11:42.122259 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.120739    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27f106d2-f008-49e7-8928-623a0f10c6c9-var-lib-kubelet\") pod \"tuned-4scq5\" (UID: \"27f106d2-f008-49e7-8928-623a0f10c6c9\") " pod="openshift-cluster-node-tuning-operator/tuned-4scq5"
Jul 28 04:11:42.122259 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:42.120744    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 04:11:42.122259 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:42.120799    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6107883-62a6-4a82-9e04-ed46057a6b0e-metrics-certs podName:e6107883-62a6-4a82-9e04-ed46057a6b0e nodeName:}" failed. No retries permitted until 2026-07-28 04:11:42.620783394 +0000 UTC m=+3.010959409 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e6107883-62a6-4a82-9e04-ed46057a6b0e-metrics-certs") pod "network-metrics-daemon-l6p44" (UID: "e6107883-62a6-4a82-9e04-ed46057a6b0e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 04:11:42.122259 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.120837    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/27f106d2-f008-49e7-8928-623a0f10c6c9-etc-sysctl-d\") pod \"tuned-4scq5\" (UID: \"27f106d2-f008-49e7-8928-623a0f10c6c9\") " pod="openshift-cluster-node-tuning-operator/tuned-4scq5"
Jul 28 04:11:42.122259 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.119674    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/93788c64-611b-40ff-be6c-8dcea894c08d-os-release\") pod \"multus-additional-cni-plugins-fk4hq\" (UID: \"93788c64-611b-40ff-be6c-8dcea894c08d\") " pod="openshift-multus/multus-additional-cni-plugins-fk4hq"
Jul 28 04:11:42.122259 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.120906    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-host-run-netns\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.122259 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.120963    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-system-cni-dir\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.123072 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.120974    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5b11f875-be7a-4fe1-8ca4-40bb7725889b-registration-dir\") pod \"aws-ebs-csi-driver-node-7v87k\" (UID: \"5b11f875-be7a-4fe1-8ca4-40bb7725889b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7v87k"
Jul 28 04:11:42.123072 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.121016    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5b11f875-be7a-4fe1-8ca4-40bb7725889b-device-dir\") pod \"aws-ebs-csi-driver-node-7v87k\" (UID: \"5b11f875-be7a-4fe1-8ca4-40bb7725889b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7v87k"
Jul 28 04:11:42.123072 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.121031    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27f106d2-f008-49e7-8928-623a0f10c6c9-sys\") pod \"tuned-4scq5\" (UID: \"27f106d2-f008-49e7-8928-623a0f10c6c9\") " pod="openshift-cluster-node-tuning-operator/tuned-4scq5"
Jul 28 04:11:42.123072 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.121042    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-os-release\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.123072 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.121080    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5b11f875-be7a-4fe1-8ca4-40bb7725889b-sys-fs\") pod \"aws-ebs-csi-driver-node-7v87k\" (UID: \"5b11f875-be7a-4fe1-8ca4-40bb7725889b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7v87k"
Jul 28 04:11:42.123072 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.121097    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/27f106d2-f008-49e7-8928-623a0f10c6c9-run\") pod \"tuned-4scq5\" (UID: \"27f106d2-f008-49e7-8928-623a0f10c6c9\") " pod="openshift-cluster-node-tuning-operator/tuned-4scq5"
Jul 28 04:11:42.123072 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.121125    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-cnibin\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.123072 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.121157    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-host-run-k8s-cni-cncf-io\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.123072 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:42.121173    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 04:11:42.123072 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.121185    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-host-run-multus-certs\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.123072 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:42.121227    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/accccd5b-603e-49d5-b9fc-b328dcb7e959-original-pull-secret podName:accccd5b-603e-49d5-b9fc-b328dcb7e959 nodeName:}" failed. No retries permitted until 2026-07-28 04:11:42.621208844 +0000 UTC m=+3.011384845 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/accccd5b-603e-49d5-b9fc-b328dcb7e959-original-pull-secret") pod "global-pull-secret-syncer-z2gn6" (UID: "accccd5b-603e-49d5-b9fc-b328dcb7e959") : object "kube-system"/"original-pull-secret" not registered
Jul 28 04:11:42.123072 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.121267    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-multus-conf-dir\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.123072 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.121336    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/068d70ee-9efb-49a5-94cb-2df7591a4652-tmp-dir\") pod \"node-resolver-tlsvx\" (UID: \"068d70ee-9efb-49a5-94cb-2df7591a4652\") " pod="openshift-dns/node-resolver-tlsvx"
Jul 28 04:11:42.123072 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.121407    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/27f106d2-f008-49e7-8928-623a0f10c6c9-etc-sysctl-conf\") pod \"tuned-4scq5\" (UID: \"27f106d2-f008-49e7-8928-623a0f10c6c9\") " pod="openshift-cluster-node-tuning-operator/tuned-4scq5"
Jul 28 04:11:42.123072 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.121456    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/accccd5b-603e-49d5-b9fc-b328dcb7e959-kubelet-config\") pod \"global-pull-secret-syncer-z2gn6\" (UID: \"accccd5b-603e-49d5-b9fc-b328dcb7e959\") " pod="kube-system/global-pull-secret-syncer-z2gn6"
Jul 28 04:11:42.123072 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.121475    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-multus-cni-dir\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.123072 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.121488    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/accccd5b-603e-49d5-b9fc-b328dcb7e959-dbus\") pod \"global-pull-secret-syncer-z2gn6\" (UID: \"accccd5b-603e-49d5-b9fc-b328dcb7e959\") " pod="kube-system/global-pull-secret-syncer-z2gn6"
Jul 28 04:11:42.123072 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.121507    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93788c64-611b-40ff-be6c-8dcea894c08d-system-cni-dir\") pod \"multus-additional-cni-plugins-fk4hq\" (UID: \"93788c64-611b-40ff-be6c-8dcea894c08d\") " pod="openshift-multus/multus-additional-cni-plugins-fk4hq"
Jul 28 04:11:42.123720 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.121529    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-host-var-lib-cni-bin\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.123720 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.121695    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-etc-kubernetes\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.123720 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.121775    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/27f106d2-f008-49e7-8928-623a0f10c6c9-etc-systemd\") pod \"tuned-4scq5\" (UID: \"27f106d2-f008-49e7-8928-623a0f10c6c9\") " pod="openshift-cluster-node-tuning-operator/tuned-4scq5"
Jul 28 04:11:42.123720 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.121831    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/068d70ee-9efb-49a5-94cb-2df7591a4652-hosts-file\") pod \"node-resolver-tlsvx\" (UID: \"068d70ee-9efb-49a5-94cb-2df7591a4652\") " pod="openshift-dns/node-resolver-tlsvx"
Jul 28 04:11:42.123720 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.121885    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-host-var-lib-cni-multus\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.123720 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.121993    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/93788c64-611b-40ff-be6c-8dcea894c08d-cni-binary-copy\") pod \"multus-additional-cni-plugins-fk4hq\" (UID: \"93788c64-611b-40ff-be6c-8dcea894c08d\") " pod="openshift-multus/multus-additional-cni-plugins-fk4hq"
Jul 28 04:11:42.123720 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.122319    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2c4c0738-fb7c-4610-aab6-7cd762ebaf83-konnectivity-ca\") pod \"konnectivity-agent-vfg8j\" (UID: \"2c4c0738-fb7c-4610-aab6-7cd762ebaf83\") " pod="kube-system/konnectivity-agent-vfg8j"
Jul 28 04:11:42.123720 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.122574    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/93788c64-611b-40ff-be6c-8dcea894c08d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fk4hq\" (UID: \"93788c64-611b-40ff-be6c-8dcea894c08d\") " pod="openshift-multus/multus-additional-cni-plugins-fk4hq"
Jul 28 04:11:42.123720 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.122616    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/27f106d2-f008-49e7-8928-623a0f10c6c9-etc-tuned\") pod \"tuned-4scq5\" (UID: \"27f106d2-f008-49e7-8928-623a0f10c6c9\") " pod="openshift-cluster-node-tuning-operator/tuned-4scq5"
Jul 28 04:11:42.123720 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.122784    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2c4c0738-fb7c-4610-aab6-7cd762ebaf83-agent-certs\") pod \"konnectivity-agent-vfg8j\" (UID: \"2c4c0738-fb7c-4610-aab6-7cd762ebaf83\") " pod="kube-system/konnectivity-agent-vfg8j"
Jul 28 04:11:42.125727 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.125702    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27f106d2-f008-49e7-8928-623a0f10c6c9-tmp\") pod \"tuned-4scq5\" (UID: \"27f106d2-f008-49e7-8928-623a0f10c6c9\") " pod="openshift-cluster-node-tuning-operator/tuned-4scq5"
Jul 28 04:11:42.130331 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:42.129787    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 04:11:42.130331 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:42.129813    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 04:11:42.130331 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:42.129836    2566 projected.go:196] Error preparing data for projected volume kube-api-access-zmnvx for pod openshift-network-diagnostics/network-check-target-qgqqn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 04:11:42.130331 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:42.129924    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33a30d7b-76be-42e9-ae95-6779c5770b15-kube-api-access-zmnvx podName:33a30d7b-76be-42e9-ae95-6779c5770b15 nodeName:}" failed. No retries permitted until 2026-07-28 04:11:42.629905937 +0000 UTC m=+3.020081955 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-zmnvx" (UniqueName: "kubernetes.io/projected/33a30d7b-76be-42e9-ae95-6779c5770b15-kube-api-access-zmnvx") pod "network-check-target-qgqqn" (UID: "33a30d7b-76be-42e9-ae95-6779c5770b15") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 04:11:42.133176 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.132772    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6mtg\" (UniqueName: \"kubernetes.io/projected/93788c64-611b-40ff-be6c-8dcea894c08d-kube-api-access-g6mtg\") pod \"multus-additional-cni-plugins-fk4hq\" (UID: \"93788c64-611b-40ff-be6c-8dcea894c08d\") " pod="openshift-multus/multus-additional-cni-plugins-fk4hq"
Jul 28 04:11:42.133176 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.133126    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddl8c\" (UniqueName: \"kubernetes.io/projected/27f106d2-f008-49e7-8928-623a0f10c6c9-kube-api-access-ddl8c\") pod \"tuned-4scq5\" (UID: \"27f106d2-f008-49e7-8928-623a0f10c6c9\") " pod="openshift-cluster-node-tuning-operator/tuned-4scq5"
Jul 28 04:11:42.133965 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.133908    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjphl\" (UniqueName: \"kubernetes.io/projected/068d70ee-9efb-49a5-94cb-2df7591a4652-kube-api-access-rjphl\") pod \"node-resolver-tlsvx\" (UID: \"068d70ee-9efb-49a5-94cb-2df7591a4652\") " pod="openshift-dns/node-resolver-tlsvx"
Jul 28 04:11:42.134096 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.134074    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbsfz\" (UniqueName: \"kubernetes.io/projected/5b11f875-be7a-4fe1-8ca4-40bb7725889b-kube-api-access-bbsfz\") pod \"aws-ebs-csi-driver-node-7v87k\" (UID: \"5b11f875-be7a-4fe1-8ca4-40bb7725889b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7v87k"
Jul 28 04:11:42.134522 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.134440    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptvxq\" (UniqueName: \"kubernetes.io/projected/aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67-kube-api-access-ptvxq\") pod \"multus-jxpf8\" (UID: \"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67\") " pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.135192 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.135149    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n528z\" (UniqueName: \"kubernetes.io/projected/e6107883-62a6-4a82-9e04-ed46057a6b0e-kube-api-access-n528z\") pod \"network-metrics-daemon-l6p44\" (UID: \"e6107883-62a6-4a82-9e04-ed46057a6b0e\") " pod="openshift-multus/network-metrics-daemon-l6p44"
Jul 28 04:11:42.216708 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.216670    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-t6nfz"
Jul 28 04:11:42.226205 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.226182    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 04:11:42.230364 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.230339    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-l2b28"
Jul 28 04:11:42.238400 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.238378    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:11:42.247302 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.247282    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7v87k"
Jul 28 04:11:42.255371 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.255345    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-4scq5"
Jul 28 04:11:42.264150 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.264130    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tlsvx"
Jul 28 04:11:42.273131 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.273110    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fk4hq"
Jul 28 04:11:42.282017 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.281998    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vfg8j"
Jul 28 04:11:42.288002 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.287983    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jxpf8"
Jul 28 04:11:42.624944 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.624903    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6107883-62a6-4a82-9e04-ed46057a6b0e-metrics-certs\") pod \"network-metrics-daemon-l6p44\" (UID: \"e6107883-62a6-4a82-9e04-ed46057a6b0e\") " pod="openshift-multus/network-metrics-daemon-l6p44"
Jul 28 04:11:42.625156 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.624993    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/accccd5b-603e-49d5-b9fc-b328dcb7e959-original-pull-secret\") pod \"global-pull-secret-syncer-z2gn6\" (UID: \"accccd5b-603e-49d5-b9fc-b328dcb7e959\") " pod="kube-system/global-pull-secret-syncer-z2gn6"
Jul 28 04:11:42.625156 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:42.625112    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 04:11:42.625274 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:42.625171    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/accccd5b-603e-49d5-b9fc-b328dcb7e959-original-pull-secret podName:accccd5b-603e-49d5-b9fc-b328dcb7e959 nodeName:}" failed. No retries permitted until 2026-07-28 04:11:43.62515462 +0000 UTC m=+4.015330620 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/accccd5b-603e-49d5-b9fc-b328dcb7e959-original-pull-secret") pod "global-pull-secret-syncer-z2gn6" (UID: "accccd5b-603e-49d5-b9fc-b328dcb7e959") : object "kube-system"/"original-pull-secret" not registered
Jul 28 04:11:42.625274 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:42.625109    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 04:11:42.625397 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:42.625306    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6107883-62a6-4a82-9e04-ed46057a6b0e-metrics-certs podName:e6107883-62a6-4a82-9e04-ed46057a6b0e nodeName:}" failed. No retries permitted until 2026-07-28 04:11:43.625281994 +0000 UTC m=+4.015457999 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e6107883-62a6-4a82-9e04-ed46057a6b0e-metrics-certs") pod "network-metrics-daemon-l6p44" (UID: "e6107883-62a6-4a82-9e04-ed46057a6b0e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 04:11:42.725672 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.725629    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmnvx\" (UniqueName: \"kubernetes.io/projected/33a30d7b-76be-42e9-ae95-6779c5770b15-kube-api-access-zmnvx\") pod \"network-check-target-qgqqn\" (UID: \"33a30d7b-76be-42e9-ae95-6779c5770b15\") " pod="openshift-network-diagnostics/network-check-target-qgqqn"
Jul 28 04:11:42.725872 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:42.725831    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 04:11:42.725956 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:42.725879    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 04:11:42.725956 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:42.725891    2566 projected.go:196] Error preparing data for projected volume kube-api-access-zmnvx for pod openshift-network-diagnostics/network-check-target-qgqqn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 04:11:42.726042 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:42.725960    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33a30d7b-76be-42e9-ae95-6779c5770b15-kube-api-access-zmnvx podName:33a30d7b-76be-42e9-ae95-6779c5770b15 nodeName:}" failed. No retries permitted until 2026-07-28 04:11:43.725940043 +0000 UTC m=+4.116116046 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-zmnvx" (UniqueName: "kubernetes.io/projected/33a30d7b-76be-42e9-ae95-6779c5770b15-kube-api-access-zmnvx") pod "network-check-target-qgqqn" (UID: "33a30d7b-76be-42e9-ae95-6779c5770b15") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 04:11:42.948985 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.948878    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 04:06:42 +0000 UTC" deadline="2028-03-29 09:03:50.518543309 +0000 UTC"
Jul 28 04:11:42.948985 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.948921    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14644h52m7.569627365s"
Jul 28 04:11:42.960600 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:42.960568    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 04:11:43.223086 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:43.222927    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod068d70ee_9efb_49a5_94cb_2df7591a4652.slice/crio-2923f30e696c8396ea456faa71c94087e57a25e31538b4a4c74526ab23b36998 WatchSource:0}: Error finding container 2923f30e696c8396ea456faa71c94087e57a25e31538b4a4c74526ab23b36998: Status 404 returned error can't find the container with id 2923f30e696c8396ea456faa71c94087e57a25e31538b4a4c74526ab23b36998
Jul 28 04:11:43.225310 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:43.225288    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaaa4008_a3fc_4b57_a3f7_eeb6bf9c9a67.slice/crio-3fc7355f307d2f95732747a32d71294e41234814617b86499242ae3dfc5556a6 WatchSource:0}: Error finding container 3fc7355f307d2f95732747a32d71294e41234814617b86499242ae3dfc5556a6: Status 404 returned error can't find the container with id 3fc7355f307d2f95732747a32d71294e41234814617b86499242ae3dfc5556a6
Jul 28 04:11:43.246651 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:43.246632    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93788c64_611b_40ff_be6c_8dcea894c08d.slice/crio-2322f64fdc2f06c81daef09884afbe5e5b39b1943e046d2eb0c929e9d82d62be WatchSource:0}: Error finding container 2322f64fdc2f06c81daef09884afbe5e5b39b1943e046d2eb0c929e9d82d62be: Status 404 returned error can't find the container with id 2322f64fdc2f06c81daef09884afbe5e5b39b1943e046d2eb0c929e9d82d62be
Jul 28 04:11:43.247172 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:43.247148    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c4c0738_fb7c_4610_aab6_7cd762ebaf83.slice/crio-ab26d151870a04774607f411b85586b9e1d072ebbce7a6319c179ea6cccd729a WatchSource:0}: Error finding container ab26d151870a04774607f411b85586b9e1d072ebbce7a6319c179ea6cccd729a: Status 404 returned error can't find the container with id ab26d151870a04774607f411b85586b9e1d072ebbce7a6319c179ea6cccd729a
Jul 28 04:11:43.249259 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:43.249223    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a6783c1_279b_45e7_acf2_b56e2e9f1908.slice/crio-5b93b51450dfd30f65d4316668fdeefeb33ab45c2adcc5a39793015ae22634e3 WatchSource:0}: Error finding container 5b93b51450dfd30f65d4316668fdeefeb33ab45c2adcc5a39793015ae22634e3: Status 404 returned error can't find the container with id 5b93b51450dfd30f65d4316668fdeefeb33ab45c2adcc5a39793015ae22634e3
Jul 28 04:11:43.250508 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:43.250486    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b11f875_be7a_4fe1_8ca4_40bb7725889b.slice/crio-04027d294dd7991b3f87c17b65016f29da1c1b869be94ab212c86d5266b3f59d WatchSource:0}: Error finding container 04027d294dd7991b3f87c17b65016f29da1c1b869be94ab212c86d5266b3f59d: Status 404 returned error can't find the container with id 04027d294dd7991b3f87c17b65016f29da1c1b869be94ab212c86d5266b3f59d
Jul 28 04:11:43.251033 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:43.250862    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f106d2_f008_49e7_8928_623a0f10c6c9.slice/crio-a97bcbf071ea67dbf6cb88799641683291dd2e281c8319651efacb6678a5968f WatchSource:0}: Error finding container a97bcbf071ea67dbf6cb88799641683291dd2e281c8319651efacb6678a5968f: Status 404 returned error can't find the container with id a97bcbf071ea67dbf6cb88799641683291dd2e281c8319651efacb6678a5968f
Jul 28 04:11:43.252208 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:11:43.252176    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabf055f7_9730_480a_897d_6fa35cd77b61.slice/crio-efc7fd0c8b1eec80e3cbd2f05eebc6af2bfaffe3c7c36dbb231e9dd472c2055c WatchSource:0}: Error finding container efc7fd0c8b1eec80e3cbd2f05eebc6af2bfaffe3c7c36dbb231e9dd472c2055c: Status 404 returned error can't find the container with id efc7fd0c8b1eec80e3cbd2f05eebc6af2bfaffe3c7c36dbb231e9dd472c2055c
Jul 28 04:11:43.633195 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:43.632948    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/accccd5b-603e-49d5-b9fc-b328dcb7e959-original-pull-secret\") pod \"global-pull-secret-syncer-z2gn6\" (UID: \"accccd5b-603e-49d5-b9fc-b328dcb7e959\") " pod="kube-system/global-pull-secret-syncer-z2gn6"
Jul 28 04:11:43.633384 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:43.633253    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6107883-62a6-4a82-9e04-ed46057a6b0e-metrics-certs\") pod \"network-metrics-daemon-l6p44\" (UID: \"e6107883-62a6-4a82-9e04-ed46057a6b0e\") " pod="openshift-multus/network-metrics-daemon-l6p44"
Jul 28 04:11:43.633453 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:43.633096    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 04:11:43.633508 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:43.633472    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/accccd5b-603e-49d5-b9fc-b328dcb7e959-original-pull-secret podName:accccd5b-603e-49d5-b9fc-b328dcb7e959 nodeName:}" failed. No retries permitted until 2026-07-28 04:11:45.633450168 +0000 UTC m=+6.023626175 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/accccd5b-603e-49d5-b9fc-b328dcb7e959-original-pull-secret") pod "global-pull-secret-syncer-z2gn6" (UID: "accccd5b-603e-49d5-b9fc-b328dcb7e959") : object "kube-system"/"original-pull-secret" not registered
Jul 28 04:11:43.633508 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:43.633400    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 04:11:43.633624 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:43.633593    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6107883-62a6-4a82-9e04-ed46057a6b0e-metrics-certs podName:e6107883-62a6-4a82-9e04-ed46057a6b0e nodeName:}" failed. No retries permitted until 2026-07-28 04:11:45.633570261 +0000 UTC m=+6.023746265 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e6107883-62a6-4a82-9e04-ed46057a6b0e-metrics-certs") pod "network-metrics-daemon-l6p44" (UID: "e6107883-62a6-4a82-9e04-ed46057a6b0e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 04:11:43.734365 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:43.734242    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmnvx\" (UniqueName: \"kubernetes.io/projected/33a30d7b-76be-42e9-ae95-6779c5770b15-kube-api-access-zmnvx\") pod \"network-check-target-qgqqn\" (UID: \"33a30d7b-76be-42e9-ae95-6779c5770b15\") " pod="openshift-network-diagnostics/network-check-target-qgqqn"
Jul 28 04:11:43.734525 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:43.734435    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 04:11:43.734525 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:43.734455    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 04:11:43.734525 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:43.734468    2566 projected.go:196] Error preparing data for projected volume kube-api-access-zmnvx for pod openshift-network-diagnostics/network-check-target-qgqqn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 04:11:43.734708 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:43.734530    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33a30d7b-76be-42e9-ae95-6779c5770b15-kube-api-access-zmnvx podName:33a30d7b-76be-42e9-ae95-6779c5770b15 nodeName:}" failed. No retries permitted until 2026-07-28 04:11:45.734512656 +0000 UTC m=+6.124688680 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-zmnvx" (UniqueName: "kubernetes.io/projected/33a30d7b-76be-42e9-ae95-6779c5770b15-kube-api-access-zmnvx") pod "network-check-target-qgqqn" (UID: "33a30d7b-76be-42e9-ae95-6779c5770b15") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 04:11:43.949789 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:43.949645    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 04:06:42 +0000 UTC" deadline="2028-03-28 09:53:19.789390809 +0000 UTC"
Jul 28 04:11:43.949789 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:43.949788    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14621h41m35.839609289s"
Jul 28 04:11:44.084489 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:44.083718    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6p44"
Jul 28 04:11:44.084489 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:44.083879    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6p44" podUID="e6107883-62a6-4a82-9e04-ed46057a6b0e"
Jul 28 04:11:44.084489 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:44.084300    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qgqqn"
Jul 28 04:11:44.084489 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:44.084438    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qgqqn" podUID="33a30d7b-76be-42e9-ae95-6779c5770b15"
Jul 28 04:11:44.085130 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:44.084970    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2gn6"
Jul 28 04:11:44.085130 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:44.085076    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2gn6" podUID="accccd5b-603e-49d5-b9fc-b328dcb7e959"
Jul 28 04:11:44.098635 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:44.098520    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4scq5" event={"ID":"27f106d2-f008-49e7-8928-623a0f10c6c9","Type":"ContainerStarted","Data":"a97bcbf071ea67dbf6cb88799641683291dd2e281c8319651efacb6678a5968f"}
Jul 28 04:11:44.107825 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:44.107760    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l2b28" event={"ID":"1a6783c1-279b-45e7-acf2-b56e2e9f1908","Type":"ContainerStarted","Data":"5b93b51450dfd30f65d4316668fdeefeb33ab45c2adcc5a39793015ae22634e3"}
Jul 28 04:11:44.121787 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:44.121720    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jxpf8" event={"ID":"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67","Type":"ContainerStarted","Data":"3fc7355f307d2f95732747a32d71294e41234814617b86499242ae3dfc5556a6"}
Jul 28 04:11:44.127724 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:44.127693    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t6nfz" event={"ID":"8cf586fe-2e9b-474b-8fa6-3ca6317e2b31","Type":"ContainerStarted","Data":"2796f8e22d0398c7388d8a84edf0ca6d6a52bc3ede0052d09f5eede379f86f6d"}
Jul 28 04:11:44.154097 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:44.153429    2566 generic.go:356] "Generic (PLEG): container finished" podID="bb2b2cec0e2ae932bf2eeb6eb1815049" containerID="e6630c3de215e61043e616cf4c1b646860a35f392017c93d8312b47e1f7bb8cd" exitCode=0
Jul 28 04:11:44.154097 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:44.153544    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-187.ec2.internal" event={"ID":"bb2b2cec0e2ae932bf2eeb6eb1815049","Type":"ContainerDied","Data":"e6630c3de215e61043e616cf4c1b646860a35f392017c93d8312b47e1f7bb8cd"}
Jul 28 04:11:44.167147 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:44.167115    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8xswd" event={"ID":"abf055f7-9730-480a-897d-6fa35cd77b61","Type":"ContainerStarted","Data":"efc7fd0c8b1eec80e3cbd2f05eebc6af2bfaffe3c7c36dbb231e9dd472c2055c"}
Jul 28 04:11:44.187869 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:44.187829    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7v87k" event={"ID":"5b11f875-be7a-4fe1-8ca4-40bb7725889b","Type":"ContainerStarted","Data":"04027d294dd7991b3f87c17b65016f29da1c1b869be94ab212c86d5266b3f59d"}
Jul 28 04:11:44.216445 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:44.216407    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vfg8j" event={"ID":"2c4c0738-fb7c-4610-aab6-7cd762ebaf83","Type":"ContainerStarted","Data":"ab26d151870a04774607f411b85586b9e1d072ebbce7a6319c179ea6cccd729a"}
Jul 28 04:11:44.227328 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:44.227293    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tlsvx" event={"ID":"068d70ee-9efb-49a5-94cb-2df7591a4652","Type":"ContainerStarted","Data":"2923f30e696c8396ea456faa71c94087e57a25e31538b4a4c74526ab23b36998"}
Jul 28 04:11:44.239565 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:44.239530    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-187.ec2.internal" event={"ID":"c9650e72cdb5110eb610920a5f32b652","Type":"ContainerStarted","Data":"60fdf04940cded9987ac71ea17a1d59736c91f2eb679628d1a34ce4ae1db8e58"}
Jul 28 04:11:44.247999 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:44.247852    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fk4hq" event={"ID":"93788c64-611b-40ff-be6c-8dcea894c08d","Type":"ContainerStarted","Data":"2322f64fdc2f06c81daef09884afbe5e5b39b1943e046d2eb0c929e9d82d62be"}
Jul 28 04:11:45.266967 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:45.266832    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-187.ec2.internal" event={"ID":"bb2b2cec0e2ae932bf2eeb6eb1815049","Type":"ContainerStarted","Data":"80aef4531a80744058cb10450c218afaf7391a25eb67c75d14d5e5b7171afa55"}
Jul 28 04:11:45.283134 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:45.283056    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-187.ec2.internal" podStartSLOduration=3.283032432 podStartE2EDuration="3.283032432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 04:11:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:11:44.253887918 +0000 UTC m=+4.644063949" watchObservedRunningTime="2026-07-28 04:11:45.283032432 +0000 UTC m=+5.673208459"
Jul 28 04:11:45.652046 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:45.651947    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/accccd5b-603e-49d5-b9fc-b328dcb7e959-original-pull-secret\") pod \"global-pull-secret-syncer-z2gn6\" (UID: \"accccd5b-603e-49d5-b9fc-b328dcb7e959\") " pod="kube-system/global-pull-secret-syncer-z2gn6"
Jul 28 04:11:45.652046 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:45.652015    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6107883-62a6-4a82-9e04-ed46057a6b0e-metrics-certs\") pod \"network-metrics-daemon-l6p44\" (UID: \"e6107883-62a6-4a82-9e04-ed46057a6b0e\") " pod="openshift-multus/network-metrics-daemon-l6p44"
Jul 28 04:11:45.652349 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:45.652170    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 04:11:45.652349 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:45.652237    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6107883-62a6-4a82-9e04-ed46057a6b0e-metrics-certs podName:e6107883-62a6-4a82-9e04-ed46057a6b0e nodeName:}" failed. No retries permitted until 2026-07-28 04:11:49.652218534 +0000 UTC m=+10.042394549 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e6107883-62a6-4a82-9e04-ed46057a6b0e-metrics-certs") pod "network-metrics-daemon-l6p44" (UID: "e6107883-62a6-4a82-9e04-ed46057a6b0e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 04:11:45.652349 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:45.652272    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 04:11:45.652349 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:45.652337    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/accccd5b-603e-49d5-b9fc-b328dcb7e959-original-pull-secret podName:accccd5b-603e-49d5-b9fc-b328dcb7e959 nodeName:}" failed. No retries permitted until 2026-07-28 04:11:49.65231957 +0000 UTC m=+10.042495573 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/accccd5b-603e-49d5-b9fc-b328dcb7e959-original-pull-secret") pod "global-pull-secret-syncer-z2gn6" (UID: "accccd5b-603e-49d5-b9fc-b328dcb7e959") : object "kube-system"/"original-pull-secret" not registered
Jul 28 04:11:45.753211 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:45.753122    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmnvx\" (UniqueName: \"kubernetes.io/projected/33a30d7b-76be-42e9-ae95-6779c5770b15-kube-api-access-zmnvx\") pod \"network-check-target-qgqqn\" (UID: \"33a30d7b-76be-42e9-ae95-6779c5770b15\") " pod="openshift-network-diagnostics/network-check-target-qgqqn"
Jul 28 04:11:45.753427 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:45.753359    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 04:11:45.753427 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:45.753383    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 04:11:45.753427 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:45.753396    2566 projected.go:196] Error preparing data for projected volume kube-api-access-zmnvx for pod openshift-network-diagnostics/network-check-target-qgqqn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 04:11:45.753734 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:45.753461    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33a30d7b-76be-42e9-ae95-6779c5770b15-kube-api-access-zmnvx podName:33a30d7b-76be-42e9-ae95-6779c5770b15 nodeName:}" failed. No retries permitted until 2026-07-28 04:11:49.753441403 +0000 UTC m=+10.143617418 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-zmnvx" (UniqueName: "kubernetes.io/projected/33a30d7b-76be-42e9-ae95-6779c5770b15-kube-api-access-zmnvx") pod "network-check-target-qgqqn" (UID: "33a30d7b-76be-42e9-ae95-6779c5770b15") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 04:11:46.084571 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:46.084027    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2gn6"
Jul 28 04:11:46.084571 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:46.084178    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2gn6" podUID="accccd5b-603e-49d5-b9fc-b328dcb7e959"
Jul 28 04:11:46.084571 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:46.084240    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qgqqn"
Jul 28 04:11:46.084571 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:46.084359    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qgqqn" podUID="33a30d7b-76be-42e9-ae95-6779c5770b15"
Jul 28 04:11:46.084571 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:46.084416    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6p44"
Jul 28 04:11:46.084571 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:46.084505    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6p44" podUID="e6107883-62a6-4a82-9e04-ed46057a6b0e"
Jul 28 04:11:48.085606 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:48.085075    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2gn6"
Jul 28 04:11:48.085606 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:48.085216    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2gn6" podUID="accccd5b-603e-49d5-b9fc-b328dcb7e959"
Jul 28 04:11:48.085606 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:48.085391    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qgqqn"
Jul 28 04:11:48.085606 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:48.085478    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qgqqn" podUID="33a30d7b-76be-42e9-ae95-6779c5770b15"
Jul 28 04:11:48.085606 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:48.085549    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6p44"
Jul 28 04:11:48.086338 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:48.085632    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6p44" podUID="e6107883-62a6-4a82-9e04-ed46057a6b0e"
Jul 28 04:11:49.687472 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:49.687229    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/accccd5b-603e-49d5-b9fc-b328dcb7e959-original-pull-secret\") pod \"global-pull-secret-syncer-z2gn6\" (UID: \"accccd5b-603e-49d5-b9fc-b328dcb7e959\") " pod="kube-system/global-pull-secret-syncer-z2gn6"
Jul 28 04:11:49.687472 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:49.687307    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6107883-62a6-4a82-9e04-ed46057a6b0e-metrics-certs\") pod \"network-metrics-daemon-l6p44\" (UID: \"e6107883-62a6-4a82-9e04-ed46057a6b0e\") " pod="openshift-multus/network-metrics-daemon-l6p44"
Jul 28 04:11:49.687472 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:49.687397    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 04:11:49.687472 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:49.687417    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 04:11:49.687472 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:49.687473    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/accccd5b-603e-49d5-b9fc-b328dcb7e959-original-pull-secret podName:accccd5b-603e-49d5-b9fc-b328dcb7e959 nodeName:}" failed. No retries permitted until 2026-07-28 04:11:57.687449886 +0000 UTC m=+18.077625893 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/accccd5b-603e-49d5-b9fc-b328dcb7e959-original-pull-secret") pod "global-pull-secret-syncer-z2gn6" (UID: "accccd5b-603e-49d5-b9fc-b328dcb7e959") : object "kube-system"/"original-pull-secret" not registered
Jul 28 04:11:49.688214 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:49.687493    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6107883-62a6-4a82-9e04-ed46057a6b0e-metrics-certs podName:e6107883-62a6-4a82-9e04-ed46057a6b0e nodeName:}" failed. No retries permitted until 2026-07-28 04:11:57.687483185 +0000 UTC m=+18.077659186 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e6107883-62a6-4a82-9e04-ed46057a6b0e-metrics-certs") pod "network-metrics-daemon-l6p44" (UID: "e6107883-62a6-4a82-9e04-ed46057a6b0e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 04:11:49.788809 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:49.788118    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmnvx\" (UniqueName: \"kubernetes.io/projected/33a30d7b-76be-42e9-ae95-6779c5770b15-kube-api-access-zmnvx\") pod \"network-check-target-qgqqn\" (UID: \"33a30d7b-76be-42e9-ae95-6779c5770b15\") " pod="openshift-network-diagnostics/network-check-target-qgqqn"
Jul 28 04:11:49.788809 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:49.788335    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 04:11:49.788809 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:49.788375    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 04:11:49.788809 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:49.788390    2566 projected.go:196] Error preparing data for projected volume kube-api-access-zmnvx for pod openshift-network-diagnostics/network-check-target-qgqqn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 04:11:49.788809 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:49.788452    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33a30d7b-76be-42e9-ae95-6779c5770b15-kube-api-access-zmnvx podName:33a30d7b-76be-42e9-ae95-6779c5770b15 nodeName:}" failed. No retries permitted until 2026-07-28 04:11:57.788432288 +0000 UTC m=+18.178608301 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-zmnvx" (UniqueName: "kubernetes.io/projected/33a30d7b-76be-42e9-ae95-6779c5770b15-kube-api-access-zmnvx") pod "network-check-target-qgqqn" (UID: "33a30d7b-76be-42e9-ae95-6779c5770b15") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 04:11:50.083705 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:50.083664    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qgqqn"
Jul 28 04:11:50.083927 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:50.083813    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qgqqn" podUID="33a30d7b-76be-42e9-ae95-6779c5770b15"
Jul 28 04:11:50.084290 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:50.084268    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2gn6"
Jul 28 04:11:50.084627 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:50.084383    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2gn6" podUID="accccd5b-603e-49d5-b9fc-b328dcb7e959"
Jul 28 04:11:50.084627 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:50.084450    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6p44"
Jul 28 04:11:50.084627 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:50.084588    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6p44" podUID="e6107883-62a6-4a82-9e04-ed46057a6b0e"
Jul 28 04:11:52.084388 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:52.084298    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qgqqn"
Jul 28 04:11:52.084388 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:52.084346    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6p44"
Jul 28 04:11:52.084948 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:52.084298    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2gn6"
Jul 28 04:11:52.084948 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:52.084439    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qgqqn" podUID="33a30d7b-76be-42e9-ae95-6779c5770b15"
Jul 28 04:11:52.084948 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:52.084510    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2gn6" podUID="accccd5b-603e-49d5-b9fc-b328dcb7e959"
Jul 28 04:11:52.084948 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:52.084592    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6p44" podUID="e6107883-62a6-4a82-9e04-ed46057a6b0e"
Jul 28 04:11:54.082287 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:54.082255    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6p44"
Jul 28 04:11:54.082741 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:54.082262    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2gn6"
Jul 28 04:11:54.082741 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:54.082370    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6p44" podUID="e6107883-62a6-4a82-9e04-ed46057a6b0e"
Jul 28 04:11:54.082741 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:54.082265    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qgqqn"
Jul 28 04:11:54.082741 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:54.082469    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2gn6" podUID="accccd5b-603e-49d5-b9fc-b328dcb7e959"
Jul 28 04:11:54.082741 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:54.082595    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qgqqn" podUID="33a30d7b-76be-42e9-ae95-6779c5770b15"
Jul 28 04:11:56.083018 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:56.082981    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2gn6"
Jul 28 04:11:56.083504 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:56.083087    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2gn6" podUID="accccd5b-603e-49d5-b9fc-b328dcb7e959"
Jul 28 04:11:56.083504 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:56.082986    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qgqqn"
Jul 28 04:11:56.083504 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:56.083126    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6p44"
Jul 28 04:11:56.083504 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:56.083214    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qgqqn" podUID="33a30d7b-76be-42e9-ae95-6779c5770b15"
Jul 28 04:11:56.083504 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:56.083312    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6p44" podUID="e6107883-62a6-4a82-9e04-ed46057a6b0e"
Jul 28 04:11:57.753890 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:57.753853    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/accccd5b-603e-49d5-b9fc-b328dcb7e959-original-pull-secret\") pod \"global-pull-secret-syncer-z2gn6\" (UID: \"accccd5b-603e-49d5-b9fc-b328dcb7e959\") " pod="kube-system/global-pull-secret-syncer-z2gn6"
Jul 28 04:11:57.754412 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:57.753929    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6107883-62a6-4a82-9e04-ed46057a6b0e-metrics-certs\") pod \"network-metrics-daemon-l6p44\" (UID: \"e6107883-62a6-4a82-9e04-ed46057a6b0e\") " pod="openshift-multus/network-metrics-daemon-l6p44"
Jul 28 04:11:57.754412 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:57.753932    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 04:11:57.754412 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:57.754034    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/accccd5b-603e-49d5-b9fc-b328dcb7e959-original-pull-secret podName:accccd5b-603e-49d5-b9fc-b328dcb7e959 nodeName:}" failed. No retries permitted until 2026-07-28 04:12:13.754019287 +0000 UTC m=+34.144195287 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/accccd5b-603e-49d5-b9fc-b328dcb7e959-original-pull-secret") pod "global-pull-secret-syncer-z2gn6" (UID: "accccd5b-603e-49d5-b9fc-b328dcb7e959") : object "kube-system"/"original-pull-secret" not registered
Jul 28 04:11:57.754412 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:57.754069    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 04:11:57.754412 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:57.754143    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6107883-62a6-4a82-9e04-ed46057a6b0e-metrics-certs podName:e6107883-62a6-4a82-9e04-ed46057a6b0e nodeName:}" failed. No retries permitted until 2026-07-28 04:12:13.754126035 +0000 UTC m=+34.144302034 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e6107883-62a6-4a82-9e04-ed46057a6b0e-metrics-certs") pod "network-metrics-daemon-l6p44" (UID: "e6107883-62a6-4a82-9e04-ed46057a6b0e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 04:11:57.854386 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:57.854348    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmnvx\" (UniqueName: \"kubernetes.io/projected/33a30d7b-76be-42e9-ae95-6779c5770b15-kube-api-access-zmnvx\") pod \"network-check-target-qgqqn\" (UID: \"33a30d7b-76be-42e9-ae95-6779c5770b15\") " pod="openshift-network-diagnostics/network-check-target-qgqqn"
Jul 28 04:11:57.854565 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:57.854499    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 04:11:57.854565 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:57.854526    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 04:11:57.854565 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:57.854538    2566 projected.go:196] Error preparing data for projected volume kube-api-access-zmnvx for pod openshift-network-diagnostics/network-check-target-qgqqn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 04:11:57.854722 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:57.854600    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33a30d7b-76be-42e9-ae95-6779c5770b15-kube-api-access-zmnvx podName:33a30d7b-76be-42e9-ae95-6779c5770b15 nodeName:}" failed. No retries permitted until 2026-07-28 04:12:13.854582431 +0000 UTC m=+34.244758432 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-zmnvx" (UniqueName: "kubernetes.io/projected/33a30d7b-76be-42e9-ae95-6779c5770b15-kube-api-access-zmnvx") pod "network-check-target-qgqqn" (UID: "33a30d7b-76be-42e9-ae95-6779c5770b15") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 04:11:58.081981 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:58.081949    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qgqqn"
Jul 28 04:11:58.082142 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:58.082055    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2gn6"
Jul 28 04:11:58.082142 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:58.082064    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qgqqn" podUID="33a30d7b-76be-42e9-ae95-6779c5770b15"
Jul 28 04:11:58.082224 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:58.082152    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2gn6" podUID="accccd5b-603e-49d5-b9fc-b328dcb7e959"
Jul 28 04:11:58.082224 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:11:58.082192    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6p44"
Jul 28 04:11:58.082285 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:11:58.082255    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6p44" podUID="e6107883-62a6-4a82-9e04-ed46057a6b0e"
Jul 28 04:12:00.082786 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:00.082748    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qgqqn"
Jul 28 04:12:00.083252 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:00.082866    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2gn6"
Jul 28 04:12:00.083252 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:00.082901    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6p44"
Jul 28 04:12:00.083252 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:00.082866    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qgqqn" podUID="33a30d7b-76be-42e9-ae95-6779c5770b15"
Jul 28 04:12:00.083252 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:00.082991    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2gn6" podUID="accccd5b-603e-49d5-b9fc-b328dcb7e959"
Jul 28 04:12:00.083252 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:00.083076    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6p44" podUID="e6107883-62a6-4a82-9e04-ed46057a6b0e"
Jul 28 04:12:02.082379 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:02.082341    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qgqqn"
Jul 28 04:12:02.082882 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:02.082341    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2gn6"
Jul 28 04:12:02.082882 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:02.082475    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qgqqn" podUID="33a30d7b-76be-42e9-ae95-6779c5770b15"
Jul 28 04:12:02.082882 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:02.082489    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6p44"
Jul 28 04:12:02.082882 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:02.082634    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2gn6" podUID="accccd5b-603e-49d5-b9fc-b328dcb7e959"
Jul 28 04:12:02.082882 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:02.082823    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6p44" podUID="e6107883-62a6-4a82-9e04-ed46057a6b0e"
Jul 28 04:12:03.310334 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:03.309963    2566 generic.go:356] "Generic (PLEG): container finished" podID="93788c64-611b-40ff-be6c-8dcea894c08d" containerID="8c955643cecc31a4fe809c6a625795f8eee738fd6c1dacd44b680318d0ec2b21" exitCode=0
Jul 28 04:12:03.311266 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:03.310044    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fk4hq" event={"ID":"93788c64-611b-40ff-be6c-8dcea894c08d","Type":"ContainerDied","Data":"8c955643cecc31a4fe809c6a625795f8eee738fd6c1dacd44b680318d0ec2b21"}
Jul 28 04:12:03.312068 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:03.312047    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-4scq5" event={"ID":"27f106d2-f008-49e7-8928-623a0f10c6c9","Type":"ContainerStarted","Data":"43890e2783d89a07f739f37e539787299be60dd07babd6d153be2e4a12e4dc76"}
Jul 28 04:12:03.313562 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:03.313541    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jxpf8" event={"ID":"aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67","Type":"ContainerStarted","Data":"e3ce5b239b945719d094a01663443c4fe1571dbbc8c4d0232e030f6a1490b549"}
Jul 28 04:12:03.314938 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:03.314913    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-t6nfz" event={"ID":"8cf586fe-2e9b-474b-8fa6-3ca6317e2b31","Type":"ContainerStarted","Data":"abd13646287a47652548fb0c9c026fb25dab5df38acd21c68e5d9c7850bb8f44"}
Jul 28 04:12:03.317725 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:03.317698    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8xswd" event={"ID":"abf055f7-9730-480a-897d-6fa35cd77b61","Type":"ContainerStarted","Data":"c3e2aafe49dc0067223cab0b62968d4b947ab73f3ac2c15f17804e5e87b70255"}
Jul 28 04:12:03.317831 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:03.317760    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8xswd" event={"ID":"abf055f7-9730-480a-897d-6fa35cd77b61","Type":"ContainerStarted","Data":"8505b5d0e2101fb4ec45d8f387380c1fd1aafb9db51c8e67785d0c2ea1d02457"}
Jul 28 04:12:03.317831 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:03.317794    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8xswd" event={"ID":"abf055f7-9730-480a-897d-6fa35cd77b61","Type":"ContainerStarted","Data":"0d895f40f2c5ad38ea75071d03686ec65dd0221e5d22074064101e017ad655b5"}
Jul 28 04:12:03.317831 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:03.317823    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8xswd" event={"ID":"abf055f7-9730-480a-897d-6fa35cd77b61","Type":"ContainerStarted","Data":"fa87f2c9c65279078fe79a63183e87dc114aa8542dbc432c810962c860136b73"}
Jul 28 04:12:03.317993 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:03.317877    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8xswd" event={"ID":"abf055f7-9730-480a-897d-6fa35cd77b61","Type":"ContainerStarted","Data":"aea35267496a7681ea1eae378300aea3a68d40b73d4ea4c1bf6fb042c2057556"}
Jul 28 04:12:03.317993 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:03.317908    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8xswd" event={"ID":"abf055f7-9730-480a-897d-6fa35cd77b61","Type":"ContainerStarted","Data":"d302b1729688b5ff8aba3b39f88809f0a2700a92d86be2607cb58f6204ede11d"}
Jul 28 04:12:03.319088 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:03.319068    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7v87k" event={"ID":"5b11f875-be7a-4fe1-8ca4-40bb7725889b","Type":"ContainerStarted","Data":"75fd72361ec7ea47219fa8cd9ac4d14c3f21babaf602a503a6caeba9d21edd9e"}
Jul 28 04:12:03.320469 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:03.320448    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vfg8j" event={"ID":"2c4c0738-fb7c-4610-aab6-7cd762ebaf83","Type":"ContainerStarted","Data":"d74efc2fba064690d3bfe1b91af2cd4ffa7955b22d595e2a3cc9e5b429782dc5"}
Jul 28 04:12:03.321652 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:03.321634    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tlsvx" event={"ID":"068d70ee-9efb-49a5-94cb-2df7591a4652","Type":"ContainerStarted","Data":"bd8adaa59fa6ed6997d1064ec5a6df9da406955a01b92c8bfdef0c8f1805593a"}
Jul 28 04:12:03.331818 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:03.331780    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-187.ec2.internal" podStartSLOduration=21.331768845 podStartE2EDuration="21.331768845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 04:11:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:11:45.283524347 +0000 UTC m=+5.673700376" watchObservedRunningTime="2026-07-28 04:12:03.331768845 +0000 UTC m=+23.721944865"
Jul 28 04:12:03.344921 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:03.344873    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-4scq5" podStartSLOduration=3.252450613 podStartE2EDuration="22.344855028s" totalImagesPullingTime="19.092404415s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 04:11:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:12:03.344569045 +0000 UTC m=+23.734745065" watchObservedRunningTime="2026-07-28 04:12:03.344855028 +0000 UTC m=+23.735031050"
Jul 28 04:12:03.357715 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:03.357671    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-tlsvx" podStartSLOduration=3.258777076 podStartE2EDuration="22.35765371s" totalImagesPullingTime="19.098876634s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 04:11:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:12:03.357575312 +0000 UTC m=+23.747751347" watchObservedRunningTime="2026-07-28 04:12:03.35765371 +0000 UTC m=+23.747829732"
Jul 28 04:12:03.376861 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:03.376809    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-t6nfz" podStartSLOduration=3.254764347 podStartE2EDuration="22.376795033s" totalImagesPullingTime="19.122030686s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 04:11:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:12:03.376363434 +0000 UTC m=+23.766539458" watchObservedRunningTime="2026-07-28 04:12:03.376795033 +0000 UTC m=+23.766971053"
Jul 28 04:12:03.391919 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:03.391880    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vfg8j" podStartSLOduration=11.216754095 podStartE2EDuration="22.391867476s" totalImagesPullingTime="11.175113381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:11:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:12:03.391482421 +0000 UTC m=+23.781658468" watchObservedRunningTime="2026-07-28 04:12:03.391867476 +0000 UTC m=+23.782043499"
Jul 28 04:12:03.406952 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:03.406908    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-jxpf8" podStartSLOduration=3.283848715 podStartE2EDuration="22.406893782s" totalImagesPullingTime="19.123045067s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 04:11:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:12:03.406489439 +0000 UTC m=+23.796665460" watchObservedRunningTime="2026-07-28 04:12:03.406893782 +0000 UTC m=+23.797069802"
Jul 28 04:12:03.707443 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:03.707407    2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 04:12:03.975584 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:03.975378    2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T04:12:03.707437501Z","UUID":"3ddced4c-780a-4cc3-b65d-dac5ec948979","Handler":null,"Name":"","Endpoint":""}
Jul 28 04:12:03.978176 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:03.978154    2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 04:12:03.978317 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:03.978201    2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 04:12:04.082994 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:04.082952    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2gn6"
Jul 28 04:12:04.082994 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:04.082996    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qgqqn"
Jul 28 04:12:04.083264 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:04.082956    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6p44"
Jul 28 04:12:04.083264 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:04.083094    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2gn6" podUID="accccd5b-603e-49d5-b9fc-b328dcb7e959"
Jul 28 04:12:04.083264 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:04.083187    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qgqqn" podUID="33a30d7b-76be-42e9-ae95-6779c5770b15"
Jul 28 04:12:04.083383 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:04.083285    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6p44" podUID="e6107883-62a6-4a82-9e04-ed46057a6b0e"
Jul 28 04:12:04.327165 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:04.327118    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7v87k" event={"ID":"5b11f875-be7a-4fe1-8ca4-40bb7725889b","Type":"ContainerStarted","Data":"9a0a8200a041cb6d992f8021b60d17ba936e22b22002cf878345c4a9818c10d0"}
Jul 28 04:12:04.329035 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:04.328931    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-l2b28" event={"ID":"1a6783c1-279b-45e7-acf2-b56e2e9f1908","Type":"ContainerStarted","Data":"c358fe7596316f4fdfb7f88dbd2e50f15fe90758fa0751636df6ed9ca889867e"}
Jul 28 04:12:04.342617 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:04.342568    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-l2b28" podStartSLOduration=4.247954873 podStartE2EDuration="23.34255219s" totalImagesPullingTime="19.094597317s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 04:11:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:12:04.341892585 +0000 UTC m=+24.732068608" watchObservedRunningTime="2026-07-28 04:12:04.34255219 +0000 UTC m=+24.732728211"
Jul 28 04:12:05.107913 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:05.107882    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vfg8j"
Jul 28 04:12:05.335426 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:05.335380    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8xswd" event={"ID":"abf055f7-9730-480a-897d-6fa35cd77b61","Type":"ContainerStarted","Data":"af7f1971057d9e7d5429d8532f7de6cd31ed5f464a12ac8f0e9f3b1eba93d596"}
Jul 28 04:12:05.338059 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:05.337976    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7v87k" event={"ID":"5b11f875-be7a-4fe1-8ca4-40bb7725889b","Type":"ContainerStarted","Data":"42418f96dd66f2bdd8d839fa25aef05ced9d900e57246177ce27a1f1cabfbc83"}
Jul 28 04:12:05.356254 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:05.356118    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7v87k" podStartSLOduration=2.953134616 podStartE2EDuration="24.356095798s" totalImagesPullingTime="21.402961182s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 04:11:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:12:05.355002864 +0000 UTC m=+25.745178884" watchObservedRunningTime="2026-07-28 04:12:05.356095798 +0000 UTC m=+25.746271820"
Jul 28 04:12:06.082403 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:06.082373    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6p44"
Jul 28 04:12:06.082403 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:06.082408    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qgqqn"
Jul 28 04:12:06.082632 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:06.082373    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2gn6"
Jul 28 04:12:06.082632 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:06.082511    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6p44" podUID="e6107883-62a6-4a82-9e04-ed46057a6b0e"
Jul 28 04:12:06.082632 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:06.082590    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2gn6" podUID="accccd5b-603e-49d5-b9fc-b328dcb7e959"
Jul 28 04:12:06.082776 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:06.082667    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qgqqn" podUID="33a30d7b-76be-42e9-ae95-6779c5770b15"
Jul 28 04:12:06.749945 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:06.749726    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vfg8j"
Jul 28 04:12:06.750398 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:06.750373    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vfg8j"
Jul 28 04:12:07.343184 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:07.343152    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vfg8j"
Jul 28 04:12:08.082803 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:08.082771    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qgqqn"
Jul 28 04:12:08.083608 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:08.082771    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2gn6"
Jul 28 04:12:08.083608 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:08.082909    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qgqqn" podUID="33a30d7b-76be-42e9-ae95-6779c5770b15"
Jul 28 04:12:08.083608 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:08.082771    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6p44"
Jul 28 04:12:08.083608 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:08.082972    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2gn6" podUID="accccd5b-603e-49d5-b9fc-b328dcb7e959"
Jul 28 04:12:08.083608 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:08.083052    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6p44" podUID="e6107883-62a6-4a82-9e04-ed46057a6b0e"
Jul 28 04:12:08.348427 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:08.348090    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8xswd" event={"ID":"abf055f7-9730-480a-897d-6fa35cd77b61","Type":"ContainerStarted","Data":"f1343d64385f2dff67beb7f77bd7b14e98d708a7be92e05f14c20e67e1ba6b3e"}
Jul 28 04:12:08.348427 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:08.348405    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:12:08.349888 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:08.349864    2566 generic.go:356] "Generic (PLEG): container finished" podID="93788c64-611b-40ff-be6c-8dcea894c08d" containerID="bd51883acff354570905573233e3204e4ea74cf437534fa80286622f0ce5ddf7" exitCode=0
Jul 28 04:12:08.350002 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:08.349931    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fk4hq" event={"ID":"93788c64-611b-40ff-be6c-8dcea894c08d","Type":"ContainerDied","Data":"bd51883acff354570905573233e3204e4ea74cf437534fa80286622f0ce5ddf7"}
Jul 28 04:12:08.364588 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:08.364569    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:12:08.373256 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:08.373218    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8xswd" podStartSLOduration=8.211413189 podStartE2EDuration="27.37320112s" totalImagesPullingTime="19.161787931s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 04:11:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:12:08.371371548 +0000 UTC m=+28.761547568" watchObservedRunningTime="2026-07-28 04:12:08.37320112 +0000 UTC m=+28.763377141"
Jul 28 04:12:09.352972 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:09.352943    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:12:09.353437 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:09.353069    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:12:09.367072 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:09.367049    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:12:09.531308 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:09.531279    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-z2gn6"]
Jul 28 04:12:09.531448 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:09.531390    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2gn6"
Jul 28 04:12:09.531513 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:09.531476    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2gn6" podUID="accccd5b-603e-49d5-b9fc-b328dcb7e959"
Jul 28 04:12:09.534007 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:09.533904    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qgqqn"]
Jul 28 04:12:09.534007 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:09.534005    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qgqqn"
Jul 28 04:12:09.534188 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:09.534099    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qgqqn" podUID="33a30d7b-76be-42e9-ae95-6779c5770b15"
Jul 28 04:12:09.534895 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:09.534874    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l6p44"]
Jul 28 04:12:09.534995 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:09.534972    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6p44"
Jul 28 04:12:09.535094 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:09.535073    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6p44" podUID="e6107883-62a6-4a82-9e04-ed46057a6b0e"
Jul 28 04:12:10.356502 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:10.356468    2566 generic.go:356] "Generic (PLEG): container finished" podID="93788c64-611b-40ff-be6c-8dcea894c08d" containerID="15fa56dd43cde54d1ba3c21ce3fd41b5f311cc98a61a970ed893a2aeb80430dd" exitCode=0
Jul 28 04:12:10.356942 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:10.356566    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fk4hq" event={"ID":"93788c64-611b-40ff-be6c-8dcea894c08d","Type":"ContainerDied","Data":"15fa56dd43cde54d1ba3c21ce3fd41b5f311cc98a61a970ed893a2aeb80430dd"}
Jul 28 04:12:11.082299 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:11.082267    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6p44"
Jul 28 04:12:11.082299 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:11.082291    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2gn6"
Jul 28 04:12:11.082506 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:11.082291    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qgqqn"
Jul 28 04:12:11.082506 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:11.082383    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6p44" podUID="e6107883-62a6-4a82-9e04-ed46057a6b0e"
Jul 28 04:12:11.082506 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:11.082448    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2gn6" podUID="accccd5b-603e-49d5-b9fc-b328dcb7e959"
Jul 28 04:12:11.082615 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:11.082504    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qgqqn" podUID="33a30d7b-76be-42e9-ae95-6779c5770b15"
Jul 28 04:12:12.362723 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:12.362686    2566 generic.go:356] "Generic (PLEG): container finished" podID="93788c64-611b-40ff-be6c-8dcea894c08d" containerID="d9beacb9ec7081cd1be988e01fa2b6f65541a937cae49bc8556966ae8256a1e7" exitCode=0
Jul 28 04:12:12.363077 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:12.362750    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fk4hq" event={"ID":"93788c64-611b-40ff-be6c-8dcea894c08d","Type":"ContainerDied","Data":"d9beacb9ec7081cd1be988e01fa2b6f65541a937cae49bc8556966ae8256a1e7"}
Jul 28 04:12:13.082687 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:13.082453    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qgqqn"
Jul 28 04:12:13.082893 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:13.082494    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2gn6"
Jul 28 04:12:13.082893 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:13.082786    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qgqqn" podUID="33a30d7b-76be-42e9-ae95-6779c5770b15"
Jul 28 04:12:13.082893 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:13.082507    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6p44"
Jul 28 04:12:13.082893 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:13.082860    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z2gn6" podUID="accccd5b-603e-49d5-b9fc-b328dcb7e959"
Jul 28 04:12:13.083116 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:13.083058    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l6p44" podUID="e6107883-62a6-4a82-9e04-ed46057a6b0e"
Jul 28 04:12:13.784446 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:13.784415    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/accccd5b-603e-49d5-b9fc-b328dcb7e959-original-pull-secret\") pod \"global-pull-secret-syncer-z2gn6\" (UID: \"accccd5b-603e-49d5-b9fc-b328dcb7e959\") " pod="kube-system/global-pull-secret-syncer-z2gn6"
Jul 28 04:12:13.784822 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:13.784482    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6107883-62a6-4a82-9e04-ed46057a6b0e-metrics-certs\") pod \"network-metrics-daemon-l6p44\" (UID: \"e6107883-62a6-4a82-9e04-ed46057a6b0e\") " pod="openshift-multus/network-metrics-daemon-l6p44"
Jul 28 04:12:13.784822 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:13.784588    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 04:12:13.784822 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:13.784622    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 04:12:13.784822 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:13.784695    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6107883-62a6-4a82-9e04-ed46057a6b0e-metrics-certs podName:e6107883-62a6-4a82-9e04-ed46057a6b0e nodeName:}" failed. No retries permitted until 2026-07-28 04:12:45.78467256 +0000 UTC m=+66.174848578 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e6107883-62a6-4a82-9e04-ed46057a6b0e-metrics-certs") pod "network-metrics-daemon-l6p44" (UID: "e6107883-62a6-4a82-9e04-ed46057a6b0e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 04:12:13.784822 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:13.784715    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/accccd5b-603e-49d5-b9fc-b328dcb7e959-original-pull-secret podName:accccd5b-603e-49d5-b9fc-b328dcb7e959 nodeName:}" failed. No retries permitted until 2026-07-28 04:12:45.784706277 +0000 UTC m=+66.174882275 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/accccd5b-603e-49d5-b9fc-b328dcb7e959-original-pull-secret") pod "global-pull-secret-syncer-z2gn6" (UID: "accccd5b-603e-49d5-b9fc-b328dcb7e959") : object "kube-system"/"original-pull-secret" not registered
Jul 28 04:12:13.885269 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:13.885224    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmnvx\" (UniqueName: \"kubernetes.io/projected/33a30d7b-76be-42e9-ae95-6779c5770b15-kube-api-access-zmnvx\") pod \"network-check-target-qgqqn\" (UID: \"33a30d7b-76be-42e9-ae95-6779c5770b15\") " pod="openshift-network-diagnostics/network-check-target-qgqqn"
Jul 28 04:12:13.885422 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:13.885399    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 04:12:13.885422 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:13.885418    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 04:12:13.885556 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:13.885427    2566 projected.go:196] Error preparing data for projected volume kube-api-access-zmnvx for pod openshift-network-diagnostics/network-check-target-qgqqn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 04:12:13.885556 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:13.885485    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/33a30d7b-76be-42e9-ae95-6779c5770b15-kube-api-access-zmnvx podName:33a30d7b-76be-42e9-ae95-6779c5770b15 nodeName:}" failed. No retries permitted until 2026-07-28 04:12:45.885472343 +0000 UTC m=+66.275648364 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-zmnvx" (UniqueName: "kubernetes.io/projected/33a30d7b-76be-42e9-ae95-6779c5770b15-kube-api-access-zmnvx") pod "network-check-target-qgqqn" (UID: "33a30d7b-76be-42e9-ae95-6779c5770b15") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 04:12:15.010738 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.010708    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-187.ec2.internal" event="NodeReady"
Jul 28 04:12:15.011149 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.010878    2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 04:12:15.042155 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.042125    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-jxd2j"]
Jul 28 04:12:15.047250 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.047229    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-66ff9f7dd7-45ggn"]
Jul 28 04:12:15.047385 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.047370    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-jxd2j"
Jul 28 04:12:15.049929 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.049898    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-c8c86\""
Jul 28 04:12:15.049929 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.049921    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 28 04:12:15.050106 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.049937    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 28 04:12:15.050671 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.050654    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn"
Jul 28 04:12:15.053385 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.053178    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 28 04:12:15.053385 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.053266    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-s6545\""
Jul 28 04:12:15.053921 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.053730    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 28 04:12:15.054032 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.053962    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 28 04:12:15.054604 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.054559    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-jxd2j"]
Jul 28 04:12:15.058171 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.058152    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-z8gxf"]
Jul 28 04:12:15.059430 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.059399    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 28 04:12:15.061492 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.061475    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66ff9f7dd7-45ggn"]
Jul 28 04:12:15.061574 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.061550    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-z8gxf"
Jul 28 04:12:15.063611 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.063598    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-m9zx6"]
Jul 28 04:12:15.064182 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.064164    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 04:12:15.064274 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.064205    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 04:12:15.064449 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.064431    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-69gm8\""
Jul 28 04:12:15.064807 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.064785    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 04:12:15.067094 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.067076    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-m9zx6"
Jul 28 04:12:15.069697 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.069680    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7zgg8\""
Jul 28 04:12:15.069803 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.069741    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 04:12:15.069954 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.069939    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 04:12:15.072483 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.072464    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-z8gxf"]
Jul 28 04:12:15.075682 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.075631    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-m9zx6"]
Jul 28 04:12:15.081997 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.081969    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qgqqn"
Jul 28 04:12:15.082526 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.082490    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6p44"
Jul 28 04:12:15.082626 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.082534    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2gn6"
Jul 28 04:12:15.085069 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.085050    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 04:12:15.085161 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.085071    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 04:12:15.085823 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.085802    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 04:12:15.085823 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.085811    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 04:12:15.086090 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.086076    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x5zc2\""
Jul 28 04:12:15.086231 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.086215    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9nvrd\""
Jul 28 04:12:15.194330 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.194294    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-installation-pull-secrets\") pod \"image-registry-66ff9f7dd7-45ggn\" (UID: \"55cd6e46-995c-4857-b7b7-c90e4e4ff93d\") " pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn"
Jul 28 04:12:15.194511 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.194384    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkvg5\" (UniqueName: \"kubernetes.io/projected/9cb032cd-6003-4249-a65d-95d48fea8a02-kube-api-access-wkvg5\") pod \"ingress-canary-z8gxf\" (UID: \"9cb032cd-6003-4249-a65d-95d48fea8a02\") " pod="openshift-ingress-canary/ingress-canary-z8gxf"
Jul 28 04:12:15.194511 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.194439    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8rxc\" (UniqueName: \"kubernetes.io/projected/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-kube-api-access-v8rxc\") pod \"image-registry-66ff9f7dd7-45ggn\" (UID: \"55cd6e46-995c-4857-b7b7-c90e4e4ff93d\") " pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn"
Jul 28 04:12:15.194511 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.194473    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c58f3317-e522-41a3-848f-906af440a23b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-jxd2j\" (UID: \"c58f3317-e522-41a3-848f-906af440a23b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-jxd2j"
Jul 28 04:12:15.194668 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.194549    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a02a02e4-e2b1-45fd-aeda-12fa4e29a996-metrics-tls\") pod \"dns-default-m9zx6\" (UID: \"a02a02e4-e2b1-45fd-aeda-12fa4e29a996\") " pod="openshift-dns/dns-default-m9zx6"
Jul 28 04:12:15.194668 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.194586    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-ca-trust-extracted\") pod \"image-registry-66ff9f7dd7-45ggn\" (UID: \"55cd6e46-995c-4857-b7b7-c90e4e4ff93d\") " pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn"
Jul 28 04:12:15.194668 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.194613    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-registry-certificates\") pod \"image-registry-66ff9f7dd7-45ggn\" (UID: \"55cd6e46-995c-4857-b7b7-c90e4e4ff93d\") " pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn"
Jul 28 04:12:15.194668 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.194641    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a02a02e4-e2b1-45fd-aeda-12fa4e29a996-tmp-dir\") pod \"dns-default-m9zx6\" (UID: \"a02a02e4-e2b1-45fd-aeda-12fa4e29a996\") " pod="openshift-dns/dns-default-m9zx6"
Jul 28 04:12:15.194951 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.194726    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a02a02e4-e2b1-45fd-aeda-12fa4e29a996-config-volume\") pod \"dns-default-m9zx6\" (UID: \"a02a02e4-e2b1-45fd-aeda-12fa4e29a996\") " pod="openshift-dns/dns-default-m9zx6"
Jul 28 04:12:15.194951 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.194764    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66dfd\" (UniqueName: \"kubernetes.io/projected/a02a02e4-e2b1-45fd-aeda-12fa4e29a996-kube-api-access-66dfd\") pod \"dns-default-m9zx6\" (UID: \"a02a02e4-e2b1-45fd-aeda-12fa4e29a996\") " pod="openshift-dns/dns-default-m9zx6"
Jul 28 04:12:15.194951 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.194787    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c58f3317-e522-41a3-848f-906af440a23b-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-jxd2j\" (UID: \"c58f3317-e522-41a3-848f-906af440a23b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-jxd2j"
Jul 28 04:12:15.194951 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.194815    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-image-registry-private-configuration\") pod \"image-registry-66ff9f7dd7-45ggn\" (UID: \"55cd6e46-995c-4857-b7b7-c90e4e4ff93d\") " pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn"
Jul 28 04:12:15.194951 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.194907    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-trusted-ca\") pod \"image-registry-66ff9f7dd7-45ggn\" (UID: \"55cd6e46-995c-4857-b7b7-c90e4e4ff93d\") " pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn"
Jul 28 04:12:15.195164 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.194955    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-bound-sa-token\") pod \"image-registry-66ff9f7dd7-45ggn\" (UID: \"55cd6e46-995c-4857-b7b7-c90e4e4ff93d\") " pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn"
Jul 28 04:12:15.195164 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.194988    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-registry-tls\") pod \"image-registry-66ff9f7dd7-45ggn\" (UID: \"55cd6e46-995c-4857-b7b7-c90e4e4ff93d\") " pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn"
Jul 28 04:12:15.195164 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.195014    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9cb032cd-6003-4249-a65d-95d48fea8a02-cert\") pod \"ingress-canary-z8gxf\" (UID: \"9cb032cd-6003-4249-a65d-95d48fea8a02\") " pod="openshift-ingress-canary/ingress-canary-z8gxf"
Jul 28 04:12:15.295654 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.295620    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-bound-sa-token\") pod \"image-registry-66ff9f7dd7-45ggn\" (UID: \"55cd6e46-995c-4857-b7b7-c90e4e4ff93d\") " pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn"
Jul 28 04:12:15.295877 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.295668    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-registry-tls\") pod \"image-registry-66ff9f7dd7-45ggn\" (UID: \"55cd6e46-995c-4857-b7b7-c90e4e4ff93d\") " pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn"
Jul 28 04:12:15.295877 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.295694    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9cb032cd-6003-4249-a65d-95d48fea8a02-cert\") pod \"ingress-canary-z8gxf\" (UID: \"9cb032cd-6003-4249-a65d-95d48fea8a02\") " pod="openshift-ingress-canary/ingress-canary-z8gxf"
Jul 28 04:12:15.295877 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.295733    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-installation-pull-secrets\") pod \"image-registry-66ff9f7dd7-45ggn\" (UID: \"55cd6e46-995c-4857-b7b7-c90e4e4ff93d\") " pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn"
Jul 28 04:12:15.295877 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:15.295818    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 04:12:15.295877 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:15.295861    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66ff9f7dd7-45ggn: secret "image-registry-tls" not found
Jul 28 04:12:15.296116 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:15.295929    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-registry-tls podName:55cd6e46-995c-4857-b7b7-c90e4e4ff93d nodeName:}" failed. No retries permitted until 2026-07-28 04:12:15.7959062 +0000 UTC m=+36.186082222 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-registry-tls") pod "image-registry-66ff9f7dd7-45ggn" (UID: "55cd6e46-995c-4857-b7b7-c90e4e4ff93d") : secret "image-registry-tls" not found
Jul 28 04:12:15.296280 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.296250    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkvg5\" (UniqueName: \"kubernetes.io/projected/9cb032cd-6003-4249-a65d-95d48fea8a02-kube-api-access-wkvg5\") pod \"ingress-canary-z8gxf\" (UID: \"9cb032cd-6003-4249-a65d-95d48fea8a02\") " pod="openshift-ingress-canary/ingress-canary-z8gxf"
Jul 28 04:12:15.296355 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.296304    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8rxc\" (UniqueName: \"kubernetes.io/projected/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-kube-api-access-v8rxc\") pod \"image-registry-66ff9f7dd7-45ggn\" (UID: \"55cd6e46-995c-4857-b7b7-c90e4e4ff93d\") " pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn"
Jul 28 04:12:15.296355 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.296330    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c58f3317-e522-41a3-848f-906af440a23b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-jxd2j\" (UID: \"c58f3317-e522-41a3-848f-906af440a23b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-jxd2j"
Jul 28 04:12:15.296556 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.296372    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a02a02e4-e2b1-45fd-aeda-12fa4e29a996-metrics-tls\") pod \"dns-default-m9zx6\" (UID: \"a02a02e4-e2b1-45fd-aeda-12fa4e29a996\") " pod="openshift-dns/dns-default-m9zx6"
Jul 28 04:12:15.296556 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.296401    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-ca-trust-extracted\") pod \"image-registry-66ff9f7dd7-45ggn\" (UID: \"55cd6e46-995c-4857-b7b7-c90e4e4ff93d\") " pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn"
Jul 28 04:12:15.296556 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.296425    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-registry-certificates\") pod \"image-registry-66ff9f7dd7-45ggn\" (UID: \"55cd6e46-995c-4857-b7b7-c90e4e4ff93d\") " pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn"
Jul 28 04:12:15.296556 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.296446    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a02a02e4-e2b1-45fd-aeda-12fa4e29a996-tmp-dir\") pod \"dns-default-m9zx6\" (UID: \"a02a02e4-e2b1-45fd-aeda-12fa4e29a996\") " pod="openshift-dns/dns-default-m9zx6"
Jul 28 04:12:15.296556 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:15.296490    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 04:12:15.296556 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.296496    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a02a02e4-e2b1-45fd-aeda-12fa4e29a996-config-volume\") pod \"dns-default-m9zx6\" (UID: \"a02a02e4-e2b1-45fd-aeda-12fa4e29a996\") " pod="openshift-dns/dns-default-m9zx6"
Jul 28 04:12:15.296556 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:15.296537    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a02a02e4-e2b1-45fd-aeda-12fa4e29a996-metrics-tls podName:a02a02e4-e2b1-45fd-aeda-12fa4e29a996 nodeName:}" failed. No retries permitted until 2026-07-28 04:12:15.796521992 +0000 UTC m=+36.186697991 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a02a02e4-e2b1-45fd-aeda-12fa4e29a996-metrics-tls") pod "dns-default-m9zx6" (UID: "a02a02e4-e2b1-45fd-aeda-12fa4e29a996") : secret "dns-default-metrics-tls" not found
Jul 28 04:12:15.296894 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.296571    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66dfd\" (UniqueName: \"kubernetes.io/projected/a02a02e4-e2b1-45fd-aeda-12fa4e29a996-kube-api-access-66dfd\") pod \"dns-default-m9zx6\" (UID: \"a02a02e4-e2b1-45fd-aeda-12fa4e29a996\") " pod="openshift-dns/dns-default-m9zx6"
Jul 28 04:12:15.296894 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:15.296578    2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 04:12:15.296894 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.296602    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c58f3317-e522-41a3-848f-906af440a23b-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-jxd2j\" (UID: \"c58f3317-e522-41a3-848f-906af440a23b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-jxd2j"
Jul 28 04:12:15.296894 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:15.296627    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c58f3317-e522-41a3-848f-906af440a23b-networking-console-plugin-cert podName:c58f3317-e522-41a3-848f-906af440a23b nodeName:}" failed. No retries permitted until 2026-07-28 04:12:15.79661249 +0000 UTC m=+36.186788497 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c58f3317-e522-41a3-848f-906af440a23b-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-jxd2j" (UID: "c58f3317-e522-41a3-848f-906af440a23b") : secret "networking-console-plugin-cert" not found
Jul 28 04:12:15.296894 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.296661    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-image-registry-private-configuration\") pod \"image-registry-66ff9f7dd7-45ggn\" (UID: \"55cd6e46-995c-4857-b7b7-c90e4e4ff93d\") " pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn"
Jul 28 04:12:15.296894 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.296698    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-trusted-ca\") pod \"image-registry-66ff9f7dd7-45ggn\" (UID: \"55cd6e46-995c-4857-b7b7-c90e4e4ff93d\") " pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn"
Jul 28 04:12:15.296894 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.296859    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-ca-trust-extracted\") pod \"image-registry-66ff9f7dd7-45ggn\" (UID: \"55cd6e46-995c-4857-b7b7-c90e4e4ff93d\") " pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn"
Jul 28 04:12:15.297199 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.296906    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a02a02e4-e2b1-45fd-aeda-12fa4e29a996-tmp-dir\") pod \"dns-default-m9zx6\" (UID: \"a02a02e4-e2b1-45fd-aeda-12fa4e29a996\") " pod="openshift-dns/dns-default-m9zx6"
Jul 28 04:12:15.297199 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.297142    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a02a02e4-e2b1-45fd-aeda-12fa4e29a996-config-volume\") pod \"dns-default-m9zx6\" (UID: \"a02a02e4-e2b1-45fd-aeda-12fa4e29a996\") " pod="openshift-dns/dns-default-m9zx6"
Jul 28 04:12:15.297199 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.297145    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-registry-certificates\") pod \"image-registry-66ff9f7dd7-45ggn\" (UID: \"55cd6e46-995c-4857-b7b7-c90e4e4ff93d\") " pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn"
Jul 28 04:12:15.297400 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.297358    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c58f3317-e522-41a3-848f-906af440a23b-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-jxd2j\" (UID: \"c58f3317-e522-41a3-848f-906af440a23b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-jxd2j"
Jul 28 04:12:15.297599 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.297576    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-trusted-ca\") pod \"image-registry-66ff9f7dd7-45ggn\" (UID: \"55cd6e46-995c-4857-b7b7-c90e4e4ff93d\") " pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn"
Jul 28 04:12:15.300764 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.300739    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-installation-pull-secrets\") pod \"image-registry-66ff9f7dd7-45ggn\" (UID: \"55cd6e46-995c-4857-b7b7-c90e4e4ff93d\") " pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn"
Jul 28 04:12:15.300871 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.300781    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-image-registry-private-configuration\") pod \"image-registry-66ff9f7dd7-45ggn\" (UID: \"55cd6e46-995c-4857-b7b7-c90e4e4ff93d\") " pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn"
Jul 28 04:12:15.300871 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.300828    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9cb032cd-6003-4249-a65d-95d48fea8a02-cert\") pod \"ingress-canary-z8gxf\" (UID: \"9cb032cd-6003-4249-a65d-95d48fea8a02\") " pod="openshift-ingress-canary/ingress-canary-z8gxf"
Jul 28 04:12:15.305339 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.305292    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66dfd\" (UniqueName: \"kubernetes.io/projected/a02a02e4-e2b1-45fd-aeda-12fa4e29a996-kube-api-access-66dfd\") pod \"dns-default-m9zx6\" (UID: \"a02a02e4-e2b1-45fd-aeda-12fa4e29a996\") " pod="openshift-dns/dns-default-m9zx6"
Jul 28 04:12:15.305446 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.305334    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkvg5\" (UniqueName: \"kubernetes.io/projected/9cb032cd-6003-4249-a65d-95d48fea8a02-kube-api-access-wkvg5\") pod \"ingress-canary-z8gxf\" (UID: \"9cb032cd-6003-4249-a65d-95d48fea8a02\") " pod="openshift-ingress-canary/ingress-canary-z8gxf"
Jul 28 04:12:15.305898 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.305868    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-bound-sa-token\") pod \"image-registry-66ff9f7dd7-45ggn\" (UID: \"55cd6e46-995c-4857-b7b7-c90e4e4ff93d\") " pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn"
Jul 28 04:12:15.306549 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.306523    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8rxc\" (UniqueName: \"kubernetes.io/projected/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-kube-api-access-v8rxc\") pod \"image-registry-66ff9f7dd7-45ggn\" (UID: \"55cd6e46-995c-4857-b7b7-c90e4e4ff93d\") " pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn"
Jul 28 04:12:15.377359 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.377290    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-z8gxf"
Jul 28 04:12:15.801995 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.801955    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-registry-tls\") pod \"image-registry-66ff9f7dd7-45ggn\" (UID: \"55cd6e46-995c-4857-b7b7-c90e4e4ff93d\") " pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn"
Jul 28 04:12:15.802146 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.802022    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c58f3317-e522-41a3-848f-906af440a23b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-jxd2j\" (UID: \"c58f3317-e522-41a3-848f-906af440a23b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-jxd2j"
Jul 28 04:12:15.802146 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:15.802063    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a02a02e4-e2b1-45fd-aeda-12fa4e29a996-metrics-tls\") pod \"dns-default-m9zx6\" (UID: \"a02a02e4-e2b1-45fd-aeda-12fa4e29a996\") " pod="openshift-dns/dns-default-m9zx6"
Jul 28 04:12:15.802146 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:15.802126    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 04:12:15.802306 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:15.802149    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66ff9f7dd7-45ggn: secret "image-registry-tls" not found
Jul 28 04:12:15.802306 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:15.802149    2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 04:12:15.802306 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:15.802188    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 04:12:15.802306 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:15.802225    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-registry-tls podName:55cd6e46-995c-4857-b7b7-c90e4e4ff93d nodeName:}" failed. No retries permitted until 2026-07-28 04:12:16.802200647 +0000 UTC m=+37.192376651 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-registry-tls") pod "image-registry-66ff9f7dd7-45ggn" (UID: "55cd6e46-995c-4857-b7b7-c90e4e4ff93d") : secret "image-registry-tls" not found
Jul 28 04:12:15.802306 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:15.802249    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a02a02e4-e2b1-45fd-aeda-12fa4e29a996-metrics-tls podName:a02a02e4-e2b1-45fd-aeda-12fa4e29a996 nodeName:}" failed. No retries permitted until 2026-07-28 04:12:16.802232386 +0000 UTC m=+37.192408391 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a02a02e4-e2b1-45fd-aeda-12fa4e29a996-metrics-tls") pod "dns-default-m9zx6" (UID: "a02a02e4-e2b1-45fd-aeda-12fa4e29a996") : secret "dns-default-metrics-tls" not found
Jul 28 04:12:15.802306 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:15.802267    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c58f3317-e522-41a3-848f-906af440a23b-networking-console-plugin-cert podName:c58f3317-e522-41a3-848f-906af440a23b nodeName:}" failed. No retries permitted until 2026-07-28 04:12:16.802257558 +0000 UTC m=+37.192433561 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c58f3317-e522-41a3-848f-906af440a23b-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-jxd2j" (UID: "c58f3317-e522-41a3-848f-906af440a23b") : secret "networking-console-plugin-cert" not found
Jul 28 04:12:16.475341 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:16.475313    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-z8gxf"]
Jul 28 04:12:16.479867 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:12:16.479659    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cb032cd_6003_4249_a65d_95d48fea8a02.slice/crio-56ad6dcfe03d62768bf8ea7594a7d0fdf53d06b57bf75c0d5628384a91c1b858 WatchSource:0}: Error finding container 56ad6dcfe03d62768bf8ea7594a7d0fdf53d06b57bf75c0d5628384a91c1b858: Status 404 returned error can't find the container with id 56ad6dcfe03d62768bf8ea7594a7d0fdf53d06b57bf75c0d5628384a91c1b858
Jul 28 04:12:16.809954 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:16.809914    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a02a02e4-e2b1-45fd-aeda-12fa4e29a996-metrics-tls\") pod \"dns-default-m9zx6\" (UID: \"a02a02e4-e2b1-45fd-aeda-12fa4e29a996\") " pod="openshift-dns/dns-default-m9zx6"
Jul 28 04:12:16.810160 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:16.810022    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-registry-tls\") pod \"image-registry-66ff9f7dd7-45ggn\" (UID: \"55cd6e46-995c-4857-b7b7-c90e4e4ff93d\") " pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn"
Jul 28 04:12:16.810160 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:16.810070    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c58f3317-e522-41a3-848f-906af440a23b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-jxd2j\" (UID: \"c58f3317-e522-41a3-848f-906af440a23b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-jxd2j"
Jul 28 04:12:16.810160 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:16.810074    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 04:12:16.810160 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:16.810121    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 04:12:16.810160 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:16.810137    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66ff9f7dd7-45ggn: secret "image-registry-tls" not found
Jul 28 04:12:16.810393 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:16.810187    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a02a02e4-e2b1-45fd-aeda-12fa4e29a996-metrics-tls podName:a02a02e4-e2b1-45fd-aeda-12fa4e29a996 nodeName:}" failed. No retries permitted until 2026-07-28 04:12:18.810152797 +0000 UTC m=+39.200328815 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a02a02e4-e2b1-45fd-aeda-12fa4e29a996-metrics-tls") pod "dns-default-m9zx6" (UID: "a02a02e4-e2b1-45fd-aeda-12fa4e29a996") : secret "dns-default-metrics-tls" not found
Jul 28 04:12:16.810393 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:16.810208    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-registry-tls podName:55cd6e46-995c-4857-b7b7-c90e4e4ff93d nodeName:}" failed. No retries permitted until 2026-07-28 04:12:18.810197659 +0000 UTC m=+39.200373665 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-registry-tls") pod "image-registry-66ff9f7dd7-45ggn" (UID: "55cd6e46-995c-4857-b7b7-c90e4e4ff93d") : secret "image-registry-tls" not found
Jul 28 04:12:16.810393 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:16.810225    2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 04:12:16.810393 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:16.810286    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c58f3317-e522-41a3-848f-906af440a23b-networking-console-plugin-cert podName:c58f3317-e522-41a3-848f-906af440a23b nodeName:}" failed. No retries permitted until 2026-07-28 04:12:18.810268139 +0000 UTC m=+39.200444156 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c58f3317-e522-41a3-848f-906af440a23b-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-jxd2j" (UID: "c58f3317-e522-41a3-848f-906af440a23b") : secret "networking-console-plugin-cert" not found
Jul 28 04:12:17.377090 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:17.377017    2566 generic.go:356] "Generic (PLEG): container finished" podID="93788c64-611b-40ff-be6c-8dcea894c08d" containerID="5987beab2087bee6c32fbf47d84a22f45d9de6404dcb3e52cd4e2808a99f45f0" exitCode=0
Jul 28 04:12:17.377090 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:17.377104    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fk4hq" event={"ID":"93788c64-611b-40ff-be6c-8dcea894c08d","Type":"ContainerDied","Data":"5987beab2087bee6c32fbf47d84a22f45d9de6404dcb3e52cd4e2808a99f45f0"}
Jul 28 04:12:17.378569 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:17.378515    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-z8gxf" event={"ID":"9cb032cd-6003-4249-a65d-95d48fea8a02","Type":"ContainerStarted","Data":"56ad6dcfe03d62768bf8ea7594a7d0fdf53d06b57bf75c0d5628384a91c1b858"}
Jul 28 04:12:18.385015 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:18.384974    2566 generic.go:356] "Generic (PLEG): container finished" podID="93788c64-611b-40ff-be6c-8dcea894c08d" containerID="5de9ac8fbee459df8151ac13f0c6d126a399a91d0111c7425865fdf042965517" exitCode=0
Jul 28 04:12:18.385371 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:18.385025    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fk4hq" event={"ID":"93788c64-611b-40ff-be6c-8dcea894c08d","Type":"ContainerDied","Data":"5de9ac8fbee459df8151ac13f0c6d126a399a91d0111c7425865fdf042965517"}
Jul 28 04:12:18.828852 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:18.828802    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-registry-tls\") pod \"image-registry-66ff9f7dd7-45ggn\" (UID: \"55cd6e46-995c-4857-b7b7-c90e4e4ff93d\") " pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn"
Jul 28 04:12:18.829014 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:18.828880    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c58f3317-e522-41a3-848f-906af440a23b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-jxd2j\" (UID: \"c58f3317-e522-41a3-848f-906af440a23b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-jxd2j"
Jul 28 04:12:18.829014 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:18.828916    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a02a02e4-e2b1-45fd-aeda-12fa4e29a996-metrics-tls\") pod \"dns-default-m9zx6\" (UID: \"a02a02e4-e2b1-45fd-aeda-12fa4e29a996\") " pod="openshift-dns/dns-default-m9zx6"
Jul 28 04:12:18.829014 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:18.828968    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 04:12:18.829014 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:18.828989    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66ff9f7dd7-45ggn: secret "image-registry-tls" not found
Jul 28 04:12:18.829208 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:18.829025    2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 04:12:18.829208 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:18.829037    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-registry-tls podName:55cd6e46-995c-4857-b7b7-c90e4e4ff93d nodeName:}" failed. No retries permitted until 2026-07-28 04:12:22.829022221 +0000 UTC m=+43.219198223 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-registry-tls") pod "image-registry-66ff9f7dd7-45ggn" (UID: "55cd6e46-995c-4857-b7b7-c90e4e4ff93d") : secret "image-registry-tls" not found
Jul 28 04:12:18.829208 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:18.829085    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c58f3317-e522-41a3-848f-906af440a23b-networking-console-plugin-cert podName:c58f3317-e522-41a3-848f-906af440a23b nodeName:}" failed. No retries permitted until 2026-07-28 04:12:22.829068855 +0000 UTC m=+43.219244872 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c58f3317-e522-41a3-848f-906af440a23b-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-jxd2j" (UID: "c58f3317-e522-41a3-848f-906af440a23b") : secret "networking-console-plugin-cert" not found
Jul 28 04:12:18.829208 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:18.829030    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 04:12:18.829208 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:18.829126    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a02a02e4-e2b1-45fd-aeda-12fa4e29a996-metrics-tls podName:a02a02e4-e2b1-45fd-aeda-12fa4e29a996 nodeName:}" failed. No retries permitted until 2026-07-28 04:12:22.829116418 +0000 UTC m=+43.219292417 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a02a02e4-e2b1-45fd-aeda-12fa4e29a996-metrics-tls") pod "dns-default-m9zx6" (UID: "a02a02e4-e2b1-45fd-aeda-12fa4e29a996") : secret "dns-default-metrics-tls" not found
Jul 28 04:12:19.392373 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:19.392173    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fk4hq" event={"ID":"93788c64-611b-40ff-be6c-8dcea894c08d","Type":"ContainerStarted","Data":"d9f3485dbcf55e0ce678727febef37383e7335ee9ba43da5b930fd918bb6746a"}
Jul 28 04:12:19.393776 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:19.393749    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-z8gxf" event={"ID":"9cb032cd-6003-4249-a65d-95d48fea8a02","Type":"ContainerStarted","Data":"f665081234f20e76fac89770524bd398804495a99b17e5047e9678857ab3a023"}
Jul 28 04:12:19.415880 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:19.415811    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-fk4hq" podStartSLOduration=8.774790093 podStartE2EDuration="38.415795658s" totalImagesPullingTime="29.641005565s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 04:11:41 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:12:19.414087273 +0000 UTC m=+39.804263294" watchObservedRunningTime="2026-07-28 04:12:19.415795658 +0000 UTC m=+39.805971692"
Jul 28 04:12:19.427887 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:19.427823    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-z8gxf" podStartSLOduration=1.806426789 podStartE2EDuration="4.427812286s" totalImagesPullingTime="2.621385497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:12:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:12:19.42770471 +0000 UTC m=+39.817880732" watchObservedRunningTime="2026-07-28 04:12:19.427812286 +0000 UTC m=+39.817988311"
Jul 28 04:12:22.863500 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:22.863459    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-registry-tls\") pod \"image-registry-66ff9f7dd7-45ggn\" (UID: \"55cd6e46-995c-4857-b7b7-c90e4e4ff93d\") " pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn"
Jul 28 04:12:22.863944 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:22.863512    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c58f3317-e522-41a3-848f-906af440a23b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-jxd2j\" (UID: \"c58f3317-e522-41a3-848f-906af440a23b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-jxd2j"
Jul 28 04:12:22.863944 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:22.863538    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a02a02e4-e2b1-45fd-aeda-12fa4e29a996-metrics-tls\") pod \"dns-default-m9zx6\" (UID: \"a02a02e4-e2b1-45fd-aeda-12fa4e29a996\") " pod="openshift-dns/dns-default-m9zx6"
Jul 28 04:12:22.863944 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:22.863619    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 04:12:22.863944 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:22.863646    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66ff9f7dd7-45ggn: secret "image-registry-tls" not found
Jul 28 04:12:22.863944 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:22.863650    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 04:12:22.863944 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:22.863652    2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 04:12:22.863944 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:22.863697    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a02a02e4-e2b1-45fd-aeda-12fa4e29a996-metrics-tls podName:a02a02e4-e2b1-45fd-aeda-12fa4e29a996 nodeName:}" failed. No retries permitted until 2026-07-28 04:12:30.863684756 +0000 UTC m=+51.253860755 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a02a02e4-e2b1-45fd-aeda-12fa4e29a996-metrics-tls") pod "dns-default-m9zx6" (UID: "a02a02e4-e2b1-45fd-aeda-12fa4e29a996") : secret "dns-default-metrics-tls" not found
Jul 28 04:12:22.863944 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:22.863711    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c58f3317-e522-41a3-848f-906af440a23b-networking-console-plugin-cert podName:c58f3317-e522-41a3-848f-906af440a23b nodeName:}" failed. No retries permitted until 2026-07-28 04:12:30.863705609 +0000 UTC m=+51.253881607 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c58f3317-e522-41a3-848f-906af440a23b-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-jxd2j" (UID: "c58f3317-e522-41a3-848f-906af440a23b") : secret "networking-console-plugin-cert" not found
Jul 28 04:12:22.863944 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:22.863721    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-registry-tls podName:55cd6e46-995c-4857-b7b7-c90e4e4ff93d nodeName:}" failed. No retries permitted until 2026-07-28 04:12:30.863716012 +0000 UTC m=+51.253892011 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-registry-tls") pod "image-registry-66ff9f7dd7-45ggn" (UID: "55cd6e46-995c-4857-b7b7-c90e4e4ff93d") : secret "image-registry-tls" not found
Jul 28 04:12:30.915328 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:30.915277    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-registry-tls\") pod \"image-registry-66ff9f7dd7-45ggn\" (UID: \"55cd6e46-995c-4857-b7b7-c90e4e4ff93d\") " pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn"
Jul 28 04:12:30.915881 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:30.915338    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c58f3317-e522-41a3-848f-906af440a23b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-jxd2j\" (UID: \"c58f3317-e522-41a3-848f-906af440a23b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-jxd2j"
Jul 28 04:12:30.915881 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:30.915367    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a02a02e4-e2b1-45fd-aeda-12fa4e29a996-metrics-tls\") pod \"dns-default-m9zx6\" (UID: \"a02a02e4-e2b1-45fd-aeda-12fa4e29a996\") " pod="openshift-dns/dns-default-m9zx6"
Jul 28 04:12:30.915881 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:30.915445    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 04:12:30.915881 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:30.915471    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66ff9f7dd7-45ggn: secret "image-registry-tls" not found
Jul 28 04:12:30.915881 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:30.915475    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 04:12:30.915881 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:30.915487    2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 04:12:30.915881 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:30.915536    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-registry-tls podName:55cd6e46-995c-4857-b7b7-c90e4e4ff93d nodeName:}" failed. No retries permitted until 2026-07-28 04:12:46.915514839 +0000 UTC m=+67.305690854 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-registry-tls") pod "image-registry-66ff9f7dd7-45ggn" (UID: "55cd6e46-995c-4857-b7b7-c90e4e4ff93d") : secret "image-registry-tls" not found
Jul 28 04:12:30.915881 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:30.915556    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a02a02e4-e2b1-45fd-aeda-12fa4e29a996-metrics-tls podName:a02a02e4-e2b1-45fd-aeda-12fa4e29a996 nodeName:}" failed. No retries permitted until 2026-07-28 04:12:46.915542954 +0000 UTC m=+67.305718955 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a02a02e4-e2b1-45fd-aeda-12fa4e29a996-metrics-tls") pod "dns-default-m9zx6" (UID: "a02a02e4-e2b1-45fd-aeda-12fa4e29a996") : secret "dns-default-metrics-tls" not found
Jul 28 04:12:30.915881 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:30.915570    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c58f3317-e522-41a3-848f-906af440a23b-networking-console-plugin-cert podName:c58f3317-e522-41a3-848f-906af440a23b nodeName:}" failed. No retries permitted until 2026-07-28 04:12:46.915564071 +0000 UTC m=+67.305740069 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c58f3317-e522-41a3-848f-906af440a23b-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-jxd2j" (UID: "c58f3317-e522-41a3-848f-906af440a23b") : secret "networking-console-plugin-cert" not found
Jul 28 04:12:41.376304 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:41.376277    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8xswd"
Jul 28 04:12:45.820511 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:45.820456    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/accccd5b-603e-49d5-b9fc-b328dcb7e959-original-pull-secret\") pod \"global-pull-secret-syncer-z2gn6\" (UID: \"accccd5b-603e-49d5-b9fc-b328dcb7e959\") " pod="kube-system/global-pull-secret-syncer-z2gn6"
Jul 28 04:12:45.820511 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:45.820526    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6107883-62a6-4a82-9e04-ed46057a6b0e-metrics-certs\") pod \"network-metrics-daemon-l6p44\" (UID: \"e6107883-62a6-4a82-9e04-ed46057a6b0e\") " pod="openshift-multus/network-metrics-daemon-l6p44"
Jul 28 04:12:45.823276 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:45.823256    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 04:12:45.823529 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:45.823514    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 04:12:45.831679 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:45.831663    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 28 04:12:45.831734 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:45.831721    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6107883-62a6-4a82-9e04-ed46057a6b0e-metrics-certs podName:e6107883-62a6-4a82-9e04-ed46057a6b0e nodeName:}" failed. No retries permitted until 2026-07-28 04:13:49.831705442 +0000 UTC m=+130.221881442 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e6107883-62a6-4a82-9e04-ed46057a6b0e-metrics-certs") pod "network-metrics-daemon-l6p44" (UID: "e6107883-62a6-4a82-9e04-ed46057a6b0e") : secret "metrics-daemon-secret" not found
Jul 28 04:12:45.835155 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:45.835137    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/accccd5b-603e-49d5-b9fc-b328dcb7e959-original-pull-secret\") pod \"global-pull-secret-syncer-z2gn6\" (UID: \"accccd5b-603e-49d5-b9fc-b328dcb7e959\") " pod="kube-system/global-pull-secret-syncer-z2gn6"
Jul 28 04:12:45.921657 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:45.921610    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmnvx\" (UniqueName: \"kubernetes.io/projected/33a30d7b-76be-42e9-ae95-6779c5770b15-kube-api-access-zmnvx\") pod \"network-check-target-qgqqn\" (UID: \"33a30d7b-76be-42e9-ae95-6779c5770b15\") " pod="openshift-network-diagnostics/network-check-target-qgqqn"
Jul 28 04:12:45.924348 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:45.924330    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 04:12:45.934665 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:45.934649    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 04:12:45.944777 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:45.944753    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmnvx\" (UniqueName: \"kubernetes.io/projected/33a30d7b-76be-42e9-ae95-6779c5770b15-kube-api-access-zmnvx\") pod \"network-check-target-qgqqn\" (UID: \"33a30d7b-76be-42e9-ae95-6779c5770b15\") " pod="openshift-network-diagnostics/network-check-target-qgqqn"
Jul 28 04:12:45.997754 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:45.997733    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9nvrd\""
Jul 28 04:12:46.005762 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:46.005742    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qgqqn"
Jul 28 04:12:46.007954 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:46.007934    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z2gn6"
Jul 28 04:12:46.151040 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:46.151017    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-z2gn6"]
Jul 28 04:12:46.153858 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:12:46.153812    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaccccd5b_603e_49d5_b9fc_b328dcb7e959.slice/crio-dd642bfeb1ec1f9d98ffa39056cbb928a95af991945d6214775d510093d11dc1 WatchSource:0}: Error finding container dd642bfeb1ec1f9d98ffa39056cbb928a95af991945d6214775d510093d11dc1: Status 404 returned error can't find the container with id dd642bfeb1ec1f9d98ffa39056cbb928a95af991945d6214775d510093d11dc1
Jul 28 04:12:46.166643 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:46.166620    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qgqqn"]
Jul 28 04:12:46.168728 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:12:46.168708    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33a30d7b_76be_42e9_ae95_6779c5770b15.slice/crio-0090dc0f4a7b6bacb32e850dc9b331a70a030065ffc7e262b9dc294a1a1f3c8c WatchSource:0}: Error finding container 0090dc0f4a7b6bacb32e850dc9b331a70a030065ffc7e262b9dc294a1a1f3c8c: Status 404 returned error can't find the container with id 0090dc0f4a7b6bacb32e850dc9b331a70a030065ffc7e262b9dc294a1a1f3c8c
Jul 28 04:12:46.455880 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:46.455770    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-z2gn6" event={"ID":"accccd5b-603e-49d5-b9fc-b328dcb7e959","Type":"ContainerStarted","Data":"dd642bfeb1ec1f9d98ffa39056cbb928a95af991945d6214775d510093d11dc1"}
Jul 28 04:12:46.456801 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:46.456779    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qgqqn" event={"ID":"33a30d7b-76be-42e9-ae95-6779c5770b15","Type":"ContainerStarted","Data":"0090dc0f4a7b6bacb32e850dc9b331a70a030065ffc7e262b9dc294a1a1f3c8c"}
Jul 28 04:12:46.930680 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:46.930637    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-registry-tls\") pod \"image-registry-66ff9f7dd7-45ggn\" (UID: \"55cd6e46-995c-4857-b7b7-c90e4e4ff93d\") " pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn"
Jul 28 04:12:46.931128 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:46.930710    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c58f3317-e522-41a3-848f-906af440a23b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-jxd2j\" (UID: \"c58f3317-e522-41a3-848f-906af440a23b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-jxd2j"
Jul 28 04:12:46.931128 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:46.930752    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a02a02e4-e2b1-45fd-aeda-12fa4e29a996-metrics-tls\") pod \"dns-default-m9zx6\" (UID: \"a02a02e4-e2b1-45fd-aeda-12fa4e29a996\") " pod="openshift-dns/dns-default-m9zx6"
Jul 28 04:12:46.931128 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:46.930809    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 04:12:46.931128 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:46.930832    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66ff9f7dd7-45ggn: secret "image-registry-tls" not found
Jul 28 04:12:46.931128 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:46.930889    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 04:12:46.931128 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:46.930926    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-registry-tls podName:55cd6e46-995c-4857-b7b7-c90e4e4ff93d nodeName:}" failed. No retries permitted until 2026-07-28 04:13:18.930902881 +0000 UTC m=+99.321078899 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-registry-tls") pod "image-registry-66ff9f7dd7-45ggn" (UID: "55cd6e46-995c-4857-b7b7-c90e4e4ff93d") : secret "image-registry-tls" not found
Jul 28 04:12:46.931128 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:46.930890    2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 04:12:46.931128 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:46.930948    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a02a02e4-e2b1-45fd-aeda-12fa4e29a996-metrics-tls podName:a02a02e4-e2b1-45fd-aeda-12fa4e29a996 nodeName:}" failed. No retries permitted until 2026-07-28 04:13:18.930937703 +0000 UTC m=+99.321113708 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a02a02e4-e2b1-45fd-aeda-12fa4e29a996-metrics-tls") pod "dns-default-m9zx6" (UID: "a02a02e4-e2b1-45fd-aeda-12fa4e29a996") : secret "dns-default-metrics-tls" not found
Jul 28 04:12:46.931128 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:12:46.930972    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c58f3317-e522-41a3-848f-906af440a23b-networking-console-plugin-cert podName:c58f3317-e522-41a3-848f-906af440a23b nodeName:}" failed. No retries permitted until 2026-07-28 04:13:18.930955227 +0000 UTC m=+99.321131231 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c58f3317-e522-41a3-848f-906af440a23b-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-jxd2j" (UID: "c58f3317-e522-41a3-848f-906af440a23b") : secret "networking-console-plugin-cert" not found
Jul 28 04:12:51.472586 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:51.472552    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qgqqn" event={"ID":"33a30d7b-76be-42e9-ae95-6779c5770b15","Type":"ContainerStarted","Data":"84fe6433fdd9a594bec48a755607a3dfcecc8727d79025e30a8c26a5467d9448"}
Jul 28 04:12:51.473057 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:51.472629    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qgqqn"
Jul 28 04:12:51.474054 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:51.474030    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-z2gn6" event={"ID":"accccd5b-603e-49d5-b9fc-b328dcb7e959","Type":"ContainerStarted","Data":"6b4b63b8408178f8244e0f15625bd2c5a4befdbb6539aa09c01989929a0969de"}
Jul 28 04:12:51.486288 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:51.486251    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qgqqn" podStartSLOduration=65.702769839 podStartE2EDuration="1m10.486238968s" totalImagesPullingTime="4.783469129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:11:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:12:51.485061356 +0000 UTC m=+71.875237377" watchObservedRunningTime="2026-07-28 04:12:51.486238968 +0000 UTC m=+71.876414989"
Jul 28 04:12:51.497852 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:12:51.497799    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-z2gn6" podStartSLOduration=65.687065563 podStartE2EDuration="1m10.497786708s" totalImagesPullingTime="4.810721145s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 04:11:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:12:51.497497201 +0000 UTC m=+71.887673222" watchObservedRunningTime="2026-07-28 04:12:51.497786708 +0000 UTC m=+71.887962728"
Jul 28 04:13:18.959798 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:13:18.959650    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-registry-tls\") pod \"image-registry-66ff9f7dd7-45ggn\" (UID: \"55cd6e46-995c-4857-b7b7-c90e4e4ff93d\") " pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn"
Jul 28 04:13:18.959798 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:13:18.959713    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c58f3317-e522-41a3-848f-906af440a23b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-jxd2j\" (UID: \"c58f3317-e522-41a3-848f-906af440a23b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-jxd2j"
Jul 28 04:13:18.959798 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:13:18.959737    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a02a02e4-e2b1-45fd-aeda-12fa4e29a996-metrics-tls\") pod \"dns-default-m9zx6\" (UID: \"a02a02e4-e2b1-45fd-aeda-12fa4e29a996\") " pod="openshift-dns/dns-default-m9zx6"
Jul 28 04:13:18.959798 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:13:18.959786    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 04:13:18.959798 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:13:18.959805    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66ff9f7dd7-45ggn: secret "image-registry-tls" not found
Jul 28 04:13:18.960430 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:13:18.959860    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 04:13:18.960430 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:13:18.959875    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-registry-tls podName:55cd6e46-995c-4857-b7b7-c90e4e4ff93d nodeName:}" failed. No retries permitted until 2026-07-28 04:14:22.959858312 +0000 UTC m=+163.350034328 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-registry-tls") pod "image-registry-66ff9f7dd7-45ggn" (UID: "55cd6e46-995c-4857-b7b7-c90e4e4ff93d") : secret "image-registry-tls" not found
Jul 28 04:13:18.960430 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:13:18.959882    2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 04:13:18.960430 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:13:18.959916    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a02a02e4-e2b1-45fd-aeda-12fa4e29a996-metrics-tls podName:a02a02e4-e2b1-45fd-aeda-12fa4e29a996 nodeName:}" failed. No retries permitted until 2026-07-28 04:14:22.959902278 +0000 UTC m=+163.350078295 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a02a02e4-e2b1-45fd-aeda-12fa4e29a996-metrics-tls") pod "dns-default-m9zx6" (UID: "a02a02e4-e2b1-45fd-aeda-12fa4e29a996") : secret "dns-default-metrics-tls" not found
Jul 28 04:13:18.960430 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:13:18.959955    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c58f3317-e522-41a3-848f-906af440a23b-networking-console-plugin-cert podName:c58f3317-e522-41a3-848f-906af440a23b nodeName:}" failed. No retries permitted until 2026-07-28 04:14:22.959935878 +0000 UTC m=+163.350111883 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c58f3317-e522-41a3-848f-906af440a23b-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-jxd2j" (UID: "c58f3317-e522-41a3-848f-906af440a23b") : secret "networking-console-plugin-cert" not found
Jul 28 04:13:22.478805 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:13:22.478776    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qgqqn"
Jul 28 04:13:49.877689 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:13:49.877636    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6107883-62a6-4a82-9e04-ed46057a6b0e-metrics-certs\") pod \"network-metrics-daemon-l6p44\" (UID: \"e6107883-62a6-4a82-9e04-ed46057a6b0e\") " pod="openshift-multus/network-metrics-daemon-l6p44"
Jul 28 04:13:49.878202 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:13:49.877789    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 28 04:13:49.878202 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:13:49.877882    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6107883-62a6-4a82-9e04-ed46057a6b0e-metrics-certs podName:e6107883-62a6-4a82-9e04-ed46057a6b0e nodeName:}" failed. No retries permitted until 2026-07-28 04:15:51.87786565 +0000 UTC m=+252.268041649 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e6107883-62a6-4a82-9e04-ed46057a6b0e-metrics-certs") pod "network-metrics-daemon-l6p44" (UID: "e6107883-62a6-4a82-9e04-ed46057a6b0e") : secret "metrics-daemon-secret" not found
Jul 28 04:14:18.060231 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:14:18.060162    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-74c9f884b7-jxd2j" podUID="c58f3317-e522-41a3-848f-906af440a23b"
Jul 28 04:14:18.067292 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:14:18.067258    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn" podUID="55cd6e46-995c-4857-b7b7-c90e4e4ff93d"
Jul 28 04:14:18.087343 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:14:18.087313    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-m9zx6" podUID="a02a02e4-e2b1-45fd-aeda-12fa4e29a996"
Jul 28 04:14:18.101932 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:14:18.101909    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-l6p44" podUID="e6107883-62a6-4a82-9e04-ed46057a6b0e"
Jul 28 04:14:18.671641 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:14:18.671607    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn"
Jul 28 04:14:18.671827 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:14:18.671619    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-m9zx6"
Jul 28 04:14:18.671827 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:14:18.671618    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-jxd2j"
Jul 28 04:14:22.998885 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:14:22.998830    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-registry-tls\") pod \"image-registry-66ff9f7dd7-45ggn\" (UID: \"55cd6e46-995c-4857-b7b7-c90e4e4ff93d\") " pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn"
Jul 28 04:14:22.999258 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:14:22.998909    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c58f3317-e522-41a3-848f-906af440a23b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-jxd2j\" (UID: \"c58f3317-e522-41a3-848f-906af440a23b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-jxd2j"
Jul 28 04:14:22.999258 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:14:22.998948    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a02a02e4-e2b1-45fd-aeda-12fa4e29a996-metrics-tls\") pod \"dns-default-m9zx6\" (UID: \"a02a02e4-e2b1-45fd-aeda-12fa4e29a996\") " pod="openshift-dns/dns-default-m9zx6"
Jul 28 04:14:22.999258 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:14:22.998994    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 04:14:22.999258 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:14:22.999012    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66ff9f7dd7-45ggn: secret "image-registry-tls" not found
Jul 28 04:14:22.999258 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:14:22.999051    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 04:14:22.999258 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:14:22.999051    2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 28 04:14:22.999258 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:14:22.999105    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-registry-tls podName:55cd6e46-995c-4857-b7b7-c90e4e4ff93d nodeName:}" failed. No retries permitted until 2026-07-28 04:16:24.99908578 +0000 UTC m=+285.389261796 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-registry-tls") pod "image-registry-66ff9f7dd7-45ggn" (UID: "55cd6e46-995c-4857-b7b7-c90e4e4ff93d") : secret "image-registry-tls" not found
Jul 28 04:14:22.999258 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:14:22.999123    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a02a02e4-e2b1-45fd-aeda-12fa4e29a996-metrics-tls podName:a02a02e4-e2b1-45fd-aeda-12fa4e29a996 nodeName:}" failed. No retries permitted until 2026-07-28 04:16:24.999113859 +0000 UTC m=+285.389289868 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a02a02e4-e2b1-45fd-aeda-12fa4e29a996-metrics-tls") pod "dns-default-m9zx6" (UID: "a02a02e4-e2b1-45fd-aeda-12fa4e29a996") : secret "dns-default-metrics-tls" not found
Jul 28 04:14:22.999258 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:14:22.999133    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c58f3317-e522-41a3-848f-906af440a23b-networking-console-plugin-cert podName:c58f3317-e522-41a3-848f-906af440a23b nodeName:}" failed. No retries permitted until 2026-07-28 04:16:24.999127169 +0000 UTC m=+285.389303168 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c58f3317-e522-41a3-848f-906af440a23b-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-jxd2j" (UID: "c58f3317-e522-41a3-848f-906af440a23b") : secret "networking-console-plugin-cert" not found
Jul 28 04:14:30.085206 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:14:30.085169    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6p44"
Jul 28 04:15:36.923496 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:36.923463    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-lmrmn"]
Jul 28 04:15:36.926338 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:36.926313    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-lmrmn"
Jul 28 04:15:36.927170 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:36.927139    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-bcwm5"]
Jul 28 04:15:36.929606 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:36.929582    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 28 04:15:36.929606 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:36.929599    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 04:15:36.929869 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:36.929827    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-mfd2q"]
Jul 28 04:15:36.929981 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:36.929891    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 28 04:15:36.929981 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:36.929926    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-bcwm5"
Jul 28 04:15:36.930099 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:36.930072    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-p4pmp\""
Jul 28 04:15:36.930099 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:36.929933    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 04:15:36.932437 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:36.932417    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-4glpr"]
Jul 28 04:15:36.932575 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:36.932560    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-mfd2q"
Jul 28 04:15:36.933610 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:36.933079    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 28 04:15:36.933940 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:36.933905    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-gqfcr\""
Jul 28 04:15:36.933940 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:36.933911    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 28 04:15:36.934120 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:36.933981    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 28 04:15:36.934120 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:36.933986    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 28 04:15:36.935153 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:36.935138    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 28 04:15:36.935225 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:36.935144    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-4glpr"
Jul 28 04:15:36.937259 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:36.937244    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 04:15:36.937346 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:36.937273    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 28 04:15:36.937546 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:36.937523    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 04:15:36.939016 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:36.938995    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 28 04:15:36.939192 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:36.938995    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-xhbx7\""
Jul 28 04:15:36.940071 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:36.940052    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 28 04:15:36.943585 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:36.943559    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 28 04:15:36.944062 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:36.944043    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-2kt8s\""
Jul 28 04:15:36.944329 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:36.944307    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 28 04:15:36.944670 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:36.944652    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 28 04:15:36.946944 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:36.946924    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 28 04:15:36.949717 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:36.949622    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-mfd2q"]
Jul 28 04:15:36.950537 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:36.950519    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-4glpr"]
Jul 28 04:15:36.951719 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:36.951699    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-lmrmn"]
Jul 28 04:15:36.962276 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:36.962257    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-bcwm5"]
Jul 28 04:15:37.012171 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.012142    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-wjm98"]
Jul 28 04:15:37.015149 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.015133    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-wjm98"
Jul 28 04:15:37.018305 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.018284    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-qw4wd\""
Jul 28 04:15:37.018950 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.018935    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-d2lpq"]
Jul 28 04:15:37.020723 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.020702    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bfd2d3d8-f287-42d8-83c7-f82185c96735-trusted-ca\") pod \"console-operator-584dcb6b54-4glpr\" (UID: \"bfd2d3d8-f287-42d8-83c7-f82185c96735\") " pod="openshift-console-operator/console-operator-584dcb6b54-4glpr"
Jul 28 04:15:37.020852 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.020803    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f77a6b03-d007-46e3-852b-4b262feca301-service-ca-bundle\") pod \"insights-operator-5885fdb759-lmrmn\" (UID: \"f77a6b03-d007-46e3-852b-4b262feca301\") " pod="openshift-insights/insights-operator-5885fdb759-lmrmn"
Jul 28 04:15:37.020913 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.020854    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f77a6b03-d007-46e3-852b-4b262feca301-serving-cert\") pod \"insights-operator-5885fdb759-lmrmn\" (UID: \"f77a6b03-d007-46e3-852b-4b262feca301\") " pod="openshift-insights/insights-operator-5885fdb759-lmrmn"
Jul 28 04:15:37.020981 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.020913    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/317c317c-02c3-4e8f-be22-5b81aea93846-serving-cert\") pod \"service-ca-operator-5d6c75f84f-bcwm5\" (UID: \"317c317c-02c3-4e8f-be22-5b81aea93846\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-bcwm5"
Jul 28 04:15:37.021036 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.020989    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89mdw\" (UniqueName: \"kubernetes.io/projected/edec740c-2287-4ba9-ae6b-0f9da7320a48-kube-api-access-89mdw\") pod \"cluster-monitoring-operator-548b489cb8-mfd2q\" (UID: \"edec740c-2287-4ba9-ae6b-0f9da7320a48\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-mfd2q"
Jul 28 04:15:37.021036 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.021021    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/317c317c-02c3-4e8f-be22-5b81aea93846-config\") pod \"service-ca-operator-5d6c75f84f-bcwm5\" (UID: \"317c317c-02c3-4e8f-be22-5b81aea93846\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-bcwm5"
Jul 28 04:15:37.021139 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.021079    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq9ph\" (UniqueName: \"kubernetes.io/projected/bfd2d3d8-f287-42d8-83c7-f82185c96735-kube-api-access-zq9ph\") pod \"console-operator-584dcb6b54-4glpr\" (UID: \"bfd2d3d8-f287-42d8-83c7-f82185c96735\") " pod="openshift-console-operator/console-operator-584dcb6b54-4glpr"
Jul 28 04:15:37.021139 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.021123    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hjwf\" (UniqueName: \"kubernetes.io/projected/f77a6b03-d007-46e3-852b-4b262feca301-kube-api-access-7hjwf\") pod \"insights-operator-5885fdb759-lmrmn\" (UID: \"f77a6b03-d007-46e3-852b-4b262feca301\") " pod="openshift-insights/insights-operator-5885fdb759-lmrmn"
Jul 28 04:15:37.021242 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.021150    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/edec740c-2287-4ba9-ae6b-0f9da7320a48-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-mfd2q\" (UID: \"edec740c-2287-4ba9-ae6b-0f9da7320a48\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-mfd2q"
Jul 28 04:15:37.021242 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.021181    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bfd2d3d8-f287-42d8-83c7-f82185c96735-config\") pod \"console-operator-584dcb6b54-4glpr\" (UID: \"bfd2d3d8-f287-42d8-83c7-f82185c96735\") " pod="openshift-console-operator/console-operator-584dcb6b54-4glpr"
Jul 28 04:15:37.021242 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.021207    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/edec740c-2287-4ba9-ae6b-0f9da7320a48-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-mfd2q\" (UID: \"edec740c-2287-4ba9-ae6b-0f9da7320a48\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-mfd2q"
Jul 28 04:15:37.021401 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.021241    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f77a6b03-d007-46e3-852b-4b262feca301-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-lmrmn\" (UID: \"f77a6b03-d007-46e3-852b-4b262feca301\") " pod="openshift-insights/insights-operator-5885fdb759-lmrmn"
Jul 28 04:15:37.021401 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.021298    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f77a6b03-d007-46e3-852b-4b262feca301-tmp\") pod \"insights-operator-5885fdb759-lmrmn\" (UID: \"f77a6b03-d007-46e3-852b-4b262feca301\") " pod="openshift-insights/insights-operator-5885fdb759-lmrmn"
Jul 28 04:15:37.021401 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.021339    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bfd2d3d8-f287-42d8-83c7-f82185c96735-serving-cert\") pod \"console-operator-584dcb6b54-4glpr\" (UID: \"bfd2d3d8-f287-42d8-83c7-f82185c96735\") " pod="openshift-console-operator/console-operator-584dcb6b54-4glpr"
Jul 28 04:15:37.021401 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.021366    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g67j\" (UniqueName: \"kubernetes.io/projected/317c317c-02c3-4e8f-be22-5b81aea93846-kube-api-access-9g67j\") pod \"service-ca-operator-5d6c75f84f-bcwm5\" (UID: \"317c317c-02c3-4e8f-be22-5b81aea93846\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-bcwm5"
Jul 28 04:15:37.021567 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.021419    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f77a6b03-d007-46e3-852b-4b262feca301-snapshots\") pod \"insights-operator-5885fdb759-lmrmn\" (UID: \"f77a6b03-d007-46e3-852b-4b262feca301\") " pod="openshift-insights/insights-operator-5885fdb759-lmrmn"
Jul 28 04:15:37.021567 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.021540    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-95c8f4d49-5ncng"]
Jul 28 04:15:37.021670 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.021654    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-d2lpq"
Jul 28 04:15:37.024384 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.024366    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-95c8f4d49-5ncng"
Jul 28 04:15:37.024901 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.024880    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 28 04:15:37.025003 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.024888    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 28 04:15:37.025198 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.025177    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 28 04:15:37.025298 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.025177    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-nz65h\""
Jul 28 04:15:37.025298 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.025178    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 28 04:15:37.025708 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.025689    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-wjm98"]
Jul 28 04:15:37.026807 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.026787    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 28 04:15:37.026899 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.026794    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 28 04:15:37.027575 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.027562    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 28 04:15:37.027889 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.027870    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 28 04:15:37.027988 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.027888    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 28 04:15:37.027988 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.027899    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-zdg7l\""
Jul 28 04:15:37.027988 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.027917    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 28 04:15:37.031977 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.031957    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-d2lpq"]
Jul 28 04:15:37.037615 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.037594    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-95c8f4d49-5ncng"]
Jul 28 04:15:37.121853 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.121807    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f77a6b03-d007-46e3-852b-4b262feca301-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-lmrmn\" (UID: \"f77a6b03-d007-46e3-852b-4b262feca301\") " pod="openshift-insights/insights-operator-5885fdb759-lmrmn"
Jul 28 04:15:37.121853 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.121857    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-622g6\" (UniqueName: \"kubernetes.io/projected/3837b644-a03e-498c-9033-1ed2e1d894d0-kube-api-access-622g6\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-d2lpq\" (UID: \"3837b644-a03e-498c-9033-1ed2e1d894d0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-d2lpq"
Jul 28 04:15:37.122131 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.121890    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f77a6b03-d007-46e3-852b-4b262feca301-tmp\") pod \"insights-operator-5885fdb759-lmrmn\" (UID: \"f77a6b03-d007-46e3-852b-4b262feca301\") " pod="openshift-insights/insights-operator-5885fdb759-lmrmn"
Jul 28 04:15:37.122131 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.121909    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bfd2d3d8-f287-42d8-83c7-f82185c96735-serving-cert\") pod \"console-operator-584dcb6b54-4glpr\" (UID: \"bfd2d3d8-f287-42d8-83c7-f82185c96735\") " pod="openshift-console-operator/console-operator-584dcb6b54-4glpr"
Jul 28 04:15:37.122131 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.121923    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g67j\" (UniqueName: \"kubernetes.io/projected/317c317c-02c3-4e8f-be22-5b81aea93846-kube-api-access-9g67j\") pod \"service-ca-operator-5d6c75f84f-bcwm5\" (UID: \"317c317c-02c3-4e8f-be22-5b81aea93846\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-bcwm5"
Jul 28 04:15:37.122131 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.121942    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f77a6b03-d007-46e3-852b-4b262feca301-snapshots\") pod \"insights-operator-5885fdb759-lmrmn\" (UID: \"f77a6b03-d007-46e3-852b-4b262feca301\") " pod="openshift-insights/insights-operator-5885fdb759-lmrmn"
Jul 28 04:15:37.122131 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.121961    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bfd2d3d8-f287-42d8-83c7-f82185c96735-trusted-ca\") pod \"console-operator-584dcb6b54-4glpr\" (UID: \"bfd2d3d8-f287-42d8-83c7-f82185c96735\") " pod="openshift-console-operator/console-operator-584dcb6b54-4glpr"
Jul 28 04:15:37.122131 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.121977    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3837b644-a03e-498c-9033-1ed2e1d894d0-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-d2lpq\" (UID: \"3837b644-a03e-498c-9033-1ed2e1d894d0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-d2lpq"
Jul 28 04:15:37.122131 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.121999    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c7f7f74-e140-4d88-bc93-c029ae1c18a7-service-ca-bundle\") pod \"router-default-95c8f4d49-5ncng\" (UID: \"5c7f7f74-e140-4d88-bc93-c029ae1c18a7\") " pod="openshift-ingress/router-default-95c8f4d49-5ncng"
Jul 28 04:15:37.122131 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.122045    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f77a6b03-d007-46e3-852b-4b262feca301-service-ca-bundle\") pod \"insights-operator-5885fdb759-lmrmn\" (UID: \"f77a6b03-d007-46e3-852b-4b262feca301\") " pod="openshift-insights/insights-operator-5885fdb759-lmrmn"
Jul 28 04:15:37.122131 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.122077    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f77a6b03-d007-46e3-852b-4b262feca301-serving-cert\") pod \"insights-operator-5885fdb759-lmrmn\" (UID: \"f77a6b03-d007-46e3-852b-4b262feca301\") " pod="openshift-insights/insights-operator-5885fdb759-lmrmn"
Jul 28 04:15:37.122131 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.122103    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/317c317c-02c3-4e8f-be22-5b81aea93846-serving-cert\") pod \"service-ca-operator-5d6c75f84f-bcwm5\" (UID: \"317c317c-02c3-4e8f-be22-5b81aea93846\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-bcwm5"
Jul 28 04:15:37.122771 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.122137    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5c7f7f74-e140-4d88-bc93-c029ae1c18a7-stats-auth\") pod \"router-default-95c8f4d49-5ncng\" (UID: \"5c7f7f74-e140-4d88-bc93-c029ae1c18a7\") " pod="openshift-ingress/router-default-95c8f4d49-5ncng"
Jul 28 04:15:37.122771 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.122168    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3837b644-a03e-498c-9033-1ed2e1d894d0-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-d2lpq\" (UID: \"3837b644-a03e-498c-9033-1ed2e1d894d0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-d2lpq"
Jul 28 04:15:37.122771 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.122196    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qtdf\" (UniqueName: \"kubernetes.io/projected/1f989c12-30f3-4710-add3-12c15a9af79c-kube-api-access-6qtdf\") pod \"network-check-source-7d88cb55d8-wjm98\" (UID: \"1f989c12-30f3-4710-add3-12c15a9af79c\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-wjm98"
Jul 28 04:15:37.122771 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.122223    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89mdw\" (UniqueName: \"kubernetes.io/projected/edec740c-2287-4ba9-ae6b-0f9da7320a48-kube-api-access-89mdw\") pod \"cluster-monitoring-operator-548b489cb8-mfd2q\" (UID: \"edec740c-2287-4ba9-ae6b-0f9da7320a48\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-mfd2q"
Jul 28 04:15:37.122771 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.122251    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/317c317c-02c3-4e8f-be22-5b81aea93846-config\") pod \"service-ca-operator-5d6c75f84f-bcwm5\" (UID: \"317c317c-02c3-4e8f-be22-5b81aea93846\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-bcwm5"
Jul 28 04:15:37.122771 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.122308    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq9ph\" (UniqueName: \"kubernetes.io/projected/bfd2d3d8-f287-42d8-83c7-f82185c96735-kube-api-access-zq9ph\") pod \"console-operator-584dcb6b54-4glpr\" (UID: \"bfd2d3d8-f287-42d8-83c7-f82185c96735\") " pod="openshift-console-operator/console-operator-584dcb6b54-4glpr"
Jul 28 04:15:37.122771 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.122347    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hjwf\" (UniqueName: \"kubernetes.io/projected/f77a6b03-d007-46e3-852b-4b262feca301-kube-api-access-7hjwf\") pod \"insights-operator-5885fdb759-lmrmn\" (UID: \"f77a6b03-d007-46e3-852b-4b262feca301\") " pod="openshift-insights/insights-operator-5885fdb759-lmrmn"
Jul 28 04:15:37.122771 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.122371    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/edec740c-2287-4ba9-ae6b-0f9da7320a48-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-mfd2q\" (UID: \"edec740c-2287-4ba9-ae6b-0f9da7320a48\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-mfd2q"
Jul 28 04:15:37.122771 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.122400    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bfd2d3d8-f287-42d8-83c7-f82185c96735-config\") pod \"console-operator-584dcb6b54-4glpr\" (UID: \"bfd2d3d8-f287-42d8-83c7-f82185c96735\") " pod="openshift-console-operator/console-operator-584dcb6b54-4glpr"
Jul 28 04:15:37.122771 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.122428    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c7f7f74-e140-4d88-bc93-c029ae1c18a7-metrics-certs\") pod \"router-default-95c8f4d49-5ncng\" (UID: \"5c7f7f74-e140-4d88-bc93-c029ae1c18a7\") " pod="openshift-ingress/router-default-95c8f4d49-5ncng"
Jul 28 04:15:37.122771 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.122455    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/edec740c-2287-4ba9-ae6b-0f9da7320a48-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-mfd2q\" (UID: \"edec740c-2287-4ba9-ae6b-0f9da7320a48\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-mfd2q"
Jul 28 04:15:37.122771 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.122458    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f77a6b03-d007-46e3-852b-4b262feca301-tmp\") pod \"insights-operator-5885fdb759-lmrmn\" (UID: \"f77a6b03-d007-46e3-852b-4b262feca301\") " pod="openshift-insights/insights-operator-5885fdb759-lmrmn"
Jul 28 04:15:37.122771 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.122480    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5c7f7f74-e140-4d88-bc93-c029ae1c18a7-default-certificate\") pod \"router-default-95c8f4d49-5ncng\" (UID: \"5c7f7f74-e140-4d88-bc93-c029ae1c18a7\") " pod="openshift-ingress/router-default-95c8f4d49-5ncng"
Jul 28 04:15:37.122771 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.122512    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ddtd\" (UniqueName: \"kubernetes.io/projected/5c7f7f74-e140-4d88-bc93-c029ae1c18a7-kube-api-access-8ddtd\") pod \"router-default-95c8f4d49-5ncng\" (UID: \"5c7f7f74-e140-4d88-bc93-c029ae1c18a7\") " pod="openshift-ingress/router-default-95c8f4d49-5ncng"
Jul 28 04:15:37.122771 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.122767    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f77a6b03-d007-46e3-852b-4b262feca301-snapshots\") pod \"insights-operator-5885fdb759-lmrmn\" (UID: \"f77a6b03-d007-46e3-852b-4b262feca301\") " pod="openshift-insights/insights-operator-5885fdb759-lmrmn"
Jul 28 04:15:37.123532 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.122882    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f77a6b03-d007-46e3-852b-4b262feca301-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-lmrmn\" (UID: \"f77a6b03-d007-46e3-852b-4b262feca301\") " pod="openshift-insights/insights-operator-5885fdb759-lmrmn"
Jul 28 04:15:37.123532 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.122990    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f77a6b03-d007-46e3-852b-4b262feca301-service-ca-bundle\") pod \"insights-operator-5885fdb759-lmrmn\" (UID: \"f77a6b03-d007-46e3-852b-4b262feca301\") " pod="openshift-insights/insights-operator-5885fdb759-lmrmn"
Jul 28 04:15:37.123532 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:37.123117    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 04:15:37.123532 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.123154    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bfd2d3d8-f287-42d8-83c7-f82185c96735-trusted-ca\") pod \"console-operator-584dcb6b54-4glpr\" (UID: \"bfd2d3d8-f287-42d8-83c7-f82185c96735\") " pod="openshift-console-operator/console-operator-584dcb6b54-4glpr"
Jul 28 04:15:37.123532 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:37.123175    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edec740c-2287-4ba9-ae6b-0f9da7320a48-cluster-monitoring-operator-tls podName:edec740c-2287-4ba9-ae6b-0f9da7320a48 nodeName:}" failed. No retries permitted until 2026-07-28 04:15:37.623155996 +0000 UTC m=+238.013331996 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/edec740c-2287-4ba9-ae6b-0f9da7320a48-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-mfd2q" (UID: "edec740c-2287-4ba9-ae6b-0f9da7320a48") : secret "cluster-monitoring-operator-tls" not found
Jul 28 04:15:37.123532 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.123346    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/317c317c-02c3-4e8f-be22-5b81aea93846-config\") pod \"service-ca-operator-5d6c75f84f-bcwm5\" (UID: \"317c317c-02c3-4e8f-be22-5b81aea93846\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-bcwm5"
Jul 28 04:15:37.123532 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.123482    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bfd2d3d8-f287-42d8-83c7-f82185c96735-config\") pod \"console-operator-584dcb6b54-4glpr\" (UID: \"bfd2d3d8-f287-42d8-83c7-f82185c96735\") " pod="openshift-console-operator/console-operator-584dcb6b54-4glpr"
Jul 28 04:15:37.123928 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.123905    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/edec740c-2287-4ba9-ae6b-0f9da7320a48-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-mfd2q\" (UID: \"edec740c-2287-4ba9-ae6b-0f9da7320a48\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-mfd2q"
Jul 28 04:15:37.124442 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.124420    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bfd2d3d8-f287-42d8-83c7-f82185c96735-serving-cert\") pod \"console-operator-584dcb6b54-4glpr\" (UID: \"bfd2d3d8-f287-42d8-83c7-f82185c96735\") " pod="openshift-console-operator/console-operator-584dcb6b54-4glpr"
Jul 28 04:15:37.124888 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.124868    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/317c317c-02c3-4e8f-be22-5b81aea93846-serving-cert\") pod \"service-ca-operator-5d6c75f84f-bcwm5\" (UID: \"317c317c-02c3-4e8f-be22-5b81aea93846\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-bcwm5"
Jul 28 04:15:37.125253 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.125237    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f77a6b03-d007-46e3-852b-4b262feca301-serving-cert\") pod \"insights-operator-5885fdb759-lmrmn\" (UID: \"f77a6b03-d007-46e3-852b-4b262feca301\") " pod="openshift-insights/insights-operator-5885fdb759-lmrmn"
Jul 28 04:15:37.131721 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.131692    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hjwf\" (UniqueName: \"kubernetes.io/projected/f77a6b03-d007-46e3-852b-4b262feca301-kube-api-access-7hjwf\") pod \"insights-operator-5885fdb759-lmrmn\" (UID: \"f77a6b03-d007-46e3-852b-4b262feca301\") " pod="openshift-insights/insights-operator-5885fdb759-lmrmn"
Jul 28 04:15:37.132186 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.132160    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89mdw\" (UniqueName: \"kubernetes.io/projected/edec740c-2287-4ba9-ae6b-0f9da7320a48-kube-api-access-89mdw\") pod \"cluster-monitoring-operator-548b489cb8-mfd2q\" (UID: \"edec740c-2287-4ba9-ae6b-0f9da7320a48\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-mfd2q"
Jul 28 04:15:37.132283 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.132260    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g67j\" (UniqueName: \"kubernetes.io/projected/317c317c-02c3-4e8f-be22-5b81aea93846-kube-api-access-9g67j\") pod \"service-ca-operator-5d6c75f84f-bcwm5\" (UID: \"317c317c-02c3-4e8f-be22-5b81aea93846\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-bcwm5"
Jul 28 04:15:37.132376 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.132359    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq9ph\" (UniqueName: \"kubernetes.io/projected/bfd2d3d8-f287-42d8-83c7-f82185c96735-kube-api-access-zq9ph\") pod \"console-operator-584dcb6b54-4glpr\" (UID: \"bfd2d3d8-f287-42d8-83c7-f82185c96735\") " pod="openshift-console-operator/console-operator-584dcb6b54-4glpr"
Jul 28 04:15:37.223329 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.223260    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c7f7f74-e140-4d88-bc93-c029ae1c18a7-metrics-certs\") pod \"router-default-95c8f4d49-5ncng\" (UID: \"5c7f7f74-e140-4d88-bc93-c029ae1c18a7\") " pod="openshift-ingress/router-default-95c8f4d49-5ncng"
Jul 28 04:15:37.223329 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.223303    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5c7f7f74-e140-4d88-bc93-c029ae1c18a7-default-certificate\") pod \"router-default-95c8f4d49-5ncng\" (UID: \"5c7f7f74-e140-4d88-bc93-c029ae1c18a7\") " pod="openshift-ingress/router-default-95c8f4d49-5ncng"
Jul 28 04:15:37.223329 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.223323    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ddtd\" (UniqueName: \"kubernetes.io/projected/5c7f7f74-e140-4d88-bc93-c029ae1c18a7-kube-api-access-8ddtd\") pod \"router-default-95c8f4d49-5ncng\" (UID: \"5c7f7f74-e140-4d88-bc93-c029ae1c18a7\") " pod="openshift-ingress/router-default-95c8f4d49-5ncng"
Jul 28 04:15:37.223507 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.223342    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-622g6\" (UniqueName: \"kubernetes.io/projected/3837b644-a03e-498c-9033-1ed2e1d894d0-kube-api-access-622g6\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-d2lpq\" (UID: \"3837b644-a03e-498c-9033-1ed2e1d894d0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-d2lpq"
Jul 28 04:15:37.223507 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.223371    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3837b644-a03e-498c-9033-1ed2e1d894d0-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-d2lpq\" (UID: \"3837b644-a03e-498c-9033-1ed2e1d894d0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-d2lpq"
Jul 28 04:15:37.223507 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.223386    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c7f7f74-e140-4d88-bc93-c029ae1c18a7-service-ca-bundle\") pod \"router-default-95c8f4d49-5ncng\" (UID: \"5c7f7f74-e140-4d88-bc93-c029ae1c18a7\") " pod="openshift-ingress/router-default-95c8f4d49-5ncng"
Jul 28 04:15:37.223507 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:37.223414    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 04:15:37.223507 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.223431    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5c7f7f74-e140-4d88-bc93-c029ae1c18a7-stats-auth\") pod \"router-default-95c8f4d49-5ncng\" (UID: \"5c7f7f74-e140-4d88-bc93-c029ae1c18a7\") " pod="openshift-ingress/router-default-95c8f4d49-5ncng"
Jul 28 04:15:37.223507 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.223458    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3837b644-a03e-498c-9033-1ed2e1d894d0-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-d2lpq\" (UID: \"3837b644-a03e-498c-9033-1ed2e1d894d0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-d2lpq"
Jul 28 04:15:37.223507 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:37.223481    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c7f7f74-e140-4d88-bc93-c029ae1c18a7-metrics-certs podName:5c7f7f74-e140-4d88-bc93-c029ae1c18a7 nodeName:}" failed. No retries permitted until 2026-07-28 04:15:37.723460209 +0000 UTC m=+238.113636208 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5c7f7f74-e140-4d88-bc93-c029ae1c18a7-metrics-certs") pod "router-default-95c8f4d49-5ncng" (UID: "5c7f7f74-e140-4d88-bc93-c029ae1c18a7") : secret "router-metrics-certs-default" not found
Jul 28 04:15:37.223809 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.223512    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qtdf\" (UniqueName: \"kubernetes.io/projected/1f989c12-30f3-4710-add3-12c15a9af79c-kube-api-access-6qtdf\") pod \"network-check-source-7d88cb55d8-wjm98\" (UID: \"1f989c12-30f3-4710-add3-12c15a9af79c\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-wjm98"
Jul 28 04:15:37.223809 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:37.223579    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5c7f7f74-e140-4d88-bc93-c029ae1c18a7-service-ca-bundle podName:5c7f7f74-e140-4d88-bc93-c029ae1c18a7 nodeName:}" failed. No retries permitted until 2026-07-28 04:15:37.723560038 +0000 UTC m=+238.113736056 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5c7f7f74-e140-4d88-bc93-c029ae1c18a7-service-ca-bundle") pod "router-default-95c8f4d49-5ncng" (UID: "5c7f7f74-e140-4d88-bc93-c029ae1c18a7") : configmap references non-existent config key: service-ca.crt
Jul 28 04:15:37.224029 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.224011    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3837b644-a03e-498c-9033-1ed2e1d894d0-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-d2lpq\" (UID: \"3837b644-a03e-498c-9033-1ed2e1d894d0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-d2lpq"
Jul 28 04:15:37.225649 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.225627    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3837b644-a03e-498c-9033-1ed2e1d894d0-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-d2lpq\" (UID: \"3837b644-a03e-498c-9033-1ed2e1d894d0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-d2lpq"
Jul 28 04:15:37.225766 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.225742    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/5c7f7f74-e140-4d88-bc93-c029ae1c18a7-stats-auth\") pod \"router-default-95c8f4d49-5ncng\" (UID: \"5c7f7f74-e140-4d88-bc93-c029ae1c18a7\") " pod="openshift-ingress/router-default-95c8f4d49-5ncng"
Jul 28 04:15:37.225872 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.225854    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/5c7f7f74-e140-4d88-bc93-c029ae1c18a7-default-certificate\") pod \"router-default-95c8f4d49-5ncng\" (UID: \"5c7f7f74-e140-4d88-bc93-c029ae1c18a7\") " pod="openshift-ingress/router-default-95c8f4d49-5ncng"
Jul 28 04:15:37.231561 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.231516    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ddtd\" (UniqueName: \"kubernetes.io/projected/5c7f7f74-e140-4d88-bc93-c029ae1c18a7-kube-api-access-8ddtd\") pod \"router-default-95c8f4d49-5ncng\" (UID: \"5c7f7f74-e140-4d88-bc93-c029ae1c18a7\") " pod="openshift-ingress/router-default-95c8f4d49-5ncng"
Jul 28 04:15:37.231857 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.231822    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qtdf\" (UniqueName: \"kubernetes.io/projected/1f989c12-30f3-4710-add3-12c15a9af79c-kube-api-access-6qtdf\") pod \"network-check-source-7d88cb55d8-wjm98\" (UID: \"1f989c12-30f3-4710-add3-12c15a9af79c\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-wjm98"
Jul 28 04:15:37.231956 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.231866    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-622g6\" (UniqueName: \"kubernetes.io/projected/3837b644-a03e-498c-9033-1ed2e1d894d0-kube-api-access-622g6\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-d2lpq\" (UID: \"3837b644-a03e-498c-9033-1ed2e1d894d0\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-d2lpq"
Jul 28 04:15:37.237018 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.236996    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-lmrmn"
Jul 28 04:15:37.245076 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.245057    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-bcwm5"
Jul 28 04:15:37.258956 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.258932    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-4glpr"
Jul 28 04:15:37.325190 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.324950    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-wjm98"
Jul 28 04:15:37.333937 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.333707    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-d2lpq"
Jul 28 04:15:37.381047 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.380638    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-lmrmn"]
Jul 28 04:15:37.389277 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:15:37.389225    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf77a6b03_d007_46e3_852b_4b262feca301.slice/crio-899683960dc6a87cf109ace2a586b970ec600f5d77d13885e5688db415027399 WatchSource:0}: Error finding container 899683960dc6a87cf109ace2a586b970ec600f5d77d13885e5688db415027399: Status 404 returned error can't find the container with id 899683960dc6a87cf109ace2a586b970ec600f5d77d13885e5688db415027399
Jul 28 04:15:37.402657 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.402405    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-bcwm5"]
Jul 28 04:15:37.409429 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:15:37.409396    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod317c317c_02c3_4e8f_be22_5b81aea93846.slice/crio-e9fa869b4f9bca5e448a2951252b369711176ef2e4631626ddce3c36d5044a36 WatchSource:0}: Error finding container e9fa869b4f9bca5e448a2951252b369711176ef2e4631626ddce3c36d5044a36: Status 404 returned error can't find the container with id e9fa869b4f9bca5e448a2951252b369711176ef2e4631626ddce3c36d5044a36
Jul 28 04:15:37.427753 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.427724    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-4glpr"]
Jul 28 04:15:37.431336 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:15:37.431307    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfd2d3d8_f287_42d8_83c7_f82185c96735.slice/crio-0582112b958ea795be72ffdeb20fa612074675abb7db7564c9b81c41a40ce41b WatchSource:0}: Error finding container 0582112b958ea795be72ffdeb20fa612074675abb7db7564c9b81c41a40ce41b: Status 404 returned error can't find the container with id 0582112b958ea795be72ffdeb20fa612074675abb7db7564c9b81c41a40ce41b
Jul 28 04:15:37.475325 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.475173    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-wjm98"]
Jul 28 04:15:37.475948 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:15:37.475851    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f989c12_30f3_4710_add3_12c15a9af79c.slice/crio-83b72e331cabc817df7a6e2bdda1d8dcf35487da8f1bd584be8ba796e4973c3a WatchSource:0}: Error finding container 83b72e331cabc817df7a6e2bdda1d8dcf35487da8f1bd584be8ba796e4973c3a: Status 404 returned error can't find the container with id 83b72e331cabc817df7a6e2bdda1d8dcf35487da8f1bd584be8ba796e4973c3a
Jul 28 04:15:37.487724 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.487667    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-d2lpq"]
Jul 28 04:15:37.489884 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:15:37.489858    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3837b644_a03e_498c_9033_1ed2e1d894d0.slice/crio-2ae9d2b7578d8316cee975662226a4c4b0d81387add0fd139b1da342ed634769 WatchSource:0}: Error finding container 2ae9d2b7578d8316cee975662226a4c4b0d81387add0fd139b1da342ed634769: Status 404 returned error can't find the container with id 2ae9d2b7578d8316cee975662226a4c4b0d81387add0fd139b1da342ed634769
Jul 28 04:15:37.627652 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.627610    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/edec740c-2287-4ba9-ae6b-0f9da7320a48-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-mfd2q\" (UID: \"edec740c-2287-4ba9-ae6b-0f9da7320a48\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-mfd2q"
Jul 28 04:15:37.627818 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:37.627761    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 04:15:37.627897 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:37.627825    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edec740c-2287-4ba9-ae6b-0f9da7320a48-cluster-monitoring-operator-tls podName:edec740c-2287-4ba9-ae6b-0f9da7320a48 nodeName:}" failed. No retries permitted until 2026-07-28 04:15:38.62780722 +0000 UTC m=+239.017983229 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/edec740c-2287-4ba9-ae6b-0f9da7320a48-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-mfd2q" (UID: "edec740c-2287-4ba9-ae6b-0f9da7320a48") : secret "cluster-monitoring-operator-tls" not found
Jul 28 04:15:37.728708 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.728623    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c7f7f74-e140-4d88-bc93-c029ae1c18a7-metrics-certs\") pod \"router-default-95c8f4d49-5ncng\" (UID: \"5c7f7f74-e140-4d88-bc93-c029ae1c18a7\") " pod="openshift-ingress/router-default-95c8f4d49-5ncng"
Jul 28 04:15:37.728708 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.728676    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c7f7f74-e140-4d88-bc93-c029ae1c18a7-service-ca-bundle\") pod \"router-default-95c8f4d49-5ncng\" (UID: \"5c7f7f74-e140-4d88-bc93-c029ae1c18a7\") " pod="openshift-ingress/router-default-95c8f4d49-5ncng"
Jul 28 04:15:37.728909 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:37.728773    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 04:15:37.728909 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:37.728793    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5c7f7f74-e140-4d88-bc93-c029ae1c18a7-service-ca-bundle podName:5c7f7f74-e140-4d88-bc93-c029ae1c18a7 nodeName:}" failed. No retries permitted until 2026-07-28 04:15:38.728778089 +0000 UTC m=+239.118954088 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5c7f7f74-e140-4d88-bc93-c029ae1c18a7-service-ca-bundle") pod "router-default-95c8f4d49-5ncng" (UID: "5c7f7f74-e140-4d88-bc93-c029ae1c18a7") : configmap references non-existent config key: service-ca.crt
Jul 28 04:15:37.728909 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:37.728824    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c7f7f74-e140-4d88-bc93-c029ae1c18a7-metrics-certs podName:5c7f7f74-e140-4d88-bc93-c029ae1c18a7 nodeName:}" failed. No retries permitted until 2026-07-28 04:15:38.728810417 +0000 UTC m=+239.118986416 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5c7f7f74-e140-4d88-bc93-c029ae1c18a7-metrics-certs") pod "router-default-95c8f4d49-5ncng" (UID: "5c7f7f74-e140-4d88-bc93-c029ae1c18a7") : secret "router-metrics-certs-default" not found
Jul 28 04:15:37.850800 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.850763    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-d2lpq" event={"ID":"3837b644-a03e-498c-9033-1ed2e1d894d0","Type":"ContainerStarted","Data":"2ae9d2b7578d8316cee975662226a4c4b0d81387add0fd139b1da342ed634769"}
Jul 28 04:15:37.852285 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.852256    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-wjm98" event={"ID":"1f989c12-30f3-4710-add3-12c15a9af79c","Type":"ContainerStarted","Data":"f47cfb827b341e2472a129817b3d05a045b6a8d91e65d5eb89ab5b3fae1d19cf"}
Jul 28 04:15:37.852423 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.852291    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-wjm98" event={"ID":"1f989c12-30f3-4710-add3-12c15a9af79c","Type":"ContainerStarted","Data":"83b72e331cabc817df7a6e2bdda1d8dcf35487da8f1bd584be8ba796e4973c3a"}
Jul 28 04:15:37.853438 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.853420    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-lmrmn" event={"ID":"f77a6b03-d007-46e3-852b-4b262feca301","Type":"ContainerStarted","Data":"899683960dc6a87cf109ace2a586b970ec600f5d77d13885e5688db415027399"}
Jul 28 04:15:37.854443 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.854427    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-4glpr" event={"ID":"bfd2d3d8-f287-42d8-83c7-f82185c96735","Type":"ContainerStarted","Data":"0582112b958ea795be72ffdeb20fa612074675abb7db7564c9b81c41a40ce41b"}
Jul 28 04:15:37.855465 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.855443    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-bcwm5" event={"ID":"317c317c-02c3-4e8f-be22-5b81aea93846","Type":"ContainerStarted","Data":"e9fa869b4f9bca5e448a2951252b369711176ef2e4631626ddce3c36d5044a36"}
Jul 28 04:15:37.867181 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:37.867138    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-wjm98" podStartSLOduration=0.867125175 podStartE2EDuration="867.125175ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:15:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:15:37.866797174 +0000 UTC m=+238.256973194" watchObservedRunningTime="2026-07-28 04:15:37.867125175 +0000 UTC m=+238.257301195"
Jul 28 04:15:38.637566 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:38.637524    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/edec740c-2287-4ba9-ae6b-0f9da7320a48-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-mfd2q\" (UID: \"edec740c-2287-4ba9-ae6b-0f9da7320a48\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-mfd2q"
Jul 28 04:15:38.638076 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:38.637677    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 04:15:38.638076 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:38.637748    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edec740c-2287-4ba9-ae6b-0f9da7320a48-cluster-monitoring-operator-tls podName:edec740c-2287-4ba9-ae6b-0f9da7320a48 nodeName:}" failed. No retries permitted until 2026-07-28 04:15:40.637728268 +0000 UTC m=+241.027904281 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/edec740c-2287-4ba9-ae6b-0f9da7320a48-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-mfd2q" (UID: "edec740c-2287-4ba9-ae6b-0f9da7320a48") : secret "cluster-monitoring-operator-tls" not found
Jul 28 04:15:38.740045 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:38.740009    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c7f7f74-e140-4d88-bc93-c029ae1c18a7-metrics-certs\") pod \"router-default-95c8f4d49-5ncng\" (UID: \"5c7f7f74-e140-4d88-bc93-c029ae1c18a7\") " pod="openshift-ingress/router-default-95c8f4d49-5ncng"
Jul 28 04:15:38.740423 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:38.740088    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c7f7f74-e140-4d88-bc93-c029ae1c18a7-service-ca-bundle\") pod \"router-default-95c8f4d49-5ncng\" (UID: \"5c7f7f74-e140-4d88-bc93-c029ae1c18a7\") " pod="openshift-ingress/router-default-95c8f4d49-5ncng"
Jul 28 04:15:38.740423 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:38.740269    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5c7f7f74-e140-4d88-bc93-c029ae1c18a7-service-ca-bundle podName:5c7f7f74-e140-4d88-bc93-c029ae1c18a7 nodeName:}" failed. No retries permitted until 2026-07-28 04:15:40.740251331 +0000 UTC m=+241.130427333 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5c7f7f74-e140-4d88-bc93-c029ae1c18a7-service-ca-bundle") pod "router-default-95c8f4d49-5ncng" (UID: "5c7f7f74-e140-4d88-bc93-c029ae1c18a7") : configmap references non-existent config key: service-ca.crt
Jul 28 04:15:38.740668 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:38.740649    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 04:15:38.740748 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:38.740699    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c7f7f74-e140-4d88-bc93-c029ae1c18a7-metrics-certs podName:5c7f7f74-e140-4d88-bc93-c029ae1c18a7 nodeName:}" failed. No retries permitted until 2026-07-28 04:15:40.740686138 +0000 UTC m=+241.130862141 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5c7f7f74-e140-4d88-bc93-c029ae1c18a7-metrics-certs") pod "router-default-95c8f4d49-5ncng" (UID: "5c7f7f74-e140-4d88-bc93-c029ae1c18a7") : secret "router-metrics-certs-default" not found
Jul 28 04:15:40.658826 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:40.658790    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/edec740c-2287-4ba9-ae6b-0f9da7320a48-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-mfd2q\" (UID: \"edec740c-2287-4ba9-ae6b-0f9da7320a48\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-mfd2q"
Jul 28 04:15:40.659196 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:40.658956    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 04:15:40.659196 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:40.659022    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edec740c-2287-4ba9-ae6b-0f9da7320a48-cluster-monitoring-operator-tls podName:edec740c-2287-4ba9-ae6b-0f9da7320a48 nodeName:}" failed. No retries permitted until 2026-07-28 04:15:44.65900687 +0000 UTC m=+245.049182868 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/edec740c-2287-4ba9-ae6b-0f9da7320a48-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-mfd2q" (UID: "edec740c-2287-4ba9-ae6b-0f9da7320a48") : secret "cluster-monitoring-operator-tls" not found
Jul 28 04:15:40.759855 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:40.759810    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c7f7f74-e140-4d88-bc93-c029ae1c18a7-service-ca-bundle\") pod \"router-default-95c8f4d49-5ncng\" (UID: \"5c7f7f74-e140-4d88-bc93-c029ae1c18a7\") " pod="openshift-ingress/router-default-95c8f4d49-5ncng"
Jul 28 04:15:40.760031 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:40.759947    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c7f7f74-e140-4d88-bc93-c029ae1c18a7-metrics-certs\") pod \"router-default-95c8f4d49-5ncng\" (UID: \"5c7f7f74-e140-4d88-bc93-c029ae1c18a7\") " pod="openshift-ingress/router-default-95c8f4d49-5ncng"
Jul 28 04:15:40.760031 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:40.760024    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5c7f7f74-e140-4d88-bc93-c029ae1c18a7-service-ca-bundle podName:5c7f7f74-e140-4d88-bc93-c029ae1c18a7 nodeName:}" failed. No retries permitted until 2026-07-28 04:15:44.760003635 +0000 UTC m=+245.150179636 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5c7f7f74-e140-4d88-bc93-c029ae1c18a7-service-ca-bundle") pod "router-default-95c8f4d49-5ncng" (UID: "5c7f7f74-e140-4d88-bc93-c029ae1c18a7") : configmap references non-existent config key: service-ca.crt
Jul 28 04:15:40.760155 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:40.760102    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 04:15:40.760210 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:40.760174    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c7f7f74-e140-4d88-bc93-c029ae1c18a7-metrics-certs podName:5c7f7f74-e140-4d88-bc93-c029ae1c18a7 nodeName:}" failed. No retries permitted until 2026-07-28 04:15:44.760156138 +0000 UTC m=+245.150332138 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5c7f7f74-e140-4d88-bc93-c029ae1c18a7-metrics-certs") pod "router-default-95c8f4d49-5ncng" (UID: "5c7f7f74-e140-4d88-bc93-c029ae1c18a7") : secret "router-metrics-certs-default" not found
Jul 28 04:15:41.870106 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:41.870076    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4glpr_bfd2d3d8-f287-42d8-83c7-f82185c96735/console-operator/0.log"
Jul 28 04:15:41.870561 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:41.870125    2566 generic.go:356] "Generic (PLEG): container finished" podID="bfd2d3d8-f287-42d8-83c7-f82185c96735" containerID="bec903fdc51781dd748b64306ddd10e4f902ad885a4f013b83cb7a5d4a391776" exitCode=255
Jul 28 04:15:41.870561 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:41.870205    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-4glpr" event={"ID":"bfd2d3d8-f287-42d8-83c7-f82185c96735","Type":"ContainerDied","Data":"bec903fdc51781dd748b64306ddd10e4f902ad885a4f013b83cb7a5d4a391776"}
Jul 28 04:15:41.870561 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:41.870470    2566 scope.go:117] "RemoveContainer" containerID="bec903fdc51781dd748b64306ddd10e4f902ad885a4f013b83cb7a5d4a391776"
Jul 28 04:15:41.872090 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:41.872064    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-bcwm5" event={"ID":"317c317c-02c3-4e8f-be22-5b81aea93846","Type":"ContainerStarted","Data":"d742b9c59c22f09980884fbd7f2ab6aa7557f820575a5b8e0c7c88ae63391eb0"}
Jul 28 04:15:41.874076 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:41.874036    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-d2lpq" event={"ID":"3837b644-a03e-498c-9033-1ed2e1d894d0","Type":"ContainerStarted","Data":"93dae67459983476f248788a47320c3e232911cba81b872f450ca944dc7248dc"}
Jul 28 04:15:41.875581 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:41.875562    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-lmrmn" event={"ID":"f77a6b03-d007-46e3-852b-4b262feca301","Type":"ContainerStarted","Data":"d63987c0d58ddb213f3007d703025d26e1312afe8b5e0884e5a6bf3b71ff2326"}
Jul 28 04:15:41.902018 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:41.901970    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-bcwm5" podStartSLOduration=2.087367516 podStartE2EDuration="5.901953456s" totalImagesPullingTime="3.81458594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:15:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:15:41.900756996 +0000 UTC m=+242.290933018" watchObservedRunningTime="2026-07-28 04:15:41.901953456 +0000 UTC m=+242.292129479"
Jul 28 04:15:41.919985 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:41.919928    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-lmrmn" podStartSLOduration=2.092803287 podStartE2EDuration="5.919907752s" totalImagesPullingTime="3.827104465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:15:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:15:41.918723149 +0000 UTC m=+242.308899172" watchObservedRunningTime="2026-07-28 04:15:41.919907752 +0000 UTC m=+242.310083774"
Jul 28 04:15:41.937435 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:41.937383    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-d2lpq" podStartSLOduration=1.209864947 podStartE2EDuration="4.937362886s" totalImagesPullingTime="3.727497939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:15:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:15:41.935962086 +0000 UTC m=+242.326138108" watchObservedRunningTime="2026-07-28 04:15:41.937362886 +0000 UTC m=+242.327538908"
Jul 28 04:15:42.555628 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:42.555592    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-k8fxt"]
Jul 28 04:15:42.559300 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:42.559272    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-k8fxt"
Jul 28 04:15:42.562608 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:42.562014    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 04:15:42.562608 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:42.562313    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tkv7l\""
Jul 28 04:15:42.562608 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:42.562485    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 04:15:42.571067 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:42.571044    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-k8fxt"]
Jul 28 04:15:42.678986 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:42.678948    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/83539dc8-61dc-4d3c-aff3-9ac4a543abd9-crio-socket\") pod \"insights-runtime-extractor-k8fxt\" (UID: \"83539dc8-61dc-4d3c-aff3-9ac4a543abd9\") " pod="openshift-insights/insights-runtime-extractor-k8fxt"
Jul 28 04:15:42.679136 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:42.679034    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/83539dc8-61dc-4d3c-aff3-9ac4a543abd9-data-volume\") pod \"insights-runtime-extractor-k8fxt\" (UID: \"83539dc8-61dc-4d3c-aff3-9ac4a543abd9\") " pod="openshift-insights/insights-runtime-extractor-k8fxt"
Jul 28 04:15:42.679136 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:42.679079    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/83539dc8-61dc-4d3c-aff3-9ac4a543abd9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k8fxt\" (UID: \"83539dc8-61dc-4d3c-aff3-9ac4a543abd9\") " pod="openshift-insights/insights-runtime-extractor-k8fxt"
Jul 28 04:15:42.679233 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:42.679167    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/83539dc8-61dc-4d3c-aff3-9ac4a543abd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k8fxt\" (UID: \"83539dc8-61dc-4d3c-aff3-9ac4a543abd9\") " pod="openshift-insights/insights-runtime-extractor-k8fxt"
Jul 28 04:15:42.679233 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:42.679213    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95rvv\" (UniqueName: \"kubernetes.io/projected/83539dc8-61dc-4d3c-aff3-9ac4a543abd9-kube-api-access-95rvv\") pod \"insights-runtime-extractor-k8fxt\" (UID: \"83539dc8-61dc-4d3c-aff3-9ac4a543abd9\") " pod="openshift-insights/insights-runtime-extractor-k8fxt"
Jul 28 04:15:42.779809 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:42.779779    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/83539dc8-61dc-4d3c-aff3-9ac4a543abd9-crio-socket\") pod \"insights-runtime-extractor-k8fxt\" (UID: \"83539dc8-61dc-4d3c-aff3-9ac4a543abd9\") " pod="openshift-insights/insights-runtime-extractor-k8fxt"
Jul 28 04:15:42.780005 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:42.779832    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/83539dc8-61dc-4d3c-aff3-9ac4a543abd9-data-volume\") pod \"insights-runtime-extractor-k8fxt\" (UID: \"83539dc8-61dc-4d3c-aff3-9ac4a543abd9\") " pod="openshift-insights/insights-runtime-extractor-k8fxt"
Jul 28 04:15:42.780005 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:42.779941    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/83539dc8-61dc-4d3c-aff3-9ac4a543abd9-crio-socket\") pod \"insights-runtime-extractor-k8fxt\" (UID: \"83539dc8-61dc-4d3c-aff3-9ac4a543abd9\") " pod="openshift-insights/insights-runtime-extractor-k8fxt"
Jul 28 04:15:42.780005 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:42.780002    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/83539dc8-61dc-4d3c-aff3-9ac4a543abd9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k8fxt\" (UID: \"83539dc8-61dc-4d3c-aff3-9ac4a543abd9\") " pod="openshift-insights/insights-runtime-extractor-k8fxt"
Jul 28 04:15:42.780168 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:42.780058    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/83539dc8-61dc-4d3c-aff3-9ac4a543abd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k8fxt\" (UID: \"83539dc8-61dc-4d3c-aff3-9ac4a543abd9\") " pod="openshift-insights/insights-runtime-extractor-k8fxt"
Jul 28 04:15:42.780168 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:42.780095    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95rvv\" (UniqueName: \"kubernetes.io/projected/83539dc8-61dc-4d3c-aff3-9ac4a543abd9-kube-api-access-95rvv\") pod \"insights-runtime-extractor-k8fxt\" (UID: \"83539dc8-61dc-4d3c-aff3-9ac4a543abd9\") " pod="openshift-insights/insights-runtime-extractor-k8fxt"
Jul 28 04:15:42.780168 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:42.780104    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 04:15:42.780168 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:42.780097    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/83539dc8-61dc-4d3c-aff3-9ac4a543abd9-data-volume\") pod \"insights-runtime-extractor-k8fxt\" (UID: \"83539dc8-61dc-4d3c-aff3-9ac4a543abd9\") " pod="openshift-insights/insights-runtime-extractor-k8fxt"
Jul 28 04:15:42.780168 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:42.780160    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 04:15:42.780347 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:42.780172    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/83539dc8-61dc-4d3c-aff3-9ac4a543abd9-kube-rbac-proxy-cm podName:83539dc8-61dc-4d3c-aff3-9ac4a543abd9 nodeName:}" failed. No retries permitted until 2026-07-28 04:15:43.280153186 +0000 UTC m=+243.670329197 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/83539dc8-61dc-4d3c-aff3-9ac4a543abd9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k8fxt" (UID: "83539dc8-61dc-4d3c-aff3-9ac4a543abd9") : configmap "kube-rbac-proxy" not found
Jul 28 04:15:42.780347 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:42.780202    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83539dc8-61dc-4d3c-aff3-9ac4a543abd9-insights-runtime-extractor-tls podName:83539dc8-61dc-4d3c-aff3-9ac4a543abd9 nodeName:}" failed. No retries permitted until 2026-07-28 04:15:43.280191479 +0000 UTC m=+243.670367478 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/83539dc8-61dc-4d3c-aff3-9ac4a543abd9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k8fxt" (UID: "83539dc8-61dc-4d3c-aff3-9ac4a543abd9") : secret "insights-runtime-extractor-tls" not found
Jul 28 04:15:42.787886 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:42.787859    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95rvv\" (UniqueName: \"kubernetes.io/projected/83539dc8-61dc-4d3c-aff3-9ac4a543abd9-kube-api-access-95rvv\") pod \"insights-runtime-extractor-k8fxt\" (UID: \"83539dc8-61dc-4d3c-aff3-9ac4a543abd9\") " pod="openshift-insights/insights-runtime-extractor-k8fxt"
Jul 28 04:15:42.880148 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:42.880050    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4glpr_bfd2d3d8-f287-42d8-83c7-f82185c96735/console-operator/1.log"
Jul 28 04:15:42.880498 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:42.880482    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4glpr_bfd2d3d8-f287-42d8-83c7-f82185c96735/console-operator/0.log"
Jul 28 04:15:42.880547 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:42.880517    2566 generic.go:356] "Generic (PLEG): container finished" podID="bfd2d3d8-f287-42d8-83c7-f82185c96735" containerID="4472c3365491b95babcf5961e2d9f8259fd4ff80934a67364ffa12ba05036d2b" exitCode=255
Jul 28 04:15:42.880597 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:42.880542    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-4glpr" event={"ID":"bfd2d3d8-f287-42d8-83c7-f82185c96735","Type":"ContainerDied","Data":"4472c3365491b95babcf5961e2d9f8259fd4ff80934a67364ffa12ba05036d2b"}
Jul 28 04:15:42.880597 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:42.880595    2566 scope.go:117] "RemoveContainer" containerID="bec903fdc51781dd748b64306ddd10e4f902ad885a4f013b83cb7a5d4a391776"
Jul 28 04:15:42.881253 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:42.881227    2566 scope.go:117] "RemoveContainer" containerID="4472c3365491b95babcf5961e2d9f8259fd4ff80934a67364ffa12ba05036d2b"
Jul 28 04:15:42.881379 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:42.881360    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-4glpr_openshift-console-operator(bfd2d3d8-f287-42d8-83c7-f82185c96735)\"" pod="openshift-console-operator/console-operator-584dcb6b54-4glpr" podUID="bfd2d3d8-f287-42d8-83c7-f82185c96735"
Jul 28 04:15:43.285760 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:43.285726    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/83539dc8-61dc-4d3c-aff3-9ac4a543abd9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k8fxt\" (UID: \"83539dc8-61dc-4d3c-aff3-9ac4a543abd9\") " pod="openshift-insights/insights-runtime-extractor-k8fxt"
Jul 28 04:15:43.285938 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:43.285778    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/83539dc8-61dc-4d3c-aff3-9ac4a543abd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k8fxt\" (UID: \"83539dc8-61dc-4d3c-aff3-9ac4a543abd9\") " pod="openshift-insights/insights-runtime-extractor-k8fxt"
Jul 28 04:15:43.285938 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:43.285885    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 04:15:43.286014 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:43.285949    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 04:15:43.286014 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:43.285955    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/83539dc8-61dc-4d3c-aff3-9ac4a543abd9-kube-rbac-proxy-cm podName:83539dc8-61dc-4d3c-aff3-9ac4a543abd9 nodeName:}" failed. No retries permitted until 2026-07-28 04:15:44.285938826 +0000 UTC m=+244.676114837 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/83539dc8-61dc-4d3c-aff3-9ac4a543abd9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k8fxt" (UID: "83539dc8-61dc-4d3c-aff3-9ac4a543abd9") : configmap "kube-rbac-proxy" not found
Jul 28 04:15:43.286014 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:43.286008    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83539dc8-61dc-4d3c-aff3-9ac4a543abd9-insights-runtime-extractor-tls podName:83539dc8-61dc-4d3c-aff3-9ac4a543abd9 nodeName:}" failed. No retries permitted until 2026-07-28 04:15:44.285994112 +0000 UTC m=+244.676170110 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/83539dc8-61dc-4d3c-aff3-9ac4a543abd9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k8fxt" (UID: "83539dc8-61dc-4d3c-aff3-9ac4a543abd9") : secret "insights-runtime-extractor-tls" not found
Jul 28 04:15:43.884851 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:43.884818    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4glpr_bfd2d3d8-f287-42d8-83c7-f82185c96735/console-operator/1.log"
Jul 28 04:15:43.885224 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:43.885156    2566 scope.go:117] "RemoveContainer" containerID="4472c3365491b95babcf5961e2d9f8259fd4ff80934a67364ffa12ba05036d2b"
Jul 28 04:15:43.885331 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:43.885311    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-4glpr_openshift-console-operator(bfd2d3d8-f287-42d8-83c7-f82185c96735)\"" pod="openshift-console-operator/console-operator-584dcb6b54-4glpr" podUID="bfd2d3d8-f287-42d8-83c7-f82185c96735"
Jul 28 04:15:44.291629 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:44.291597    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/83539dc8-61dc-4d3c-aff3-9ac4a543abd9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k8fxt\" (UID: \"83539dc8-61dc-4d3c-aff3-9ac4a543abd9\") " pod="openshift-insights/insights-runtime-extractor-k8fxt"
Jul 28 04:15:44.291797 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:44.291647    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/83539dc8-61dc-4d3c-aff3-9ac4a543abd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k8fxt\" (UID: \"83539dc8-61dc-4d3c-aff3-9ac4a543abd9\") " pod="openshift-insights/insights-runtime-extractor-k8fxt"
Jul 28 04:15:44.291797 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:44.291748    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 04:15:44.291944 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:44.291817    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 04:15:44.291944 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:44.291861    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/83539dc8-61dc-4d3c-aff3-9ac4a543abd9-kube-rbac-proxy-cm podName:83539dc8-61dc-4d3c-aff3-9ac4a543abd9 nodeName:}" failed. No retries permitted until 2026-07-28 04:15:46.291820753 +0000 UTC m=+246.681996763 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/83539dc8-61dc-4d3c-aff3-9ac4a543abd9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k8fxt" (UID: "83539dc8-61dc-4d3c-aff3-9ac4a543abd9") : configmap "kube-rbac-proxy" not found
Jul 28 04:15:44.291944 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:44.291885    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83539dc8-61dc-4d3c-aff3-9ac4a543abd9-insights-runtime-extractor-tls podName:83539dc8-61dc-4d3c-aff3-9ac4a543abd9 nodeName:}" failed. No retries permitted until 2026-07-28 04:15:46.29187421 +0000 UTC m=+246.682050210 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/83539dc8-61dc-4d3c-aff3-9ac4a543abd9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k8fxt" (UID: "83539dc8-61dc-4d3c-aff3-9ac4a543abd9") : secret "insights-runtime-extractor-tls" not found
Jul 28 04:15:44.694794 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:44.694703    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/edec740c-2287-4ba9-ae6b-0f9da7320a48-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-mfd2q\" (UID: \"edec740c-2287-4ba9-ae6b-0f9da7320a48\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-mfd2q"
Jul 28 04:15:44.694957 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:44.694858    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 04:15:44.694957 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:44.694922    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edec740c-2287-4ba9-ae6b-0f9da7320a48-cluster-monitoring-operator-tls podName:edec740c-2287-4ba9-ae6b-0f9da7320a48 nodeName:}" failed. No retries permitted until 2026-07-28 04:15:52.694906603 +0000 UTC m=+253.085082602 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/edec740c-2287-4ba9-ae6b-0f9da7320a48-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-mfd2q" (UID: "edec740c-2287-4ba9-ae6b-0f9da7320a48") : secret "cluster-monitoring-operator-tls" not found
Jul 28 04:15:44.795908 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:44.795877    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c7f7f74-e140-4d88-bc93-c029ae1c18a7-metrics-certs\") pod \"router-default-95c8f4d49-5ncng\" (UID: \"5c7f7f74-e140-4d88-bc93-c029ae1c18a7\") " pod="openshift-ingress/router-default-95c8f4d49-5ncng"
Jul 28 04:15:44.796077 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:44.795930    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c7f7f74-e140-4d88-bc93-c029ae1c18a7-service-ca-bundle\") pod \"router-default-95c8f4d49-5ncng\" (UID: \"5c7f7f74-e140-4d88-bc93-c029ae1c18a7\") " pod="openshift-ingress/router-default-95c8f4d49-5ncng"
Jul 28 04:15:44.796077 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:44.796034    2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 04:15:44.796167 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:44.796040    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5c7f7f74-e140-4d88-bc93-c029ae1c18a7-service-ca-bundle podName:5c7f7f74-e140-4d88-bc93-c029ae1c18a7 nodeName:}" failed. No retries permitted until 2026-07-28 04:15:52.796027305 +0000 UTC m=+253.186203305 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5c7f7f74-e140-4d88-bc93-c029ae1c18a7-service-ca-bundle") pod "router-default-95c8f4d49-5ncng" (UID: "5c7f7f74-e140-4d88-bc93-c029ae1c18a7") : configmap references non-existent config key: service-ca.crt
Jul 28 04:15:44.796167 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:44.796104    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c7f7f74-e140-4d88-bc93-c029ae1c18a7-metrics-certs podName:5c7f7f74-e140-4d88-bc93-c029ae1c18a7 nodeName:}" failed. No retries permitted until 2026-07-28 04:15:52.796086997 +0000 UTC m=+253.186263000 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5c7f7f74-e140-4d88-bc93-c029ae1c18a7-metrics-certs") pod "router-default-95c8f4d49-5ncng" (UID: "5c7f7f74-e140-4d88-bc93-c029ae1c18a7") : secret "router-metrics-certs-default" not found
Jul 28 04:15:45.183029 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:45.182998    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-9qd2n"]
Jul 28 04:15:45.187421 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:45.187399    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-9qd2n"
Jul 28 04:15:45.190052 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:45.190023    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 28 04:15:45.190165 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:45.190103    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-4kpbj\""
Jul 28 04:15:45.190165 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:45.190115    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 28 04:15:45.190165 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:45.190105    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 28 04:15:45.190332 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:45.190181    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 28 04:15:45.194905 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:45.194887    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-9qd2n"]
Jul 28 04:15:45.300822 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:45.300792    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e31885f0-d688-4340-8a38-299a71dd629d-signing-cabundle\") pod \"service-ca-658d7b7c78-9qd2n\" (UID: \"e31885f0-d688-4340-8a38-299a71dd629d\") " pod="openshift-service-ca/service-ca-658d7b7c78-9qd2n"
Jul 28 04:15:45.301002 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:45.300891    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s8qm\" (UniqueName: \"kubernetes.io/projected/e31885f0-d688-4340-8a38-299a71dd629d-kube-api-access-9s8qm\") pod \"service-ca-658d7b7c78-9qd2n\" (UID: \"e31885f0-d688-4340-8a38-299a71dd629d\") " pod="openshift-service-ca/service-ca-658d7b7c78-9qd2n"
Jul 28 04:15:45.301002 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:45.300930    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e31885f0-d688-4340-8a38-299a71dd629d-signing-key\") pod \"service-ca-658d7b7c78-9qd2n\" (UID: \"e31885f0-d688-4340-8a38-299a71dd629d\") " pod="openshift-service-ca/service-ca-658d7b7c78-9qd2n"
Jul 28 04:15:45.359575 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:45.359549    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tlsvx_068d70ee-9efb-49a5-94cb-2df7591a4652/dns-node-resolver/0.log"
Jul 28 04:15:45.401480 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:45.401439    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s8qm\" (UniqueName: \"kubernetes.io/projected/e31885f0-d688-4340-8a38-299a71dd629d-kube-api-access-9s8qm\") pod \"service-ca-658d7b7c78-9qd2n\" (UID: \"e31885f0-d688-4340-8a38-299a71dd629d\") " pod="openshift-service-ca/service-ca-658d7b7c78-9qd2n"
Jul 28 04:15:45.401606 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:45.401502    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e31885f0-d688-4340-8a38-299a71dd629d-signing-key\") pod \"service-ca-658d7b7c78-9qd2n\" (UID: \"e31885f0-d688-4340-8a38-299a71dd629d\") " pod="openshift-service-ca/service-ca-658d7b7c78-9qd2n"
Jul 28 04:15:45.401682 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:45.401652    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e31885f0-d688-4340-8a38-299a71dd629d-signing-cabundle\") pod \"service-ca-658d7b7c78-9qd2n\" (UID: \"e31885f0-d688-4340-8a38-299a71dd629d\") " pod="openshift-service-ca/service-ca-658d7b7c78-9qd2n"
Jul 28 04:15:45.402232 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:45.402211    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e31885f0-d688-4340-8a38-299a71dd629d-signing-cabundle\") pod \"service-ca-658d7b7c78-9qd2n\" (UID: \"e31885f0-d688-4340-8a38-299a71dd629d\") " pod="openshift-service-ca/service-ca-658d7b7c78-9qd2n"
Jul 28 04:15:45.403957 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:45.403933    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e31885f0-d688-4340-8a38-299a71dd629d-signing-key\") pod \"service-ca-658d7b7c78-9qd2n\" (UID: \"e31885f0-d688-4340-8a38-299a71dd629d\") " pod="openshift-service-ca/service-ca-658d7b7c78-9qd2n"
Jul 28 04:15:45.409050 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:45.409028    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s8qm\" (UniqueName: \"kubernetes.io/projected/e31885f0-d688-4340-8a38-299a71dd629d-kube-api-access-9s8qm\") pod \"service-ca-658d7b7c78-9qd2n\" (UID: \"e31885f0-d688-4340-8a38-299a71dd629d\") " pod="openshift-service-ca/service-ca-658d7b7c78-9qd2n"
Jul 28 04:15:45.497038 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:45.496962    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-9qd2n"
Jul 28 04:15:45.617448 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:45.617417    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-9qd2n"]
Jul 28 04:15:45.620075 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:15:45.620049    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode31885f0_d688_4340_8a38_299a71dd629d.slice/crio-073f338bb7188ce8c003309f551cffccb2ab940925c9427b510917e5b06e2ebf WatchSource:0}: Error finding container 073f338bb7188ce8c003309f551cffccb2ab940925c9427b510917e5b06e2ebf: Status 404 returned error can't find the container with id 073f338bb7188ce8c003309f551cffccb2ab940925c9427b510917e5b06e2ebf
Jul 28 04:15:45.892554 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:45.892519    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-9qd2n" event={"ID":"e31885f0-d688-4340-8a38-299a71dd629d","Type":"ContainerStarted","Data":"2030536fb246a9e4bb2d3dd953954a041b2b976fbda14b257ed1e23c075762e0"}
Jul 28 04:15:45.892720 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:45.892568    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-9qd2n" event={"ID":"e31885f0-d688-4340-8a38-299a71dd629d","Type":"ContainerStarted","Data":"073f338bb7188ce8c003309f551cffccb2ab940925c9427b510917e5b06e2ebf"}
Jul 28 04:15:45.909508 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:45.909452    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-9qd2n" podStartSLOduration=0.909429481 podStartE2EDuration="909.429481ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:15:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:15:45.90924562 +0000 UTC m=+246.299421641" watchObservedRunningTime="2026-07-28 04:15:45.909429481 +0000 UTC m=+246.299605503"
Jul 28 04:15:46.311182 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:46.311132    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/83539dc8-61dc-4d3c-aff3-9ac4a543abd9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k8fxt\" (UID: \"83539dc8-61dc-4d3c-aff3-9ac4a543abd9\") " pod="openshift-insights/insights-runtime-extractor-k8fxt"
Jul 28 04:15:46.311636 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:46.311209    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/83539dc8-61dc-4d3c-aff3-9ac4a543abd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k8fxt\" (UID: \"83539dc8-61dc-4d3c-aff3-9ac4a543abd9\") " pod="openshift-insights/insights-runtime-extractor-k8fxt"
Jul 28 04:15:46.311636 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:46.311249    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 04:15:46.311636 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:46.311318    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/83539dc8-61dc-4d3c-aff3-9ac4a543abd9-kube-rbac-proxy-cm podName:83539dc8-61dc-4d3c-aff3-9ac4a543abd9 nodeName:}" failed. No retries permitted until 2026-07-28 04:15:50.311303026 +0000 UTC m=+250.701479029 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/83539dc8-61dc-4d3c-aff3-9ac4a543abd9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k8fxt" (UID: "83539dc8-61dc-4d3c-aff3-9ac4a543abd9") : configmap "kube-rbac-proxy" not found
Jul 28 04:15:46.311636 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:46.311380    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 04:15:46.311636 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:46.311466    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83539dc8-61dc-4d3c-aff3-9ac4a543abd9-insights-runtime-extractor-tls podName:83539dc8-61dc-4d3c-aff3-9ac4a543abd9 nodeName:}" failed. No retries permitted until 2026-07-28 04:15:50.311447183 +0000 UTC m=+250.701623189 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/83539dc8-61dc-4d3c-aff3-9ac4a543abd9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k8fxt" (UID: "83539dc8-61dc-4d3c-aff3-9ac4a543abd9") : secret "insights-runtime-extractor-tls" not found
Jul 28 04:15:46.359039 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:46.359009    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-t6nfz_8cf586fe-2e9b-474b-8fa6-3ca6317e2b31/node-ca/0.log"
Jul 28 04:15:47.259574 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:47.259486    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-4glpr"
Jul 28 04:15:47.259730 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:47.259591    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-4glpr"
Jul 28 04:15:47.260025 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:47.260005    2566 scope.go:117] "RemoveContainer" containerID="4472c3365491b95babcf5961e2d9f8259fd4ff80934a67364ffa12ba05036d2b"
Jul 28 04:15:47.260218 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:47.260192    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-4glpr_openshift-console-operator(bfd2d3d8-f287-42d8-83c7-f82185c96735)\"" pod="openshift-console-operator/console-operator-584dcb6b54-4glpr" podUID="bfd2d3d8-f287-42d8-83c7-f82185c96735"
Jul 28 04:15:47.359692 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:47.359646    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-z8gxf_9cb032cd-6003-4249-a65d-95d48fea8a02/serve-healthcheck-canary/0.log"
Jul 28 04:15:47.960137 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:47.960100    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-d2lpq_3837b644-a03e-498c-9033-1ed2e1d894d0/kube-storage-version-migrator-operator/0.log"
Jul 28 04:15:50.343298 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:50.343262    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/83539dc8-61dc-4d3c-aff3-9ac4a543abd9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k8fxt\" (UID: \"83539dc8-61dc-4d3c-aff3-9ac4a543abd9\") " pod="openshift-insights/insights-runtime-extractor-k8fxt"
Jul 28 04:15:50.343775 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:50.343309    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/83539dc8-61dc-4d3c-aff3-9ac4a543abd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k8fxt\" (UID: \"83539dc8-61dc-4d3c-aff3-9ac4a543abd9\") " pod="openshift-insights/insights-runtime-extractor-k8fxt"
Jul 28 04:15:50.343775 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:50.343400    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 04:15:50.343775 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:50.343410    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 04:15:50.343775 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:50.343494    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/83539dc8-61dc-4d3c-aff3-9ac4a543abd9-kube-rbac-proxy-cm podName:83539dc8-61dc-4d3c-aff3-9ac4a543abd9 nodeName:}" failed. No retries permitted until 2026-07-28 04:15:58.343475113 +0000 UTC m=+258.733651116 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/83539dc8-61dc-4d3c-aff3-9ac4a543abd9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-k8fxt" (UID: "83539dc8-61dc-4d3c-aff3-9ac4a543abd9") : configmap "kube-rbac-proxy" not found
Jul 28 04:15:50.343775 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:50.343538    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/83539dc8-61dc-4d3c-aff3-9ac4a543abd9-insights-runtime-extractor-tls podName:83539dc8-61dc-4d3c-aff3-9ac4a543abd9 nodeName:}" failed. No retries permitted until 2026-07-28 04:15:58.343523567 +0000 UTC m=+258.733699565 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/83539dc8-61dc-4d3c-aff3-9ac4a543abd9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-k8fxt" (UID: "83539dc8-61dc-4d3c-aff3-9ac4a543abd9") : secret "insights-runtime-extractor-tls" not found
Jul 28 04:15:51.957252 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:51.957217    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6107883-62a6-4a82-9e04-ed46057a6b0e-metrics-certs\") pod \"network-metrics-daemon-l6p44\" (UID: \"e6107883-62a6-4a82-9e04-ed46057a6b0e\") " pod="openshift-multus/network-metrics-daemon-l6p44"
Jul 28 04:15:51.957632 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:51.957382    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 28 04:15:51.957632 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:51.957441    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6107883-62a6-4a82-9e04-ed46057a6b0e-metrics-certs podName:e6107883-62a6-4a82-9e04-ed46057a6b0e nodeName:}" failed. No retries permitted until 2026-07-28 04:17:53.957422807 +0000 UTC m=+374.347598812 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e6107883-62a6-4a82-9e04-ed46057a6b0e-metrics-certs") pod "network-metrics-daemon-l6p44" (UID: "e6107883-62a6-4a82-9e04-ed46057a6b0e") : secret "metrics-daemon-secret" not found
Jul 28 04:15:52.764919 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:52.764881    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/edec740c-2287-4ba9-ae6b-0f9da7320a48-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-mfd2q\" (UID: \"edec740c-2287-4ba9-ae6b-0f9da7320a48\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-mfd2q"
Jul 28 04:15:52.765090 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:52.765062    2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 28 04:15:52.765154 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:52.765142    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edec740c-2287-4ba9-ae6b-0f9da7320a48-cluster-monitoring-operator-tls podName:edec740c-2287-4ba9-ae6b-0f9da7320a48 nodeName:}" failed. No retries permitted until 2026-07-28 04:16:08.76512185 +0000 UTC m=+269.155297862 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/edec740c-2287-4ba9-ae6b-0f9da7320a48-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-mfd2q" (UID: "edec740c-2287-4ba9-ae6b-0f9da7320a48") : secret "cluster-monitoring-operator-tls" not found
Jul 28 04:15:52.865775 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:52.865744    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c7f7f74-e140-4d88-bc93-c029ae1c18a7-metrics-certs\") pod \"router-default-95c8f4d49-5ncng\" (UID: \"5c7f7f74-e140-4d88-bc93-c029ae1c18a7\") " pod="openshift-ingress/router-default-95c8f4d49-5ncng"
Jul 28 04:15:52.865959 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:52.865800    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c7f7f74-e140-4d88-bc93-c029ae1c18a7-service-ca-bundle\") pod \"router-default-95c8f4d49-5ncng\" (UID: \"5c7f7f74-e140-4d88-bc93-c029ae1c18a7\") " pod="openshift-ingress/router-default-95c8f4d49-5ncng"
Jul 28 04:15:52.866049 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:52.866029    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5c7f7f74-e140-4d88-bc93-c029ae1c18a7-service-ca-bundle podName:5c7f7f74-e140-4d88-bc93-c029ae1c18a7 nodeName:}" failed. No retries permitted until 2026-07-28 04:16:08.866003595 +0000 UTC m=+269.256179601 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/5c7f7f74-e140-4d88-bc93-c029ae1c18a7-service-ca-bundle") pod "router-default-95c8f4d49-5ncng" (UID: "5c7f7f74-e140-4d88-bc93-c029ae1c18a7") : configmap references non-existent config key: service-ca.crt
Jul 28 04:15:52.868030 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:52.868008    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c7f7f74-e140-4d88-bc93-c029ae1c18a7-metrics-certs\") pod \"router-default-95c8f4d49-5ncng\" (UID: \"5c7f7f74-e140-4d88-bc93-c029ae1c18a7\") " pod="openshift-ingress/router-default-95c8f4d49-5ncng"
Jul 28 04:15:58.082989 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:58.082953    2566 scope.go:117] "RemoveContainer" containerID="4472c3365491b95babcf5961e2d9f8259fd4ff80934a67364ffa12ba05036d2b"
Jul 28 04:15:58.414708 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:58.414592    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/83539dc8-61dc-4d3c-aff3-9ac4a543abd9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k8fxt\" (UID: \"83539dc8-61dc-4d3c-aff3-9ac4a543abd9\") " pod="openshift-insights/insights-runtime-extractor-k8fxt"
Jul 28 04:15:58.414887 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:58.414764    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/83539dc8-61dc-4d3c-aff3-9ac4a543abd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k8fxt\" (UID: \"83539dc8-61dc-4d3c-aff3-9ac4a543abd9\") " pod="openshift-insights/insights-runtime-extractor-k8fxt"
Jul 28 04:15:58.415169 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:58.415147    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/83539dc8-61dc-4d3c-aff3-9ac4a543abd9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-k8fxt\" (UID: \"83539dc8-61dc-4d3c-aff3-9ac4a543abd9\") " pod="openshift-insights/insights-runtime-extractor-k8fxt"
Jul 28 04:15:58.417042 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:58.417022    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/83539dc8-61dc-4d3c-aff3-9ac4a543abd9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-k8fxt\" (UID: \"83539dc8-61dc-4d3c-aff3-9ac4a543abd9\") " pod="openshift-insights/insights-runtime-extractor-k8fxt"
Jul 28 04:15:58.472673 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:58.472637    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-k8fxt"
Jul 28 04:15:58.593977 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:58.593931    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-k8fxt"]
Jul 28 04:15:58.595963 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:15:58.595935    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83539dc8_61dc_4d3c_aff3_9ac4a543abd9.slice/crio-916bcdfd49278cacf528397b5a976806e387adfb654dad668ab8bd6d2f893ce3 WatchSource:0}: Error finding container 916bcdfd49278cacf528397b5a976806e387adfb654dad668ab8bd6d2f893ce3: Status 404 returned error can't find the container with id 916bcdfd49278cacf528397b5a976806e387adfb654dad668ab8bd6d2f893ce3
Jul 28 04:15:58.934081 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:58.934058    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4glpr_bfd2d3d8-f287-42d8-83c7-f82185c96735/console-operator/2.log"
Jul 28 04:15:58.934477 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:58.934452    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4glpr_bfd2d3d8-f287-42d8-83c7-f82185c96735/console-operator/1.log"
Jul 28 04:15:58.934592 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:58.934493    2566 generic.go:356] "Generic (PLEG): container finished" podID="bfd2d3d8-f287-42d8-83c7-f82185c96735" containerID="4f23c03d69dd582d7c16234310186ac45bef62f6d1387e91b2be3cd2a4c23c48" exitCode=255
Jul 28 04:15:58.934592 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:58.934571    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-4glpr" event={"ID":"bfd2d3d8-f287-42d8-83c7-f82185c96735","Type":"ContainerDied","Data":"4f23c03d69dd582d7c16234310186ac45bef62f6d1387e91b2be3cd2a4c23c48"}
Jul 28 04:15:58.934695 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:58.934626    2566 scope.go:117] "RemoveContainer" containerID="4472c3365491b95babcf5961e2d9f8259fd4ff80934a67364ffa12ba05036d2b"
Jul 28 04:15:58.934977 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:58.934960    2566 scope.go:117] "RemoveContainer" containerID="4f23c03d69dd582d7c16234310186ac45bef62f6d1387e91b2be3cd2a4c23c48"
Jul 28 04:15:58.935166 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:15:58.935133    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-4glpr_openshift-console-operator(bfd2d3d8-f287-42d8-83c7-f82185c96735)\"" pod="openshift-console-operator/console-operator-584dcb6b54-4glpr" podUID="bfd2d3d8-f287-42d8-83c7-f82185c96735"
Jul 28 04:15:58.935706 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:58.935687    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k8fxt" event={"ID":"83539dc8-61dc-4d3c-aff3-9ac4a543abd9","Type":"ContainerStarted","Data":"916bcdfd49278cacf528397b5a976806e387adfb654dad668ab8bd6d2f893ce3"}
Jul 28 04:15:59.941329 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:15:59.941295    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4glpr_bfd2d3d8-f287-42d8-83c7-f82185c96735/console-operator/2.log"
Jul 28 04:16:03.957168 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:03.957120    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k8fxt" event={"ID":"83539dc8-61dc-4d3c-aff3-9ac4a543abd9","Type":"ContainerStarted","Data":"e5d870b76593a5b297c24383567336f5f860c20a8a00862361d6a5c4447911d8"}
Jul 28 04:16:07.259535 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:07.259452    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-4glpr"
Jul 28 04:16:07.259535 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:07.259500    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-4glpr"
Jul 28 04:16:07.259920 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:07.259754    2566 scope.go:117] "RemoveContainer" containerID="4f23c03d69dd582d7c16234310186ac45bef62f6d1387e91b2be3cd2a4c23c48"
Jul 28 04:16:07.259969 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:16:07.259918    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-4glpr_openshift-console-operator(bfd2d3d8-f287-42d8-83c7-f82185c96735)\"" pod="openshift-console-operator/console-operator-584dcb6b54-4glpr" podUID="bfd2d3d8-f287-42d8-83c7-f82185c96735"
Jul 28 04:16:07.976628 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:07.976587    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k8fxt" event={"ID":"83539dc8-61dc-4d3c-aff3-9ac4a543abd9","Type":"ContainerStarted","Data":"040bcb2157ba896ecbe64c1e03384b7887d6c8814ae2e363294fabb4dbbda4f3"}
Jul 28 04:16:08.810668 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:08.810634    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/edec740c-2287-4ba9-ae6b-0f9da7320a48-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-mfd2q\" (UID: \"edec740c-2287-4ba9-ae6b-0f9da7320a48\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-mfd2q"
Jul 28 04:16:08.813126 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:08.813102    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/edec740c-2287-4ba9-ae6b-0f9da7320a48-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-mfd2q\" (UID: \"edec740c-2287-4ba9-ae6b-0f9da7320a48\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-mfd2q"
Jul 28 04:16:08.911828 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:08.911787    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c7f7f74-e140-4d88-bc93-c029ae1c18a7-service-ca-bundle\") pod \"router-default-95c8f4d49-5ncng\" (UID: \"5c7f7f74-e140-4d88-bc93-c029ae1c18a7\") " pod="openshift-ingress/router-default-95c8f4d49-5ncng"
Jul 28 04:16:08.912538 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:08.912512    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5c7f7f74-e140-4d88-bc93-c029ae1c18a7-service-ca-bundle\") pod \"router-default-95c8f4d49-5ncng\" (UID: \"5c7f7f74-e140-4d88-bc93-c029ae1c18a7\") " pod="openshift-ingress/router-default-95c8f4d49-5ncng"
Jul 28 04:16:09.056002 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:09.055971    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-xhbx7\""
Jul 28 04:16:09.064169 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:09.064095    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-mfd2q"
Jul 28 04:16:09.141713 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:09.141683    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-zdg7l\""
Jul 28 04:16:09.149793 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:09.149763    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-95c8f4d49-5ncng"
Jul 28 04:16:10.510425 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:10.510390    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-95c8f4d49-5ncng"]
Jul 28 04:16:10.513330 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:16:10.513301    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c7f7f74_e140_4d88_bc93_c029ae1c18a7.slice/crio-43e52e8d98985a75483b6cf1bf36614447a1fc87934bcb883e88a858022d684b WatchSource:0}: Error finding container 43e52e8d98985a75483b6cf1bf36614447a1fc87934bcb883e88a858022d684b: Status 404 returned error can't find the container with id 43e52e8d98985a75483b6cf1bf36614447a1fc87934bcb883e88a858022d684b
Jul 28 04:16:10.526201 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:10.526184    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-mfd2q"]
Jul 28 04:16:10.528988 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:16:10.528958    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedec740c_2287_4ba9_ae6b_0f9da7320a48.slice/crio-6dbc4eae007c51555398097b009903685e32ecc4fb857379843926671379edd1 WatchSource:0}: Error finding container 6dbc4eae007c51555398097b009903685e32ecc4fb857379843926671379edd1: Status 404 returned error can't find the container with id 6dbc4eae007c51555398097b009903685e32ecc4fb857379843926671379edd1
Jul 28 04:16:10.988249 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:10.988211    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-k8fxt" event={"ID":"83539dc8-61dc-4d3c-aff3-9ac4a543abd9","Type":"ContainerStarted","Data":"32ed3fd2e0832fa6482d7e61982051d448decc1ef97fce77622270ce49c09df0"}
Jul 28 04:16:10.989382 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:10.989356    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-mfd2q" event={"ID":"edec740c-2287-4ba9-ae6b-0f9da7320a48","Type":"ContainerStarted","Data":"6dbc4eae007c51555398097b009903685e32ecc4fb857379843926671379edd1"}
Jul 28 04:16:10.990566 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:10.990544    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-95c8f4d49-5ncng" event={"ID":"5c7f7f74-e140-4d88-bc93-c029ae1c18a7","Type":"ContainerStarted","Data":"895ce9e77410cd39f589122cb5e093cb5ce9783a54da29352806608ac1e6278a"}
Jul 28 04:16:10.990628 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:10.990586    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-95c8f4d49-5ncng" event={"ID":"5c7f7f74-e140-4d88-bc93-c029ae1c18a7","Type":"ContainerStarted","Data":"43e52e8d98985a75483b6cf1bf36614447a1fc87934bcb883e88a858022d684b"}
Jul 28 04:16:11.003985 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:11.003942    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-k8fxt" podStartSLOduration=17.278404989 podStartE2EDuration="29.00391764s" totalImagesPullingTime="11.725512651s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 04:15:42 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:16:11.003201793 +0000 UTC m=+271.393377814" watchObservedRunningTime="2026-07-28 04:16:11.00391764 +0000 UTC m=+271.394093660"
Jul 28 04:16:11.019304 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:11.019265    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-95c8f4d49-5ncng" podStartSLOduration=34.019250119 podStartE2EDuration="34.019250119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:15:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:16:11.018895557 +0000 UTC m=+271.409071577" watchObservedRunningTime="2026-07-28 04:16:11.019250119 +0000 UTC m=+271.409426140"
Jul 28 04:16:11.150622 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:11.150586    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-95c8f4d49-5ncng"
Jul 28 04:16:11.153550 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:11.153528    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-95c8f4d49-5ncng"
Jul 28 04:16:11.994760 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:11.994726    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-95c8f4d49-5ncng"
Jul 28 04:16:11.996099 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:11.996077    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-95c8f4d49-5ncng"
Jul 28 04:16:12.999448 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:12.999409    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-mfd2q" event={"ID":"edec740c-2287-4ba9-ae6b-0f9da7320a48","Type":"ContainerStarted","Data":"bb44bbeb5fd2b7172980c25d5c8761d53f4164c52f82b843cc42f7de02c3d3c6"}
Jul 28 04:16:13.014928 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:13.014883    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-mfd2q" podStartSLOduration=35.190640765 podStartE2EDuration="37.014869498s" totalImagesPullingTime="1.824228733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:15:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:16:13.013510411 +0000 UTC m=+273.403686432" watchObservedRunningTime="2026-07-28 04:16:13.014869498 +0000 UTC m=+273.405045518"
Jul 28 04:16:15.872239 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:15.872202    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-65jws"]
Jul 28 04:16:16.064098 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:16.064065    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-65jws"]
Jul 28 04:16:16.064247 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:16.064181    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-65jws"
Jul 28 04:16:16.067377 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:16.067347    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 04:16:16.067613 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:16.067597    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 28 04:16:16.067664 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:16.067628    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-lbdm2\""
Jul 28 04:16:16.067709 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:16.067605    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 28 04:16:16.167142 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:16.167085    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa80dba1-35cc-4655-9984-d85d596e27d8-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-65jws\" (UID: \"fa80dba1-35cc-4655-9984-d85d596e27d8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-65jws"
Jul 28 04:16:16.167249 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:16.167139    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxx6b\" (UniqueName: \"kubernetes.io/projected/fa80dba1-35cc-4655-9984-d85d596e27d8-kube-api-access-xxx6b\") pod \"prometheus-operator-84d69f794d-65jws\" (UID: \"fa80dba1-35cc-4655-9984-d85d596e27d8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-65jws"
Jul 28 04:16:16.167249 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:16.167170    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fa80dba1-35cc-4655-9984-d85d596e27d8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-65jws\" (UID: \"fa80dba1-35cc-4655-9984-d85d596e27d8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-65jws"
Jul 28 04:16:16.167249 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:16.167197    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/fa80dba1-35cc-4655-9984-d85d596e27d8-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-65jws\" (UID: \"fa80dba1-35cc-4655-9984-d85d596e27d8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-65jws"
Jul 28 04:16:16.268356 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:16.268327    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa80dba1-35cc-4655-9984-d85d596e27d8-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-65jws\" (UID: \"fa80dba1-35cc-4655-9984-d85d596e27d8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-65jws"
Jul 28 04:16:16.268459 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:16.268379    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxx6b\" (UniqueName: \"kubernetes.io/projected/fa80dba1-35cc-4655-9984-d85d596e27d8-kube-api-access-xxx6b\") pod \"prometheus-operator-84d69f794d-65jws\" (UID: \"fa80dba1-35cc-4655-9984-d85d596e27d8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-65jws"
Jul 28 04:16:16.268525 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:16.268503    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fa80dba1-35cc-4655-9984-d85d596e27d8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-65jws\" (UID: \"fa80dba1-35cc-4655-9984-d85d596e27d8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-65jws"
Jul 28 04:16:16.268585 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:16.268568    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/fa80dba1-35cc-4655-9984-d85d596e27d8-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-65jws\" (UID: \"fa80dba1-35cc-4655-9984-d85d596e27d8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-65jws"
Jul 28 04:16:16.269119 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:16.269098    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa80dba1-35cc-4655-9984-d85d596e27d8-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-65jws\" (UID: \"fa80dba1-35cc-4655-9984-d85d596e27d8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-65jws"
Jul 28 04:16:16.270967 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:16.270942    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fa80dba1-35cc-4655-9984-d85d596e27d8-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-65jws\" (UID: \"fa80dba1-35cc-4655-9984-d85d596e27d8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-65jws"
Jul 28 04:16:16.271048 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:16.271007    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/fa80dba1-35cc-4655-9984-d85d596e27d8-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-65jws\" (UID: \"fa80dba1-35cc-4655-9984-d85d596e27d8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-65jws"
Jul 28 04:16:16.275925 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:16.275905    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxx6b\" (UniqueName: \"kubernetes.io/projected/fa80dba1-35cc-4655-9984-d85d596e27d8-kube-api-access-xxx6b\") pod \"prometheus-operator-84d69f794d-65jws\" (UID: \"fa80dba1-35cc-4655-9984-d85d596e27d8\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-65jws"
Jul 28 04:16:16.373347 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:16.373324    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-65jws"
Jul 28 04:16:16.530668 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:16.530645    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-65jws"]
Jul 28 04:16:16.532747 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:16:16.532707    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa80dba1_35cc_4655_9984_d85d596e27d8.slice/crio-a59593e11f408d1be973d18e33211aa6216ad472cd5a734a40346e59835ec490 WatchSource:0}: Error finding container a59593e11f408d1be973d18e33211aa6216ad472cd5a734a40346e59835ec490: Status 404 returned error can't find the container with id a59593e11f408d1be973d18e33211aa6216ad472cd5a734a40346e59835ec490
Jul 28 04:16:17.014045 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:17.014011    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-65jws" event={"ID":"fa80dba1-35cc-4655-9984-d85d596e27d8","Type":"ContainerStarted","Data":"a59593e11f408d1be973d18e33211aa6216ad472cd5a734a40346e59835ec490"}
Jul 28 04:16:19.024413 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:19.024384    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-65jws" event={"ID":"fa80dba1-35cc-4655-9984-d85d596e27d8","Type":"ContainerStarted","Data":"d3e7892624f73760a9d0b635b3cd2791c5e3e738c3f1623276a22ec79e23966f"}
Jul 28 04:16:19.024874 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:19.024433    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-65jws" event={"ID":"fa80dba1-35cc-4655-9984-d85d596e27d8","Type":"ContainerStarted","Data":"53f11a59e5a38b758fd1c24669cc26be7ac45d56ac200182dac0be907a85117d"}
Jul 28 04:16:19.041973 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:19.041926    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-65jws" podStartSLOduration=2.053435833 podStartE2EDuration="4.041911176s" totalImagesPullingTime="1.988475343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:16:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:16:19.039559539 +0000 UTC m=+279.429735559" watchObservedRunningTime="2026-07-28 04:16:19.041911176 +0000 UTC m=+279.432087196"
Jul 28 04:16:21.228799 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:21.228761    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4sdhs"]
Jul 28 04:16:21.246736 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:21.246713    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4sdhs"
Jul 28 04:16:21.249331 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:21.249306    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 04:16:21.249464 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:21.249334    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 04:16:21.252360 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:21.250097    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 04:16:21.252360 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:21.250259    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6692x\""
Jul 28 04:16:21.305700 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:21.305672    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/71b82494-fe22-4a00-9e59-44d7d84fb7a5-root\") pod \"node-exporter-4sdhs\" (UID: \"71b82494-fe22-4a00-9e59-44d7d84fb7a5\") " pod="openshift-monitoring/node-exporter-4sdhs"
Jul 28 04:16:21.305815 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:21.305712    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71b82494-fe22-4a00-9e59-44d7d84fb7a5-metrics-client-ca\") pod \"node-exporter-4sdhs\" (UID: \"71b82494-fe22-4a00-9e59-44d7d84fb7a5\") " pod="openshift-monitoring/node-exporter-4sdhs"
Jul 28 04:16:21.305815 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:21.305745    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/71b82494-fe22-4a00-9e59-44d7d84fb7a5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4sdhs\" (UID: \"71b82494-fe22-4a00-9e59-44d7d84fb7a5\") " pod="openshift-monitoring/node-exporter-4sdhs"
Jul 28 04:16:21.305815 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:21.305773    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/71b82494-fe22-4a00-9e59-44d7d84fb7a5-sys\") pod \"node-exporter-4sdhs\" (UID: \"71b82494-fe22-4a00-9e59-44d7d84fb7a5\") " pod="openshift-monitoring/node-exporter-4sdhs"
Jul 28 04:16:21.305815 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:21.305797    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktqsd\" (UniqueName: \"kubernetes.io/projected/71b82494-fe22-4a00-9e59-44d7d84fb7a5-kube-api-access-ktqsd\") pod \"node-exporter-4sdhs\" (UID: \"71b82494-fe22-4a00-9e59-44d7d84fb7a5\") " pod="openshift-monitoring/node-exporter-4sdhs"
Jul 28 04:16:21.306019 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:21.305826    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/71b82494-fe22-4a00-9e59-44d7d84fb7a5-node-exporter-accelerators-collector-config\") pod \"node-exporter-4sdhs\" (UID: \"71b82494-fe22-4a00-9e59-44d7d84fb7a5\") " pod="openshift-monitoring/node-exporter-4sdhs"
Jul 28 04:16:21.306019 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:21.305870    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/71b82494-fe22-4a00-9e59-44d7d84fb7a5-node-exporter-tls\") pod \"node-exporter-4sdhs\" (UID: \"71b82494-fe22-4a00-9e59-44d7d84fb7a5\") " pod="openshift-monitoring/node-exporter-4sdhs"
Jul 28 04:16:21.306019 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:21.305918    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/71b82494-fe22-4a00-9e59-44d7d84fb7a5-node-exporter-textfile\") pod \"node-exporter-4sdhs\" (UID: \"71b82494-fe22-4a00-9e59-44d7d84fb7a5\") " pod="openshift-monitoring/node-exporter-4sdhs"
Jul 28 04:16:21.306019 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:21.305960    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/71b82494-fe22-4a00-9e59-44d7d84fb7a5-node-exporter-wtmp\") pod \"node-exporter-4sdhs\" (UID: \"71b82494-fe22-4a00-9e59-44d7d84fb7a5\") " pod="openshift-monitoring/node-exporter-4sdhs"
Jul 28 04:16:21.406999 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:21.406973    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/71b82494-fe22-4a00-9e59-44d7d84fb7a5-node-exporter-tls\") pod \"node-exporter-4sdhs\" (UID: \"71b82494-fe22-4a00-9e59-44d7d84fb7a5\") " pod="openshift-monitoring/node-exporter-4sdhs"
Jul 28 04:16:21.407142 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:21.407023    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/71b82494-fe22-4a00-9e59-44d7d84fb7a5-node-exporter-textfile\") pod \"node-exporter-4sdhs\" (UID: \"71b82494-fe22-4a00-9e59-44d7d84fb7a5\") " pod="openshift-monitoring/node-exporter-4sdhs"
Jul 28 04:16:21.407205 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:21.407183    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/71b82494-fe22-4a00-9e59-44d7d84fb7a5-node-exporter-wtmp\") pod \"node-exporter-4sdhs\" (UID: \"71b82494-fe22-4a00-9e59-44d7d84fb7a5\") " pod="openshift-monitoring/node-exporter-4sdhs"
Jul 28 04:16:21.407307 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:21.407289    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/71b82494-fe22-4a00-9e59-44d7d84fb7a5-root\") pod \"node-exporter-4sdhs\" (UID: \"71b82494-fe22-4a00-9e59-44d7d84fb7a5\") " pod="openshift-monitoring/node-exporter-4sdhs"
Jul 28 04:16:21.407372 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:21.407321    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71b82494-fe22-4a00-9e59-44d7d84fb7a5-metrics-client-ca\") pod \"node-exporter-4sdhs\" (UID: \"71b82494-fe22-4a00-9e59-44d7d84fb7a5\") " pod="openshift-monitoring/node-exporter-4sdhs"
Jul 28 04:16:21.407372 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:21.407341    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/71b82494-fe22-4a00-9e59-44d7d84fb7a5-node-exporter-wtmp\") pod \"node-exporter-4sdhs\" (UID: \"71b82494-fe22-4a00-9e59-44d7d84fb7a5\") " pod="openshift-monitoring/node-exporter-4sdhs"
Jul 28 04:16:21.407372 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:21.407358    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/71b82494-fe22-4a00-9e59-44d7d84fb7a5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4sdhs\" (UID: \"71b82494-fe22-4a00-9e59-44d7d84fb7a5\") " pod="openshift-monitoring/node-exporter-4sdhs"
Jul 28 04:16:21.407372 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:21.407349    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/71b82494-fe22-4a00-9e59-44d7d84fb7a5-node-exporter-textfile\") pod \"node-exporter-4sdhs\" (UID: \"71b82494-fe22-4a00-9e59-44d7d84fb7a5\") " pod="openshift-monitoring/node-exporter-4sdhs"
Jul 28 04:16:21.407568 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:21.407393    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/71b82494-fe22-4a00-9e59-44d7d84fb7a5-sys\") pod \"node-exporter-4sdhs\" (UID: \"71b82494-fe22-4a00-9e59-44d7d84fb7a5\") " pod="openshift-monitoring/node-exporter-4sdhs"
Jul 28 04:16:21.407568 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:21.407423    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktqsd\" (UniqueName: \"kubernetes.io/projected/71b82494-fe22-4a00-9e59-44d7d84fb7a5-kube-api-access-ktqsd\") pod \"node-exporter-4sdhs\" (UID: \"71b82494-fe22-4a00-9e59-44d7d84fb7a5\") " pod="openshift-monitoring/node-exporter-4sdhs"
Jul 28 04:16:21.407568 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:21.407453    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/71b82494-fe22-4a00-9e59-44d7d84fb7a5-node-exporter-accelerators-collector-config\") pod \"node-exporter-4sdhs\" (UID: \"71b82494-fe22-4a00-9e59-44d7d84fb7a5\") " pod="openshift-monitoring/node-exporter-4sdhs"
Jul 28 04:16:21.407718 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:21.407393    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/71b82494-fe22-4a00-9e59-44d7d84fb7a5-root\") pod \"node-exporter-4sdhs\" (UID: \"71b82494-fe22-4a00-9e59-44d7d84fb7a5\") " pod="openshift-monitoring/node-exporter-4sdhs"
Jul 28 04:16:21.407718 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:21.407631    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/71b82494-fe22-4a00-9e59-44d7d84fb7a5-sys\") pod \"node-exporter-4sdhs\" (UID: \"71b82494-fe22-4a00-9e59-44d7d84fb7a5\") " pod="openshift-monitoring/node-exporter-4sdhs"
Jul 28 04:16:21.408064 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:21.408043    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/71b82494-fe22-4a00-9e59-44d7d84fb7a5-metrics-client-ca\") pod \"node-exporter-4sdhs\" (UID: \"71b82494-fe22-4a00-9e59-44d7d84fb7a5\") " pod="openshift-monitoring/node-exporter-4sdhs"
Jul 28 04:16:21.408435 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:21.408412    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/71b82494-fe22-4a00-9e59-44d7d84fb7a5-node-exporter-accelerators-collector-config\") pod \"node-exporter-4sdhs\" (UID: \"71b82494-fe22-4a00-9e59-44d7d84fb7a5\") " pod="openshift-monitoring/node-exporter-4sdhs"
Jul 28 04:16:21.409653 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:21.409626    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/71b82494-fe22-4a00-9e59-44d7d84fb7a5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4sdhs\" (UID: \"71b82494-fe22-4a00-9e59-44d7d84fb7a5\") " pod="openshift-monitoring/node-exporter-4sdhs"
Jul 28 04:16:21.409880 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:21.409859    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/71b82494-fe22-4a00-9e59-44d7d84fb7a5-node-exporter-tls\") pod \"node-exporter-4sdhs\" (UID: \"71b82494-fe22-4a00-9e59-44d7d84fb7a5\") " pod="openshift-monitoring/node-exporter-4sdhs"
Jul 28 04:16:21.415545 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:21.415522    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktqsd\" (UniqueName: \"kubernetes.io/projected/71b82494-fe22-4a00-9e59-44d7d84fb7a5-kube-api-access-ktqsd\") pod \"node-exporter-4sdhs\" (UID: \"71b82494-fe22-4a00-9e59-44d7d84fb7a5\") " pod="openshift-monitoring/node-exporter-4sdhs"
Jul 28 04:16:21.564578 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:21.564544    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4sdhs"
Jul 28 04:16:21.575191 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:16:21.575162    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71b82494_fe22_4a00_9e59_44d7d84fb7a5.slice/crio-d38fbc0400696891996964ce1309d599699117ce789a4f80cae0a26fef97a70e WatchSource:0}: Error finding container d38fbc0400696891996964ce1309d599699117ce789a4f80cae0a26fef97a70e: Status 404 returned error can't find the container with id d38fbc0400696891996964ce1309d599699117ce789a4f80cae0a26fef97a70e
Jul 28 04:16:21.673110 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:16:21.673064    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn" podUID="55cd6e46-995c-4857-b7b7-c90e4e4ff93d"
Jul 28 04:16:21.673110 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:16:21.673064    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-m9zx6" podUID="a02a02e4-e2b1-45fd-aeda-12fa4e29a996"
Jul 28 04:16:21.673300 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:16:21.673084    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-74c9f884b7-jxd2j" podUID="c58f3317-e522-41a3-848f-906af440a23b"
Jul 28 04:16:22.036477 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:22.036404    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-jxd2j"
Jul 28 04:16:22.036477 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:22.036442    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn"
Jul 28 04:16:22.036477 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:22.036436    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4sdhs" event={"ID":"71b82494-fe22-4a00-9e59-44d7d84fb7a5","Type":"ContainerStarted","Data":"d38fbc0400696891996964ce1309d599699117ce789a4f80cae0a26fef97a70e"}
Jul 28 04:16:22.036754 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:22.036449    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-m9zx6"
Jul 28 04:16:22.082383 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:22.082356    2566 scope.go:117] "RemoveContainer" containerID="4f23c03d69dd582d7c16234310186ac45bef62f6d1387e91b2be3cd2a4c23c48"
Jul 28 04:16:23.043391 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:23.043361    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4glpr_bfd2d3d8-f287-42d8-83c7-f82185c96735/console-operator/2.log"
Jul 28 04:16:23.043786 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:23.043465    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-4glpr" event={"ID":"bfd2d3d8-f287-42d8-83c7-f82185c96735","Type":"ContainerStarted","Data":"a7931f44f8535bebbbbe46165542bcb3a49f6772a97e30d372b989aaf24f3f5f"}
Jul 28 04:16:23.043786 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:23.043760    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-4glpr"
Jul 28 04:16:23.045237 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:23.045207    2566 generic.go:356] "Generic (PLEG): container finished" podID="71b82494-fe22-4a00-9e59-44d7d84fb7a5" containerID="26934bd1bdceea2f37c24215b73096eb27cf6509314df0b6b61b827639291d4a" exitCode=0
Jul 28 04:16:23.045356 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:23.045272    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4sdhs" event={"ID":"71b82494-fe22-4a00-9e59-44d7d84fb7a5","Type":"ContainerDied","Data":"26934bd1bdceea2f37c24215b73096eb27cf6509314df0b6b61b827639291d4a"}
Jul 28 04:16:23.049910 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:23.049890    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-4glpr"
Jul 28 04:16:23.060200 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:23.060158    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-4glpr" podStartSLOduration=43.271045922 podStartE2EDuration="47.060139737s" totalImagesPullingTime="3.789093815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:15:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:16:23.058768706 +0000 UTC m=+283.448944751" watchObservedRunningTime="2026-07-28 04:16:23.060139737 +0000 UTC m=+283.450315761"
Jul 28 04:16:23.121989 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:23.121968    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-rs96k"]
Jul 28 04:16:23.125107 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:23.124464    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-rs96k"
Jul 28 04:16:23.127220 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:23.127152    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-s7p5p\""
Jul 28 04:16:23.127564 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:23.127388    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 28 04:16:23.128030 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:23.128010    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 28 04:16:23.136606 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:23.136573    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-rs96k"]
Jul 28 04:16:23.225270 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:23.225240    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9b4q\" (UniqueName: \"kubernetes.io/projected/e1817115-022a-447e-9431-62a0d5c2e4c6-kube-api-access-b9b4q\") pod \"downloads-7c97dc4555-rs96k\" (UID: \"e1817115-022a-447e-9431-62a0d5c2e4c6\") " pod="openshift-console/downloads-7c97dc4555-rs96k"
Jul 28 04:16:23.325994 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:23.325920    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9b4q\" (UniqueName: \"kubernetes.io/projected/e1817115-022a-447e-9431-62a0d5c2e4c6-kube-api-access-b9b4q\") pod \"downloads-7c97dc4555-rs96k\" (UID: \"e1817115-022a-447e-9431-62a0d5c2e4c6\") " pod="openshift-console/downloads-7c97dc4555-rs96k"
Jul 28 04:16:23.333660 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:23.333630    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9b4q\" (UniqueName: \"kubernetes.io/projected/e1817115-022a-447e-9431-62a0d5c2e4c6-kube-api-access-b9b4q\") pod \"downloads-7c97dc4555-rs96k\" (UID: \"e1817115-022a-447e-9431-62a0d5c2e4c6\") " pod="openshift-console/downloads-7c97dc4555-rs96k"
Jul 28 04:16:23.437043 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:23.437012    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-rs96k"
Jul 28 04:16:23.556971 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:23.556859    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-rs96k"]
Jul 28 04:16:23.559428 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:16:23.559405    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1817115_022a_447e_9431_62a0d5c2e4c6.slice/crio-197abdf2a95680e830aef2b0e145075850084c5072612f681b007616efbe8965 WatchSource:0}: Error finding container 197abdf2a95680e830aef2b0e145075850084c5072612f681b007616efbe8965: Status 404 returned error can't find the container with id 197abdf2a95680e830aef2b0e145075850084c5072612f681b007616efbe8965
Jul 28 04:16:24.050776 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:24.050739    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4sdhs" event={"ID":"71b82494-fe22-4a00-9e59-44d7d84fb7a5","Type":"ContainerStarted","Data":"ef5009d183c339bc4715364213fb06ed75ed6f78e377d9e7c61c25132378da88"}
Jul 28 04:16:24.051246 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:24.050799    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4sdhs" event={"ID":"71b82494-fe22-4a00-9e59-44d7d84fb7a5","Type":"ContainerStarted","Data":"61b9f30f10080386995c62222bdd2ba958a61ccf4b9a8dce584b97e71297967c"}
Jul 28 04:16:24.051888 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:24.051867    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-rs96k" event={"ID":"e1817115-022a-447e-9431-62a0d5c2e4c6","Type":"ContainerStarted","Data":"197abdf2a95680e830aef2b0e145075850084c5072612f681b007616efbe8965"}
Jul 28 04:16:24.069970 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:24.069932    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4sdhs" podStartSLOduration=2.143715541 podStartE2EDuration="3.069919643s" totalImagesPullingTime="860.254884ms" totalInitContainerRuntime="65.949218ms" isStatefulPod=true podCreationTimestamp="2026-07-28 04:16:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:16:24.068870381 +0000 UTC m=+284.459046402" watchObservedRunningTime="2026-07-28 04:16:24.069919643 +0000 UTC m=+284.460095663"
Jul 28 04:16:25.044092 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:25.044060    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-registry-tls\") pod \"image-registry-66ff9f7dd7-45ggn\" (UID: \"55cd6e46-995c-4857-b7b7-c90e4e4ff93d\") " pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn"
Jul 28 04:16:25.044276 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:25.044104    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a02a02e4-e2b1-45fd-aeda-12fa4e29a996-metrics-tls\") pod \"dns-default-m9zx6\" (UID: \"a02a02e4-e2b1-45fd-aeda-12fa4e29a996\") " pod="openshift-dns/dns-default-m9zx6"
Jul 28 04:16:25.044276 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:25.044179    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c58f3317-e522-41a3-848f-906af440a23b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-jxd2j\" (UID: \"c58f3317-e522-41a3-848f-906af440a23b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-jxd2j"
Jul 28 04:16:25.046778 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:25.046746    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-registry-tls\") pod \"image-registry-66ff9f7dd7-45ggn\" (UID: \"55cd6e46-995c-4857-b7b7-c90e4e4ff93d\") " pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn"
Jul 28 04:16:25.046917 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:25.046789    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c58f3317-e522-41a3-848f-906af440a23b-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-jxd2j\" (UID: \"c58f3317-e522-41a3-848f-906af440a23b\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-jxd2j"
Jul 28 04:16:25.046985 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:25.046912    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a02a02e4-e2b1-45fd-aeda-12fa4e29a996-metrics-tls\") pod \"dns-default-m9zx6\" (UID: \"a02a02e4-e2b1-45fd-aeda-12fa4e29a996\") " pod="openshift-dns/dns-default-m9zx6"
Jul 28 04:16:25.341379 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:25.341303    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-s6545\""
Jul 28 04:16:25.341379 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:25.341339    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-c8c86\""
Jul 28 04:16:25.341749 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:25.341307    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7zgg8\""
Jul 28 04:16:25.347918 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:25.347898    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-m9zx6"
Jul 28 04:16:25.347997 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:25.347984    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn"
Jul 28 04:16:25.348037 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:25.347995    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-jxd2j"
Jul 28 04:16:25.507756 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:25.507722    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-jxd2j"]
Jul 28 04:16:25.511303 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:16:25.511262    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc58f3317_e522_41a3_848f_906af440a23b.slice/crio-a62f388acc8b45ffdb9df138b46a4c134942361e2375cf0ed6ed112be9a9b5b1 WatchSource:0}: Error finding container a62f388acc8b45ffdb9df138b46a4c134942361e2375cf0ed6ed112be9a9b5b1: Status 404 returned error can't find the container with id a62f388acc8b45ffdb9df138b46a4c134942361e2375cf0ed6ed112be9a9b5b1
Jul 28 04:16:25.537559 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:25.537532    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-67c5866966-hssnd"]
Jul 28 04:16:25.543030 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:25.541331    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-67c5866966-hssnd"
Jul 28 04:16:25.544339 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:25.544319    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 28 04:16:25.544571 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:25.544554    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-5pnrq\""
Jul 28 04:16:25.544661 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:25.544634    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 28 04:16:25.544721 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:25.544674    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 28 04:16:25.545531 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:25.545516    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 28 04:16:25.545584 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:25.545557    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-8mh8f7u84hhcu\""
Jul 28 04:16:25.551321 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:25.551289    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-67c5866966-hssnd"]
Jul 28 04:16:25.649775 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:25.649687    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58f2d\" (UniqueName: \"kubernetes.io/projected/819152a5-89ff-4974-bfb7-7025060775b1-kube-api-access-58f2d\") pod \"metrics-server-67c5866966-hssnd\" (UID: \"819152a5-89ff-4974-bfb7-7025060775b1\") " pod="openshift-monitoring/metrics-server-67c5866966-hssnd"
Jul 28 04:16:25.649941 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:25.649810    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/819152a5-89ff-4974-bfb7-7025060775b1-metrics-server-audit-profiles\") pod \"metrics-server-67c5866966-hssnd\" (UID: \"819152a5-89ff-4974-bfb7-7025060775b1\") " pod="openshift-monitoring/metrics-server-67c5866966-hssnd"
Jul 28 04:16:25.649941 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:25.649869    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/819152a5-89ff-4974-bfb7-7025060775b1-secret-metrics-server-client-certs\") pod \"metrics-server-67c5866966-hssnd\" (UID: \"819152a5-89ff-4974-bfb7-7025060775b1\") " pod="openshift-monitoring/metrics-server-67c5866966-hssnd"
Jul 28 04:16:25.649941 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:25.649889    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/819152a5-89ff-4974-bfb7-7025060775b1-secret-metrics-server-tls\") pod \"metrics-server-67c5866966-hssnd\" (UID: \"819152a5-89ff-4974-bfb7-7025060775b1\") " pod="openshift-monitoring/metrics-server-67c5866966-hssnd"
Jul 28 04:16:25.649941 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:25.649929    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/819152a5-89ff-4974-bfb7-7025060775b1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-67c5866966-hssnd\" (UID: \"819152a5-89ff-4974-bfb7-7025060775b1\") " pod="openshift-monitoring/metrics-server-67c5866966-hssnd"
Jul 28 04:16:25.650116 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:25.649969    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/819152a5-89ff-4974-bfb7-7025060775b1-audit-log\") pod \"metrics-server-67c5866966-hssnd\" (UID: \"819152a5-89ff-4974-bfb7-7025060775b1\") " pod="openshift-monitoring/metrics-server-67c5866966-hssnd"
Jul 28 04:16:25.650116 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:25.650025    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/819152a5-89ff-4974-bfb7-7025060775b1-client-ca-bundle\") pod \"metrics-server-67c5866966-hssnd\" (UID: \"819152a5-89ff-4974-bfb7-7025060775b1\") " pod="openshift-monitoring/metrics-server-67c5866966-hssnd"
Jul 28 04:16:25.747289 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:25.747266    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-m9zx6"]
Jul 28 04:16:25.750042 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:25.750000    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66ff9f7dd7-45ggn"]
Jul 28 04:16:25.750133 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:16:25.750069    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda02a02e4_e2b1_45fd_aeda_12fa4e29a996.slice/crio-8d8b333b9de3438cb084ec0e6db403f2aa7cb22a05d70fa3756872de3b522025 WatchSource:0}: Error finding container 8d8b333b9de3438cb084ec0e6db403f2aa7cb22a05d70fa3756872de3b522025: Status 404 returned error can't find the container with id 8d8b333b9de3438cb084ec0e6db403f2aa7cb22a05d70fa3756872de3b522025
Jul 28 04:16:25.750356 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:25.750338    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58f2d\" (UniqueName: \"kubernetes.io/projected/819152a5-89ff-4974-bfb7-7025060775b1-kube-api-access-58f2d\") pod \"metrics-server-67c5866966-hssnd\" (UID: \"819152a5-89ff-4974-bfb7-7025060775b1\") " pod="openshift-monitoring/metrics-server-67c5866966-hssnd"
Jul 28 04:16:25.750424 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:25.750400    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/819152a5-89ff-4974-bfb7-7025060775b1-metrics-server-audit-profiles\") pod \"metrics-server-67c5866966-hssnd\" (UID: \"819152a5-89ff-4974-bfb7-7025060775b1\") " pod="openshift-monitoring/metrics-server-67c5866966-hssnd"
Jul 28 04:16:25.750526 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:25.750432    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/819152a5-89ff-4974-bfb7-7025060775b1-secret-metrics-server-client-certs\") pod \"metrics-server-67c5866966-hssnd\" (UID: \"819152a5-89ff-4974-bfb7-7025060775b1\") " pod="openshift-monitoring/metrics-server-67c5866966-hssnd"
Jul 28 04:16:25.750526 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:25.750450    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/819152a5-89ff-4974-bfb7-7025060775b1-secret-metrics-server-tls\") pod \"metrics-server-67c5866966-hssnd\" (UID: \"819152a5-89ff-4974-bfb7-7025060775b1\") " pod="openshift-monitoring/metrics-server-67c5866966-hssnd"
Jul 28 04:16:25.750526 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:25.750472    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/819152a5-89ff-4974-bfb7-7025060775b1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-67c5866966-hssnd\" (UID: \"819152a5-89ff-4974-bfb7-7025060775b1\") " pod="openshift-monitoring/metrics-server-67c5866966-hssnd"
Jul 28 04:16:25.750526 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:25.750496    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/819152a5-89ff-4974-bfb7-7025060775b1-audit-log\") pod \"metrics-server-67c5866966-hssnd\" (UID: \"819152a5-89ff-4974-bfb7-7025060775b1\") " pod="openshift-monitoring/metrics-server-67c5866966-hssnd"
Jul 28 04:16:25.750792 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:25.750546    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/819152a5-89ff-4974-bfb7-7025060775b1-client-ca-bundle\") pod \"metrics-server-67c5866966-hssnd\" (UID: \"819152a5-89ff-4974-bfb7-7025060775b1\") " pod="openshift-monitoring/metrics-server-67c5866966-hssnd"
Jul 28 04:16:25.751300 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:25.751186    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/819152a5-89ff-4974-bfb7-7025060775b1-audit-log\") pod \"metrics-server-67c5866966-hssnd\" (UID: \"819152a5-89ff-4974-bfb7-7025060775b1\") " pod="openshift-monitoring/metrics-server-67c5866966-hssnd"
Jul 28 04:16:25.751497 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:25.751471    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/819152a5-89ff-4974-bfb7-7025060775b1-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-67c5866966-hssnd\" (UID: \"819152a5-89ff-4974-bfb7-7025060775b1\") " pod="openshift-monitoring/metrics-server-67c5866966-hssnd"
Jul 28 04:16:25.751745 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:25.751724    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/819152a5-89ff-4974-bfb7-7025060775b1-metrics-server-audit-profiles\") pod \"metrics-server-67c5866966-hssnd\" (UID: \"819152a5-89ff-4974-bfb7-7025060775b1\") " pod="openshift-monitoring/metrics-server-67c5866966-hssnd"
Jul 28 04:16:25.753196 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:25.753172    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/819152a5-89ff-4974-bfb7-7025060775b1-secret-metrics-server-tls\") pod \"metrics-server-67c5866966-hssnd\" (UID: \"819152a5-89ff-4974-bfb7-7025060775b1\") " pod="openshift-monitoring/metrics-server-67c5866966-hssnd"
Jul 28 04:16:25.753427 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:25.753406    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/819152a5-89ff-4974-bfb7-7025060775b1-client-ca-bundle\") pod \"metrics-server-67c5866966-hssnd\" (UID: \"819152a5-89ff-4974-bfb7-7025060775b1\") " pod="openshift-monitoring/metrics-server-67c5866966-hssnd"
Jul 28 04:16:25.753518 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:25.753438    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/819152a5-89ff-4974-bfb7-7025060775b1-secret-metrics-server-client-certs\") pod \"metrics-server-67c5866966-hssnd\" (UID: \"819152a5-89ff-4974-bfb7-7025060775b1\") " pod="openshift-monitoring/metrics-server-67c5866966-hssnd"
Jul 28 04:16:25.753518 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:16:25.753445    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55cd6e46_995c_4857_b7b7_c90e4e4ff93d.slice/crio-2f5f833c60eb502e81c5abf1943859e45b0cf6826d97ac574fa0e46b2cdfe285 WatchSource:0}: Error finding container 2f5f833c60eb502e81c5abf1943859e45b0cf6826d97ac574fa0e46b2cdfe285: Status 404 returned error can't find the container with id 2f5f833c60eb502e81c5abf1943859e45b0cf6826d97ac574fa0e46b2cdfe285
Jul 28 04:16:25.758062 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:25.758044    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58f2d\" (UniqueName: \"kubernetes.io/projected/819152a5-89ff-4974-bfb7-7025060775b1-kube-api-access-58f2d\") pod \"metrics-server-67c5866966-hssnd\" (UID: \"819152a5-89ff-4974-bfb7-7025060775b1\") " pod="openshift-monitoring/metrics-server-67c5866966-hssnd"
Jul 28 04:16:25.854354 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:25.854321    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-67c5866966-hssnd"
Jul 28 04:16:26.025700 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:26.025508    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-67c5866966-hssnd"]
Jul 28 04:16:26.031537 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:16:26.031502    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod819152a5_89ff_4974_bfb7_7025060775b1.slice/crio-b48cfab43692af91b69e6da3f8f24eb4c5bfa3c52274ce680642bb502b461297 WatchSource:0}: Error finding container b48cfab43692af91b69e6da3f8f24eb4c5bfa3c52274ce680642bb502b461297: Status 404 returned error can't find the container with id b48cfab43692af91b69e6da3f8f24eb4c5bfa3c52274ce680642bb502b461297
Jul 28 04:16:26.069775 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:26.069743    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-67c5866966-hssnd" event={"ID":"819152a5-89ff-4974-bfb7-7025060775b1","Type":"ContainerStarted","Data":"b48cfab43692af91b69e6da3f8f24eb4c5bfa3c52274ce680642bb502b461297"}
Jul 28 04:16:26.071258 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:26.071217    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m9zx6" event={"ID":"a02a02e4-e2b1-45fd-aeda-12fa4e29a996","Type":"ContainerStarted","Data":"8d8b333b9de3438cb084ec0e6db403f2aa7cb22a05d70fa3756872de3b522025"}
Jul 28 04:16:26.072738 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:26.072690    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-jxd2j" event={"ID":"c58f3317-e522-41a3-848f-906af440a23b","Type":"ContainerStarted","Data":"a62f388acc8b45ffdb9df138b46a4c134942361e2375cf0ed6ed112be9a9b5b1"}
Jul 28 04:16:26.074396 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:26.074372    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn" event={"ID":"55cd6e46-995c-4857-b7b7-c90e4e4ff93d","Type":"ContainerStarted","Data":"5977c6e9258debfac052de32987d5a83fadf4747c247f59b3a000a1456b4569d"}
Jul 28 04:16:26.074490 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:26.074413    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn" event={"ID":"55cd6e46-995c-4857-b7b7-c90e4e4ff93d","Type":"ContainerStarted","Data":"2f5f833c60eb502e81c5abf1943859e45b0cf6826d97ac574fa0e46b2cdfe285"}
Jul 28 04:16:26.074571 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:26.074550    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn"
Jul 28 04:16:26.093550 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:26.093499    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn" podStartSLOduration=285.093479395 podStartE2EDuration="4m45.093479395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:11:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:16:26.092049559 +0000 UTC m=+286.482225592" watchObservedRunningTime="2026-07-28 04:16:26.093479395 +0000 UTC m=+286.483655417"
Jul 28 04:16:27.081471 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.081209    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-jxd2j" event={"ID":"c58f3317-e522-41a3-848f-906af440a23b","Type":"ContainerStarted","Data":"de23b5179ff8a39c4998cefd8ff9cc6f85a8ba66bee9867a11a2dc70504a10c0"}
Jul 28 04:16:27.099959 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.099903    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-jxd2j" podStartSLOduration=282.677071779 podStartE2EDuration="4m44.099884919s" totalImagesPullingTime="1.42281314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:11:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:16:27.097571514 +0000 UTC m=+287.487747537" watchObservedRunningTime="2026-07-28 04:16:27.099884919 +0000 UTC m=+287.490060943"
Jul 28 04:16:27.452870 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.451738    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5df5d4d44b-ngmlc"]
Jul 28 04:16:27.463648 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.462588    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5df5d4d44b-ngmlc"
Jul 28 04:16:27.467188 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.467135    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 28 04:16:27.467891 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.467382    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 28 04:16:27.471427 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.471168    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 28 04:16:27.471427 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.471273    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7n5sw\""
Jul 28 04:16:27.471584 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.471523    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 28 04:16:27.472242 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.472028    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 28 04:16:27.472242 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.472204    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5df5d4d44b-ngmlc"]
Jul 28 04:16:27.477392 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.477364    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 04:16:27.491445 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.491417    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.498424 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.497351    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dmmeekb5lj9gh\""
Jul 28 04:16:27.498424 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.497549    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 28 04:16:27.498424 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.497736    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 28 04:16:27.498424 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.498274    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 28 04:16:27.506102 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.505697    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 28 04:16:27.506102 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.505740    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 28 04:16:27.506102 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.505974    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 28 04:16:27.506313 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.506148    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 28 04:16:27.506375 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.506343    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 28 04:16:27.506541 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.506524    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 28 04:16:27.506974 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.506524    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 28 04:16:27.508276 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.506569    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 28 04:16:27.508528 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.507197    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-mvl8j\""
Jul 28 04:16:27.508830 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.507265    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 28 04:16:27.509151 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.507286    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 28 04:16:27.512410 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.512390    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 28 04:16:27.544622 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.544590    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 04:16:27.567539 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.567505    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-config-out\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.567708 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.567552    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.567708 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.567582    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3ec31cf3-eec3-4c57-96e6-fe88637a67c3-console-config\") pod \"console-5df5d4d44b-ngmlc\" (UID: \"3ec31cf3-eec3-4c57-96e6-fe88637a67c3\") " pod="openshift-console/console-5df5d4d44b-ngmlc"
Jul 28 04:16:27.567708 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.567610    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3ec31cf3-eec3-4c57-96e6-fe88637a67c3-console-serving-cert\") pod \"console-5df5d4d44b-ngmlc\" (UID: \"3ec31cf3-eec3-4c57-96e6-fe88637a67c3\") " pod="openshift-console/console-5df5d4d44b-ngmlc"
Jul 28 04:16:27.567708 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.567640    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.567708 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.567664    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.567708 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.567692    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.568338 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.567717    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.568338 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.567740    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.568338 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.567773    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8w4c\" (UniqueName: \"kubernetes.io/projected/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-kube-api-access-k8w4c\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.568338 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.567801    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3ec31cf3-eec3-4c57-96e6-fe88637a67c3-oauth-serving-cert\") pod \"console-5df5d4d44b-ngmlc\" (UID: \"3ec31cf3-eec3-4c57-96e6-fe88637a67c3\") " pod="openshift-console/console-5df5d4d44b-ngmlc"
Jul 28 04:16:27.568338 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.567853    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.568338 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.567882    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-web-config\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.568338 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.568287    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.568703 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.568355    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3ec31cf3-eec3-4c57-96e6-fe88637a67c3-console-oauth-config\") pod \"console-5df5d4d44b-ngmlc\" (UID: \"3ec31cf3-eec3-4c57-96e6-fe88637a67c3\") " pod="openshift-console/console-5df5d4d44b-ngmlc"
Jul 28 04:16:27.568703 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.568393    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.568703 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.568470    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.568703 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.568531    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25nhs\" (UniqueName: \"kubernetes.io/projected/3ec31cf3-eec3-4c57-96e6-fe88637a67c3-kube-api-access-25nhs\") pod \"console-5df5d4d44b-ngmlc\" (UID: \"3ec31cf3-eec3-4c57-96e6-fe88637a67c3\") " pod="openshift-console/console-5df5d4d44b-ngmlc"
Jul 28 04:16:27.568703 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.568620    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-config\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.568703 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.568689    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.569104 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.568788    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3ec31cf3-eec3-4c57-96e6-fe88637a67c3-service-ca\") pod \"console-5df5d4d44b-ngmlc\" (UID: \"3ec31cf3-eec3-4c57-96e6-fe88637a67c3\") " pod="openshift-console/console-5df5d4d44b-ngmlc"
Jul 28 04:16:27.569104 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.568945    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.569104 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.569024    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.570018 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.569102    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.570018 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.569896    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.570018 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.569926    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.671026 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.670386    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.671026 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.670433    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-web-config\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.671026 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.670458    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.671026 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.670490    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3ec31cf3-eec3-4c57-96e6-fe88637a67c3-console-oauth-config\") pod \"console-5df5d4d44b-ngmlc\" (UID: \"3ec31cf3-eec3-4c57-96e6-fe88637a67c3\") " pod="openshift-console/console-5df5d4d44b-ngmlc"
Jul 28 04:16:27.671026 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.670527    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.671026 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.670551    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.671026 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.670583    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25nhs\" (UniqueName: \"kubernetes.io/projected/3ec31cf3-eec3-4c57-96e6-fe88637a67c3-kube-api-access-25nhs\") pod \"console-5df5d4d44b-ngmlc\" (UID: \"3ec31cf3-eec3-4c57-96e6-fe88637a67c3\") " pod="openshift-console/console-5df5d4d44b-ngmlc"
Jul 28 04:16:27.671026 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.670631    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-config\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.671548 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.671299    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.671548 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.671350    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.671548 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.671494    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3ec31cf3-eec3-4c57-96e6-fe88637a67c3-service-ca\") pod \"console-5df5d4d44b-ngmlc\" (UID: \"3ec31cf3-eec3-4c57-96e6-fe88637a67c3\") " pod="openshift-console/console-5df5d4d44b-ngmlc"
Jul 28 04:16:27.671705 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.671591    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.671705 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.671639    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.671705 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.671667    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.671866 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.671729    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.671866 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.671778    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.671866 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.671809    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-config-out\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.672016 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.671868    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.672016 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.671886    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.672016 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.671922    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3ec31cf3-eec3-4c57-96e6-fe88637a67c3-console-config\") pod \"console-5df5d4d44b-ngmlc\" (UID: \"3ec31cf3-eec3-4c57-96e6-fe88637a67c3\") " pod="openshift-console/console-5df5d4d44b-ngmlc"
Jul 28 04:16:27.672016 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.671988    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3ec31cf3-eec3-4c57-96e6-fe88637a67c3-console-serving-cert\") pod \"console-5df5d4d44b-ngmlc\" (UID: \"3ec31cf3-eec3-4c57-96e6-fe88637a67c3\") " pod="openshift-console/console-5df5d4d44b-ngmlc"
Jul 28 04:16:27.672206 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.672047    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.672206 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.672074    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.672206 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.672119    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.672206 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.672151    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.672206 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.672194    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.672442 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.672232    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8w4c\" (UniqueName: \"kubernetes.io/projected/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-kube-api-access-k8w4c\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.672442 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.672286    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3ec31cf3-eec3-4c57-96e6-fe88637a67c3-oauth-serving-cert\") pod \"console-5df5d4d44b-ngmlc\" (UID: \"3ec31cf3-eec3-4c57-96e6-fe88637a67c3\") " pod="openshift-console/console-5df5d4d44b-ngmlc"
Jul 28 04:16:27.672540 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.672442    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.673124 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.673097    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.673530 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.673425    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3ec31cf3-eec3-4c57-96e6-fe88637a67c3-oauth-serving-cert\") pod \"console-5df5d4d44b-ngmlc\" (UID: \"3ec31cf3-eec3-4c57-96e6-fe88637a67c3\") " pod="openshift-console/console-5df5d4d44b-ngmlc"
Jul 28 04:16:27.674015 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.673979    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3ec31cf3-eec3-4c57-96e6-fe88637a67c3-service-ca\") pod \"console-5df5d4d44b-ngmlc\" (UID: \"3ec31cf3-eec3-4c57-96e6-fe88637a67c3\") " pod="openshift-console/console-5df5d4d44b-ngmlc"
Jul 28 04:16:27.677434 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.677412    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.677603 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.677515    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.678112 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.678087    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3ec31cf3-eec3-4c57-96e6-fe88637a67c3-console-config\") pod \"console-5df5d4d44b-ngmlc\" (UID: \"3ec31cf3-eec3-4c57-96e6-fe88637a67c3\") " pod="openshift-console/console-5df5d4d44b-ngmlc"
Jul 28 04:16:27.678961 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.678942    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.679311 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.679214    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.679940 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.679907    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3ec31cf3-eec3-4c57-96e6-fe88637a67c3-console-oauth-config\") pod \"console-5df5d4d44b-ngmlc\" (UID: \"3ec31cf3-eec3-4c57-96e6-fe88637a67c3\") " pod="openshift-console/console-5df5d4d44b-ngmlc"
Jul 28 04:16:27.682312 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.682291    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-web-config\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.687103 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.686984    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.687103 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.687001    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-config\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.690558 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.690541    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8w4c\" (UniqueName: \"kubernetes.io/projected/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-kube-api-access-k8w4c\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.691819 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.691777    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.692382 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.692338    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.692862 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.692803    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.693330 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.693292    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-config-out\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.693941 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.693906    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.694464 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.694423    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.696681 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.696410    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3ec31cf3-eec3-4c57-96e6-fe88637a67c3-console-serving-cert\") pod \"console-5df5d4d44b-ngmlc\" (UID: \"3ec31cf3-eec3-4c57-96e6-fe88637a67c3\") " pod="openshift-console/console-5df5d4d44b-ngmlc"
Jul 28 04:16:27.696681 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.696638    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.697657 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.697044    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25nhs\" (UniqueName: \"kubernetes.io/projected/3ec31cf3-eec3-4c57-96e6-fe88637a67c3-kube-api-access-25nhs\") pod \"console-5df5d4d44b-ngmlc\" (UID: \"3ec31cf3-eec3-4c57-96e6-fe88637a67c3\") " pod="openshift-console/console-5df5d4d44b-ngmlc"
Jul 28 04:16:27.698352 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.698332    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:27.791041 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.790746    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5df5d4d44b-ngmlc"
Jul 28 04:16:27.806434 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:27.806399    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:28.785423 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:28.784753    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 04:16:28.789226 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:16:28.789200    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod981d6a0f_d0e1_491e_803c_e6c81b0e0a40.slice/crio-e8d8816deb3b0807ce9db8ca069ec4014fd117e0959f70385e90997647a11535 WatchSource:0}: Error finding container e8d8816deb3b0807ce9db8ca069ec4014fd117e0959f70385e90997647a11535: Status 404 returned error can't find the container with id e8d8816deb3b0807ce9db8ca069ec4014fd117e0959f70385e90997647a11535
Jul 28 04:16:28.796769 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:28.796695    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5df5d4d44b-ngmlc"]
Jul 28 04:16:28.803052 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:16:28.803015    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ec31cf3_eec3_4c57_96e6_fe88637a67c3.slice/crio-92fa9412f54efe9f84f91d4ff0797ec035bf00936ca68822ac3c3245d9369609 WatchSource:0}: Error finding container 92fa9412f54efe9f84f91d4ff0797ec035bf00936ca68822ac3c3245d9369609: Status 404 returned error can't find the container with id 92fa9412f54efe9f84f91d4ff0797ec035bf00936ca68822ac3c3245d9369609
Jul 28 04:16:29.092427 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:29.092345    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"981d6a0f-d0e1-491e-803c-e6c81b0e0a40","Type":"ContainerStarted","Data":"e8d8816deb3b0807ce9db8ca069ec4014fd117e0959f70385e90997647a11535"}
Jul 28 04:16:29.093760 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:29.093719    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5df5d4d44b-ngmlc" event={"ID":"3ec31cf3-eec3-4c57-96e6-fe88637a67c3","Type":"ContainerStarted","Data":"92fa9412f54efe9f84f91d4ff0797ec035bf00936ca68822ac3c3245d9369609"}
Jul 28 04:16:29.095460 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:29.095430    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-67c5866966-hssnd" event={"ID":"819152a5-89ff-4974-bfb7-7025060775b1","Type":"ContainerStarted","Data":"520ee6e792749c93e6bf2a84ee3ff005c6b48a9b1af72f63cbf545605b9e30f3"}
Jul 28 04:16:29.097450 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:29.097418    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m9zx6" event={"ID":"a02a02e4-e2b1-45fd-aeda-12fa4e29a996","Type":"ContainerStarted","Data":"002525845ee91ceaf54d9ce4df75bbde4b107f448e1a77b1e766c7ad9359b316"}
Jul 28 04:16:29.097550 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:29.097470    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-m9zx6" event={"ID":"a02a02e4-e2b1-45fd-aeda-12fa4e29a996","Type":"ContainerStarted","Data":"993b42d37b3582e64d98cf62a0429560fd657a331bc404d2725354d2aecea08d"}
Jul 28 04:16:29.097613 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:29.097546    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-m9zx6"
Jul 28 04:16:29.111399 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:29.111354    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-67c5866966-hssnd" podStartSLOduration=1.528770131 podStartE2EDuration="4.111335724s" totalImagesPullingTime="2.582565593s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:16:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:16:29.110342023 +0000 UTC m=+289.500518044" watchObservedRunningTime="2026-07-28 04:16:29.111335724 +0000 UTC m=+289.501511773"
Jul 28 04:16:29.126372 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:29.126288    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-m9zx6" podStartSLOduration=251.265797195 podStartE2EDuration="4m14.126269118s" totalImagesPullingTime="2.860471923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:12:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:16:29.124705282 +0000 UTC m=+289.514881303" watchObservedRunningTime="2026-07-28 04:16:29.126269118 +0000 UTC m=+289.516445141"
Jul 28 04:16:31.109179 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:31.109144    2566 generic.go:356] "Generic (PLEG): container finished" podID="981d6a0f-d0e1-491e-803c-e6c81b0e0a40" containerID="57072d126e9f2f6421f153ed376900b6f1ef209ce951c824f82ed5b34a06c6dd" exitCode=0
Jul 28 04:16:31.109606 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:31.109222    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"981d6a0f-d0e1-491e-803c-e6c81b0e0a40","Type":"ContainerDied","Data":"57072d126e9f2f6421f153ed376900b6f1ef209ce951c824f82ed5b34a06c6dd"}
Jul 28 04:16:33.091241 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:16:33.086616    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-l6p44" podUID="e6107883-62a6-4a82-9e04-ed46057a6b0e"
Jul 28 04:16:33.121588 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:33.121547    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5df5d4d44b-ngmlc" event={"ID":"3ec31cf3-eec3-4c57-96e6-fe88637a67c3","Type":"ContainerStarted","Data":"e707703701dfe264c1bdd96a2b05b697e82e6ab24134f0c32d322d7bccbd559a"}
Jul 28 04:16:33.137942 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:33.137891    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5df5d4d44b-ngmlc" podStartSLOduration=2.703391153 podStartE2EDuration="6.137876848s" totalImagesPullingTime="3.434485695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:16:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:16:33.137350693 +0000 UTC m=+293.527526717" watchObservedRunningTime="2026-07-28 04:16:33.137876848 +0000 UTC m=+293.528052868"
Jul 28 04:16:37.791451 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:37.791412    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5df5d4d44b-ngmlc"
Jul 28 04:16:37.792022 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:37.791477    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5df5d4d44b-ngmlc"
Jul 28 04:16:37.793073 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:37.793033    2566 patch_prober.go:28] interesting pod/console-5df5d4d44b-ngmlc container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.22:8443/health\": dial tcp 10.132.0.22:8443: connect: connection refused" start-of-body=
Jul 28 04:16:37.793194 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:37.793115    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-5df5d4d44b-ngmlc" podUID="3ec31cf3-eec3-4c57-96e6-fe88637a67c3" containerName="console" probeResult="failure" output="Get \"https://10.132.0.22:8443/health\": dial tcp 10.132.0.22:8443: connect: connection refused"
Jul 28 04:16:39.105009 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:39.104974    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-m9zx6"
Jul 28 04:16:42.957136 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:42.957107    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4glpr_bfd2d3d8-f287-42d8-83c7-f82185c96735/console-operator/2.log"
Jul 28 04:16:42.957912 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:42.957890    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4glpr_bfd2d3d8-f287-42d8-83c7-f82185c96735/console-operator/2.log"
Jul 28 04:16:42.968374 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:42.968351    2566 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 04:16:43.109111 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:43.109090    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 04:16:43.165448 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:43.165407    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-rs96k" event={"ID":"e1817115-022a-447e-9431-62a0d5c2e4c6","Type":"ContainerStarted","Data":"c09748b40d5da015a626f4f6440f4876227d148cf6b7c11c4f502dc638561ba2"}
Jul 28 04:16:43.165606 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:43.165515    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-rs96k"
Jul 28 04:16:43.166882 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:43.166852    2566 patch_prober.go:28] interesting pod/downloads-7c97dc4555-rs96k container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.20:8080/\": dial tcp 10.132.0.20:8080: connect: connection refused" start-of-body=
Jul 28 04:16:43.166987 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:43.166911    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-7c97dc4555-rs96k" podUID="e1817115-022a-447e-9431-62a0d5c2e4c6" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.20:8080/\": dial tcp 10.132.0.20:8080: connect: connection refused"
Jul 28 04:16:43.168117 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:43.168092    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"981d6a0f-d0e1-491e-803c-e6c81b0e0a40","Type":"ContainerStarted","Data":"fe966e374a546305195a0039383a543bf32d46ebd9c69bad9ff734888c263c4d"}
Jul 28 04:16:43.168200 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:43.168174    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"981d6a0f-d0e1-491e-803c-e6c81b0e0a40","Type":"ContainerStarted","Data":"9cd004fd3e50129ed6f8b124333db1431ab0b936b7d04629d7894a9c6e4a87ec"}
Jul 28 04:16:44.177592 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:44.177526    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-rs96k"
Jul 28 04:16:44.197222 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:44.197163    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-rs96k" podStartSLOduration=1.7676229989999999 podStartE2EDuration="21.197143003s" totalImagesPullingTime="19.429520004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:16:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:16:43.1890605 +0000 UTC m=+303.579236523" watchObservedRunningTime="2026-07-28 04:16:44.197143003 +0000 UTC m=+304.587319042"
Jul 28 04:16:45.352923 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:45.352884    2566 patch_prober.go:28] interesting pod/image-registry-66ff9f7dd7-45ggn container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 28 04:16:45.353355 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:45.352941    2566 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn" podUID="55cd6e46-995c-4857-b7b7-c90e4e4ff93d" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 04:16:45.854945 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:45.854914    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-67c5866966-hssnd"
Jul 28 04:16:45.855097 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:45.854975    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-67c5866966-hssnd"
Jul 28 04:16:46.082256 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:46.082232    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6p44"
Jul 28 04:16:46.186573 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:46.186537    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"981d6a0f-d0e1-491e-803c-e6c81b0e0a40","Type":"ContainerStarted","Data":"d5e65baccc4e3374550788db2407d911a0016b90c794153433c8eda4b324de04"}
Jul 28 04:16:46.186709 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:46.186626    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"981d6a0f-d0e1-491e-803c-e6c81b0e0a40","Type":"ContainerStarted","Data":"3de83ec883b6c1be0d49cd644963cc573bb6695d513f8ce18fbceca519429433"}
Jul 28 04:16:46.186709 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:46.186661    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"981d6a0f-d0e1-491e-803c-e6c81b0e0a40","Type":"ContainerStarted","Data":"90e9428ac89fed20cb9e237f23eb39faa71f5327fc5dc06d4f29b0022511b4fb"}
Jul 28 04:16:46.186808 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:46.186699    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"981d6a0f-d0e1-491e-803c-e6c81b0e0a40","Type":"ContainerStarted","Data":"e8f713475790f4f893142e7e142ae100604b444d6ca27090bd46ed08253c8f96"}
Jul 28 04:16:46.215163 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:46.215115    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.428517024 podStartE2EDuration="19.215095702s" totalImagesPullingTime="15.76499736s" totalInitContainerRuntime="21.581318ms" isStatefulPod=false podCreationTimestamp="2026-07-28 04:16:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:16:46.212331311 +0000 UTC m=+306.602507334" watchObservedRunningTime="2026-07-28 04:16:46.215095702 +0000 UTC m=+306.605271723"
Jul 28 04:16:47.086669 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:47.086638    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn"
Jul 28 04:16:47.796979 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:47.796947    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5df5d4d44b-ngmlc"
Jul 28 04:16:47.801926 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:47.801893    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5df5d4d44b-ngmlc"
Jul 28 04:16:47.807518 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:47.807495    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:16:47.916436 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:47.916402    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5df5d4d44b-ngmlc"]
Jul 28 04:16:52.215362 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:52.215318    2566 generic.go:356] "Generic (PLEG): container finished" podID="317c317c-02c3-4e8f-be22-5b81aea93846" containerID="d742b9c59c22f09980884fbd7f2ab6aa7557f820575a5b8e0c7c88ae63391eb0" exitCode=0
Jul 28 04:16:52.215901 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:52.215395    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-bcwm5" event={"ID":"317c317c-02c3-4e8f-be22-5b81aea93846","Type":"ContainerDied","Data":"d742b9c59c22f09980884fbd7f2ab6aa7557f820575a5b8e0c7c88ae63391eb0"}
Jul 28 04:16:52.215901 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:52.215774    2566 scope.go:117] "RemoveContainer" containerID="d742b9c59c22f09980884fbd7f2ab6aa7557f820575a5b8e0c7c88ae63391eb0"
Jul 28 04:16:52.217251 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:52.217230    2566 generic.go:356] "Generic (PLEG): container finished" podID="3837b644-a03e-498c-9033-1ed2e1d894d0" containerID="93dae67459983476f248788a47320c3e232911cba81b872f450ca944dc7248dc" exitCode=0
Jul 28 04:16:52.217381 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:52.217268    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-d2lpq" event={"ID":"3837b644-a03e-498c-9033-1ed2e1d894d0","Type":"ContainerDied","Data":"93dae67459983476f248788a47320c3e232911cba81b872f450ca944dc7248dc"}
Jul 28 04:16:52.217515 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:52.217502    2566 scope.go:117] "RemoveContainer" containerID="93dae67459983476f248788a47320c3e232911cba81b872f450ca944dc7248dc"
Jul 28 04:16:53.223256 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:53.223219    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-bcwm5" event={"ID":"317c317c-02c3-4e8f-be22-5b81aea93846","Type":"ContainerStarted","Data":"e5bf315f552ba6eb8f1de19de4b51ab7f2e7bb8a320c1f2a3d1843cdd6142c99"}
Jul 28 04:16:53.224888 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:53.224863    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-d2lpq" event={"ID":"3837b644-a03e-498c-9033-1ed2e1d894d0","Type":"ContainerStarted","Data":"d97b63c982daa8733bc6186a3116d7e8a4fe787e8a911a930572098484ff16c4"}
Jul 28 04:16:56.827338 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:16:56.827306    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-66ff9f7dd7-45ggn"]
Jul 28 04:17:05.859560 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:05.859528    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-67c5866966-hssnd"
Jul 28 04:17:05.863247 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:05.863222    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-67c5866966-hssnd"
Jul 28 04:17:12.297536 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:12.297501    2566 generic.go:356] "Generic (PLEG): container finished" podID="f77a6b03-d007-46e3-852b-4b262feca301" containerID="d63987c0d58ddb213f3007d703025d26e1312afe8b5e0884e5a6bf3b71ff2326" exitCode=0
Jul 28 04:17:12.298059 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:12.297580    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-lmrmn" event={"ID":"f77a6b03-d007-46e3-852b-4b262feca301","Type":"ContainerDied","Data":"d63987c0d58ddb213f3007d703025d26e1312afe8b5e0884e5a6bf3b71ff2326"}
Jul 28 04:17:12.298059 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:12.298047    2566 scope.go:117] "RemoveContainer" containerID="d63987c0d58ddb213f3007d703025d26e1312afe8b5e0884e5a6bf3b71ff2326"
Jul 28 04:17:13.303102 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:13.303073    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-lmrmn" event={"ID":"f77a6b03-d007-46e3-852b-4b262feca301","Type":"ContainerStarted","Data":"526a06d1518ccd63403056e2dffd9da9b0d00fce22e09290b62539e13e549ec2"}
Jul 28 04:17:14.220821 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:14.220783    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5df5d4d44b-ngmlc" podUID="3ec31cf3-eec3-4c57-96e6-fe88637a67c3" containerName="console" containerID="cri-o://e707703701dfe264c1bdd96a2b05b697e82e6ab24134f0c32d322d7bccbd559a" gracePeriod=15
Jul 28 04:17:14.493465 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:14.493404    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5df5d4d44b-ngmlc"
Jul 28 04:17:14.613980 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:14.613946    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3ec31cf3-eec3-4c57-96e6-fe88637a67c3-console-oauth-config\") pod \"3ec31cf3-eec3-4c57-96e6-fe88637a67c3\" (UID: \"3ec31cf3-eec3-4c57-96e6-fe88637a67c3\") "
Jul 28 04:17:14.613980 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:14.613998    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3ec31cf3-eec3-4c57-96e6-fe88637a67c3-console-serving-cert\") pod \"3ec31cf3-eec3-4c57-96e6-fe88637a67c3\" (UID: \"3ec31cf3-eec3-4c57-96e6-fe88637a67c3\") "
Jul 28 04:17:14.614248 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:14.614017    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3ec31cf3-eec3-4c57-96e6-fe88637a67c3-console-config\") pod \"3ec31cf3-eec3-4c57-96e6-fe88637a67c3\" (UID: \"3ec31cf3-eec3-4c57-96e6-fe88637a67c3\") "
Jul 28 04:17:14.614248 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:14.614034    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3ec31cf3-eec3-4c57-96e6-fe88637a67c3-service-ca\") pod \"3ec31cf3-eec3-4c57-96e6-fe88637a67c3\" (UID: \"3ec31cf3-eec3-4c57-96e6-fe88637a67c3\") "
Jul 28 04:17:14.614248 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:14.614094    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3ec31cf3-eec3-4c57-96e6-fe88637a67c3-oauth-serving-cert\") pod \"3ec31cf3-eec3-4c57-96e6-fe88637a67c3\" (UID: \"3ec31cf3-eec3-4c57-96e6-fe88637a67c3\") "
Jul 28 04:17:14.614248 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:14.614159    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25nhs\" (UniqueName: \"kubernetes.io/projected/3ec31cf3-eec3-4c57-96e6-fe88637a67c3-kube-api-access-25nhs\") pod \"3ec31cf3-eec3-4c57-96e6-fe88637a67c3\" (UID: \"3ec31cf3-eec3-4c57-96e6-fe88637a67c3\") "
Jul 28 04:17:14.614473 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:14.614442    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ec31cf3-eec3-4c57-96e6-fe88637a67c3-console-config" (OuterVolumeSpecName: "console-config") pod "3ec31cf3-eec3-4c57-96e6-fe88637a67c3" (UID: "3ec31cf3-eec3-4c57-96e6-fe88637a67c3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 04:17:14.614473 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:14.614464    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ec31cf3-eec3-4c57-96e6-fe88637a67c3-service-ca" (OuterVolumeSpecName: "service-ca") pod "3ec31cf3-eec3-4c57-96e6-fe88637a67c3" (UID: "3ec31cf3-eec3-4c57-96e6-fe88637a67c3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 04:17:14.614577 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:14.614550    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ec31cf3-eec3-4c57-96e6-fe88637a67c3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3ec31cf3-eec3-4c57-96e6-fe88637a67c3" (UID: "3ec31cf3-eec3-4c57-96e6-fe88637a67c3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 04:17:14.616368 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:14.616335    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ec31cf3-eec3-4c57-96e6-fe88637a67c3-kube-api-access-25nhs" (OuterVolumeSpecName: "kube-api-access-25nhs") pod "3ec31cf3-eec3-4c57-96e6-fe88637a67c3" (UID: "3ec31cf3-eec3-4c57-96e6-fe88637a67c3"). InnerVolumeSpecName "kube-api-access-25nhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:17:14.616752 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:14.616728    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ec31cf3-eec3-4c57-96e6-fe88637a67c3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3ec31cf3-eec3-4c57-96e6-fe88637a67c3" (UID: "3ec31cf3-eec3-4c57-96e6-fe88637a67c3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 04:17:14.616809 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:14.616746    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ec31cf3-eec3-4c57-96e6-fe88637a67c3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3ec31cf3-eec3-4c57-96e6-fe88637a67c3" (UID: "3ec31cf3-eec3-4c57-96e6-fe88637a67c3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 04:17:14.715497 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:14.715460    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25nhs\" (UniqueName: \"kubernetes.io/projected/3ec31cf3-eec3-4c57-96e6-fe88637a67c3-kube-api-access-25nhs\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:17:14.715497 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:14.715490    2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3ec31cf3-eec3-4c57-96e6-fe88637a67c3-console-oauth-config\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:17:14.715497 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:14.715503    2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3ec31cf3-eec3-4c57-96e6-fe88637a67c3-console-serving-cert\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:17:14.715728 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:14.715515    2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3ec31cf3-eec3-4c57-96e6-fe88637a67c3-console-config\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:17:14.715728 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:14.715527    2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3ec31cf3-eec3-4c57-96e6-fe88637a67c3-service-ca\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:17:14.715728 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:14.715538    2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3ec31cf3-eec3-4c57-96e6-fe88637a67c3-oauth-serving-cert\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:17:15.311152 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:15.311114    2566 generic.go:356] "Generic (PLEG): container finished" podID="3ec31cf3-eec3-4c57-96e6-fe88637a67c3" containerID="e707703701dfe264c1bdd96a2b05b697e82e6ab24134f0c32d322d7bccbd559a" exitCode=0
Jul 28 04:17:15.311318 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:15.311164    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5df5d4d44b-ngmlc"
Jul 28 04:17:15.311318 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:15.311189    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5df5d4d44b-ngmlc" event={"ID":"3ec31cf3-eec3-4c57-96e6-fe88637a67c3","Type":"ContainerDied","Data":"e707703701dfe264c1bdd96a2b05b697e82e6ab24134f0c32d322d7bccbd559a"}
Jul 28 04:17:15.311318 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:15.311236    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5df5d4d44b-ngmlc" event={"ID":"3ec31cf3-eec3-4c57-96e6-fe88637a67c3","Type":"ContainerDied","Data":"92fa9412f54efe9f84f91d4ff0797ec035bf00936ca68822ac3c3245d9369609"}
Jul 28 04:17:15.311318 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:15.311263    2566 scope.go:117] "RemoveContainer" containerID="e707703701dfe264c1bdd96a2b05b697e82e6ab24134f0c32d322d7bccbd559a"
Jul 28 04:17:15.320441 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:15.320421    2566 scope.go:117] "RemoveContainer" containerID="e707703701dfe264c1bdd96a2b05b697e82e6ab24134f0c32d322d7bccbd559a"
Jul 28 04:17:15.320732 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:17:15.320714    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e707703701dfe264c1bdd96a2b05b697e82e6ab24134f0c32d322d7bccbd559a\": container with ID starting with e707703701dfe264c1bdd96a2b05b697e82e6ab24134f0c32d322d7bccbd559a not found: ID does not exist" containerID="e707703701dfe264c1bdd96a2b05b697e82e6ab24134f0c32d322d7bccbd559a"
Jul 28 04:17:15.320775 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:15.320739    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e707703701dfe264c1bdd96a2b05b697e82e6ab24134f0c32d322d7bccbd559a"} err="failed to get container status \"e707703701dfe264c1bdd96a2b05b697e82e6ab24134f0c32d322d7bccbd559a\": rpc error: code = NotFound desc = could not find container \"e707703701dfe264c1bdd96a2b05b697e82e6ab24134f0c32d322d7bccbd559a\": container with ID starting with e707703701dfe264c1bdd96a2b05b697e82e6ab24134f0c32d322d7bccbd559a not found: ID does not exist"
Jul 28 04:17:15.331582 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:15.331556    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5df5d4d44b-ngmlc"]
Jul 28 04:17:15.334533 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:15.334511    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5df5d4d44b-ngmlc"]
Jul 28 04:17:16.087092 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:16.087056    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ec31cf3-eec3-4c57-96e6-fe88637a67c3" path="/var/lib/kubelet/pods/3ec31cf3-eec3-4c57-96e6-fe88637a67c3/volumes"
Jul 28 04:17:21.846146 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:21.846103    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn" podUID="55cd6e46-995c-4857-b7b7-c90e4e4ff93d" containerName="registry" containerID="cri-o://5977c6e9258debfac052de32987d5a83fadf4747c247f59b3a000a1456b4569d" gracePeriod=30
Jul 28 04:17:22.115091 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:22.115038    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn"
Jul 28 04:17:22.183894 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:22.183867    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-installation-pull-secrets\") pod \"55cd6e46-995c-4857-b7b7-c90e4e4ff93d\" (UID: \"55cd6e46-995c-4857-b7b7-c90e4e4ff93d\") "
Jul 28 04:17:22.184073 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:22.183906    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-registry-tls\") pod \"55cd6e46-995c-4857-b7b7-c90e4e4ff93d\" (UID: \"55cd6e46-995c-4857-b7b7-c90e4e4ff93d\") "
Jul 28 04:17:22.184073 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:22.183972    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-image-registry-private-configuration\") pod \"55cd6e46-995c-4857-b7b7-c90e4e4ff93d\" (UID: \"55cd6e46-995c-4857-b7b7-c90e4e4ff93d\") "
Jul 28 04:17:22.184073 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:22.184003    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-bound-sa-token\") pod \"55cd6e46-995c-4857-b7b7-c90e4e4ff93d\" (UID: \"55cd6e46-995c-4857-b7b7-c90e4e4ff93d\") "
Jul 28 04:17:22.184073 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:22.184025    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-registry-certificates\") pod \"55cd6e46-995c-4857-b7b7-c90e4e4ff93d\" (UID: \"55cd6e46-995c-4857-b7b7-c90e4e4ff93d\") "
Jul 28 04:17:22.184299 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:22.184079    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-ca-trust-extracted\") pod \"55cd6e46-995c-4857-b7b7-c90e4e4ff93d\" (UID: \"55cd6e46-995c-4857-b7b7-c90e4e4ff93d\") "
Jul 28 04:17:22.184299 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:22.184121    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8rxc\" (UniqueName: \"kubernetes.io/projected/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-kube-api-access-v8rxc\") pod \"55cd6e46-995c-4857-b7b7-c90e4e4ff93d\" (UID: \"55cd6e46-995c-4857-b7b7-c90e4e4ff93d\") "
Jul 28 04:17:22.184299 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:22.184145    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-trusted-ca\") pod \"55cd6e46-995c-4857-b7b7-c90e4e4ff93d\" (UID: \"55cd6e46-995c-4857-b7b7-c90e4e4ff93d\") "
Jul 28 04:17:22.184533 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:22.184507    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "55cd6e46-995c-4857-b7b7-c90e4e4ff93d" (UID: "55cd6e46-995c-4857-b7b7-c90e4e4ff93d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 04:17:22.184854 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:22.184803    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "55cd6e46-995c-4857-b7b7-c90e4e4ff93d" (UID: "55cd6e46-995c-4857-b7b7-c90e4e4ff93d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 04:17:22.186336 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:22.186295    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "55cd6e46-995c-4857-b7b7-c90e4e4ff93d" (UID: "55cd6e46-995c-4857-b7b7-c90e4e4ff93d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:17:22.186553 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:22.186524    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "55cd6e46-995c-4857-b7b7-c90e4e4ff93d" (UID: "55cd6e46-995c-4857-b7b7-c90e4e4ff93d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 04:17:22.186654 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:22.186561    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "55cd6e46-995c-4857-b7b7-c90e4e4ff93d" (UID: "55cd6e46-995c-4857-b7b7-c90e4e4ff93d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:17:22.186854 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:22.186813    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-kube-api-access-v8rxc" (OuterVolumeSpecName: "kube-api-access-v8rxc") pod "55cd6e46-995c-4857-b7b7-c90e4e4ff93d" (UID: "55cd6e46-995c-4857-b7b7-c90e4e4ff93d"). InnerVolumeSpecName "kube-api-access-v8rxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:17:22.186924 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:22.186864    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "55cd6e46-995c-4857-b7b7-c90e4e4ff93d" (UID: "55cd6e46-995c-4857-b7b7-c90e4e4ff93d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 04:17:22.192740 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:22.192714    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "55cd6e46-995c-4857-b7b7-c90e4e4ff93d" (UID: "55cd6e46-995c-4857-b7b7-c90e4e4ff93d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:17:22.285472 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:22.285444    2566 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-registry-tls\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:17:22.285472 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:22.285470    2566 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-image-registry-private-configuration\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:17:22.285631 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:22.285484    2566 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-bound-sa-token\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:17:22.285631 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:22.285498    2566 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-registry-certificates\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:17:22.285631 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:22.285511    2566 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-ca-trust-extracted\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:17:22.285631 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:22.285524    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8rxc\" (UniqueName: \"kubernetes.io/projected/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-kube-api-access-v8rxc\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:17:22.285631 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:22.285535    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-trusted-ca\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:17:22.285631 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:22.285547    2566 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/55cd6e46-995c-4857-b7b7-c90e4e4ff93d-installation-pull-secrets\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:17:22.342860 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:22.342813    2566 generic.go:356] "Generic (PLEG): container finished" podID="55cd6e46-995c-4857-b7b7-c90e4e4ff93d" containerID="5977c6e9258debfac052de32987d5a83fadf4747c247f59b3a000a1456b4569d" exitCode=0
Jul 28 04:17:22.342951 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:22.342889    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn"
Jul 28 04:17:22.342951 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:22.342882    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn" event={"ID":"55cd6e46-995c-4857-b7b7-c90e4e4ff93d","Type":"ContainerDied","Data":"5977c6e9258debfac052de32987d5a83fadf4747c247f59b3a000a1456b4569d"}
Jul 28 04:17:22.342951 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:22.342939    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-66ff9f7dd7-45ggn" event={"ID":"55cd6e46-995c-4857-b7b7-c90e4e4ff93d","Type":"ContainerDied","Data":"2f5f833c60eb502e81c5abf1943859e45b0cf6826d97ac574fa0e46b2cdfe285"}
Jul 28 04:17:22.343052 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:22.342959    2566 scope.go:117] "RemoveContainer" containerID="5977c6e9258debfac052de32987d5a83fadf4747c247f59b3a000a1456b4569d"
Jul 28 04:17:22.356867 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:22.356819    2566 scope.go:117] "RemoveContainer" containerID="5977c6e9258debfac052de32987d5a83fadf4747c247f59b3a000a1456b4569d"
Jul 28 04:17:22.357164 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:17:22.357142    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5977c6e9258debfac052de32987d5a83fadf4747c247f59b3a000a1456b4569d\": container with ID starting with 5977c6e9258debfac052de32987d5a83fadf4747c247f59b3a000a1456b4569d not found: ID does not exist" containerID="5977c6e9258debfac052de32987d5a83fadf4747c247f59b3a000a1456b4569d"
Jul 28 04:17:22.357248 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:22.357169    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5977c6e9258debfac052de32987d5a83fadf4747c247f59b3a000a1456b4569d"} err="failed to get container status \"5977c6e9258debfac052de32987d5a83fadf4747c247f59b3a000a1456b4569d\": rpc error: code = NotFound desc = could not find container \"5977c6e9258debfac052de32987d5a83fadf4747c247f59b3a000a1456b4569d\": container with ID starting with 5977c6e9258debfac052de32987d5a83fadf4747c247f59b3a000a1456b4569d not found: ID does not exist"
Jul 28 04:17:22.366669 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:22.366616    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-66ff9f7dd7-45ggn"]
Jul 28 04:17:22.369718 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:22.369696    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-66ff9f7dd7-45ggn"]
Jul 28 04:17:24.087510 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:24.087468    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55cd6e46-995c-4857-b7b7-c90e4e4ff93d" path="/var/lib/kubelet/pods/55cd6e46-995c-4857-b7b7-c90e4e4ff93d/volumes"
Jul 28 04:17:27.807657 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:27.807615    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:27.823157 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:27.823135    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:28.380280 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:28.380254    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:45.775532 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:45.775493    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 04:17:45.776038 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:45.775878    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="981d6a0f-d0e1-491e-803c-e6c81b0e0a40" containerName="prometheus" containerID="cri-o://9cd004fd3e50129ed6f8b124333db1431ab0b936b7d04629d7894a9c6e4a87ec" gracePeriod=600
Jul 28 04:17:45.776118 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:45.775944    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="981d6a0f-d0e1-491e-803c-e6c81b0e0a40" containerName="thanos-sidecar" containerID="cri-o://e8f713475790f4f893142e7e142ae100604b444d6ca27090bd46ed08253c8f96" gracePeriod=600
Jul 28 04:17:45.776215 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:45.775934    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="981d6a0f-d0e1-491e-803c-e6c81b0e0a40" containerName="kube-rbac-proxy-thanos" containerID="cri-o://d5e65baccc4e3374550788db2407d911a0016b90c794153433c8eda4b324de04" gracePeriod=600
Jul 28 04:17:45.776281 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:45.775952    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="981d6a0f-d0e1-491e-803c-e6c81b0e0a40" containerName="kube-rbac-proxy-web" containerID="cri-o://90e9428ac89fed20cb9e237f23eb39faa71f5327fc5dc06d4f29b0022511b4fb" gracePeriod=600
Jul 28 04:17:45.776354 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:45.775974    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="981d6a0f-d0e1-491e-803c-e6c81b0e0a40" containerName="kube-rbac-proxy" containerID="cri-o://3de83ec883b6c1be0d49cd644963cc573bb6695d513f8ce18fbceca519429433" gracePeriod=600
Jul 28 04:17:45.776939 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:45.775990    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="981d6a0f-d0e1-491e-803c-e6c81b0e0a40" containerName="config-reloader" containerID="cri-o://fe966e374a546305195a0039383a543bf32d46ebd9c69bad9ff734888c263c4d" gracePeriod=600
Jul 28 04:17:46.036368 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.036343    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.084099 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.080177    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") "
Jul 28 04:17:46.084099 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.080228    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-tls-assets\") pod \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") "
Jul 28 04:17:46.084099 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.080271    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-prometheus-k8s-rulefiles-1\") pod \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") "
Jul 28 04:17:46.084099 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.080302    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-prometheus-k8s-rulefiles-0\") pod \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") "
Jul 28 04:17:46.084099 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.080342    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8w4c\" (UniqueName: \"kubernetes.io/projected/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-kube-api-access-k8w4c\") pod \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") "
Jul 28 04:17:46.084099 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.080369    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-config-out\") pod \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") "
Jul 28 04:17:46.084099 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.080401    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-configmap-serving-certs-ca-bundle\") pod \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") "
Jul 28 04:17:46.084099 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.080439    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-web-config\") pod \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") "
Jul 28 04:17:46.084099 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.080468    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-secret-prometheus-k8s-tls\") pod \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") "
Jul 28 04:17:46.084099 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.080509    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-configmap-kubelet-serving-ca-bundle\") pod \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") "
Jul 28 04:17:46.084099 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.080537    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-prometheus-k8s-rulefiles-2\") pod \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") "
Jul 28 04:17:46.084099 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.080564    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-configmap-metrics-client-ca\") pod \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") "
Jul 28 04:17:46.084099 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.080605    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-prometheus-trusted-ca-bundle\") pod \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") "
Jul 28 04:17:46.084099 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.080635    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-thanos-prometheus-http-client-file\") pod \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") "
Jul 28 04:17:46.084099 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.080676    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-secret-kube-rbac-proxy\") pod \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") "
Jul 28 04:17:46.084099 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.080702    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-secret-grpc-tls\") pod \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") "
Jul 28 04:17:46.084099 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.081672    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-secret-metrics-client-certs\") pod \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") "
Jul 28 04:17:46.084099 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.081707    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-config\") pod \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") "
Jul 28 04:17:46.085118 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.081742    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") "
Jul 28 04:17:46.085118 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.081769    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-prometheus-k8s-db\") pod \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\" (UID: \"981d6a0f-d0e1-491e-803c-e6c81b0e0a40\") "
Jul 28 04:17:46.085510 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.081039    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "981d6a0f-d0e1-491e-803c-e6c81b0e0a40" (UID: "981d6a0f-d0e1-491e-803c-e6c81b0e0a40"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 04:17:46.085601 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.081621    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "981d6a0f-d0e1-491e-803c-e6c81b0e0a40" (UID: "981d6a0f-d0e1-491e-803c-e6c81b0e0a40"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 04:17:46.085601 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.082768    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "981d6a0f-d0e1-491e-803c-e6c81b0e0a40" (UID: "981d6a0f-d0e1-491e-803c-e6c81b0e0a40"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 04:17:46.085601 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.082820    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "981d6a0f-d0e1-491e-803c-e6c81b0e0a40" (UID: "981d6a0f-d0e1-491e-803c-e6c81b0e0a40"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 04:17:46.085601 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.084710    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "981d6a0f-d0e1-491e-803c-e6c81b0e0a40" (UID: "981d6a0f-d0e1-491e-803c-e6c81b0e0a40"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 04:17:46.085601 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.084877    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "981d6a0f-d0e1-491e-803c-e6c81b0e0a40" (UID: "981d6a0f-d0e1-491e-803c-e6c81b0e0a40"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 04:17:46.086227 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.086200    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "981d6a0f-d0e1-491e-803c-e6c81b0e0a40" (UID: "981d6a0f-d0e1-491e-803c-e6c81b0e0a40"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:17:46.089011 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.088960    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "981d6a0f-d0e1-491e-803c-e6c81b0e0a40" (UID: "981d6a0f-d0e1-491e-803c-e6c81b0e0a40"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 04:17:46.091005 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.090964    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "981d6a0f-d0e1-491e-803c-e6c81b0e0a40" (UID: "981d6a0f-d0e1-491e-803c-e6c81b0e0a40"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 04:17:46.091230 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.091197    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "981d6a0f-d0e1-491e-803c-e6c81b0e0a40" (UID: "981d6a0f-d0e1-491e-803c-e6c81b0e0a40"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 04:17:46.091565 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.091387    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-config-out" (OuterVolumeSpecName: "config-out") pod "981d6a0f-d0e1-491e-803c-e6c81b0e0a40" (UID: "981d6a0f-d0e1-491e-803c-e6c81b0e0a40"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:17:46.091565 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.091471    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "981d6a0f-d0e1-491e-803c-e6c81b0e0a40" (UID: "981d6a0f-d0e1-491e-803c-e6c81b0e0a40"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 04:17:46.091565 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.091532    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-kube-api-access-k8w4c" (OuterVolumeSpecName: "kube-api-access-k8w4c") pod "981d6a0f-d0e1-491e-803c-e6c81b0e0a40" (UID: "981d6a0f-d0e1-491e-803c-e6c81b0e0a40"). InnerVolumeSpecName "kube-api-access-k8w4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:17:46.093895 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.092071    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "981d6a0f-d0e1-491e-803c-e6c81b0e0a40" (UID: "981d6a0f-d0e1-491e-803c-e6c81b0e0a40"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 04:17:46.094769 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.094726    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "981d6a0f-d0e1-491e-803c-e6c81b0e0a40" (UID: "981d6a0f-d0e1-491e-803c-e6c81b0e0a40"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 04:17:46.095015 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.094975    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "981d6a0f-d0e1-491e-803c-e6c81b0e0a40" (UID: "981d6a0f-d0e1-491e-803c-e6c81b0e0a40"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:17:46.095224 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.095187    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "981d6a0f-d0e1-491e-803c-e6c81b0e0a40" (UID: "981d6a0f-d0e1-491e-803c-e6c81b0e0a40"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 04:17:46.096151 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.096125    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-config" (OuterVolumeSpecName: "config") pod "981d6a0f-d0e1-491e-803c-e6c81b0e0a40" (UID: "981d6a0f-d0e1-491e-803c-e6c81b0e0a40"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 04:17:46.097227 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.097196    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "981d6a0f-d0e1-491e-803c-e6c81b0e0a40" (UID: "981d6a0f-d0e1-491e-803c-e6c81b0e0a40"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 04:17:46.104279 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.104254    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-web-config" (OuterVolumeSpecName: "web-config") pod "981d6a0f-d0e1-491e-803c-e6c81b0e0a40" (UID: "981d6a0f-d0e1-491e-803c-e6c81b0e0a40"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 04:17:46.186187 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.186160    2566 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:17:46.186187 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.186184    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:17:46.186330 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.186194    2566 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-configmap-metrics-client-ca\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:17:46.186330 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.186203    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-prometheus-trusted-ca-bundle\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:17:46.186330 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.186211    2566 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:17:46.186330 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.186220    2566 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-secret-kube-rbac-proxy\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:17:46.186330 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.186230    2566 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-secret-grpc-tls\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:17:46.186330 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.186239    2566 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-secret-metrics-client-certs\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:17:46.186330 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.186247    2566 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-config\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:17:46.186330 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.186256    2566 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:17:46.186330 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.186265    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-prometheus-k8s-db\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:17:46.186330 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.186273    2566 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:17:46.186330 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.186281    2566 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-tls-assets\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:17:46.186330 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.186290    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:17:46.186330 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.186297    2566 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:17:46.186330 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.186305    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8w4c\" (UniqueName: \"kubernetes.io/projected/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-kube-api-access-k8w4c\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:17:46.186330 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.186313    2566 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-config-out\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:17:46.186330 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.186321    2566 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:17:46.186330 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.186329    2566 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-web-config\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:17:46.186330 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.186337    2566 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/981d6a0f-d0e1-491e-803c-e6c81b0e0a40-secret-prometheus-k8s-tls\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:17:46.436091 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.435996    2566 generic.go:356] "Generic (PLEG): container finished" podID="981d6a0f-d0e1-491e-803c-e6c81b0e0a40" containerID="d5e65baccc4e3374550788db2407d911a0016b90c794153433c8eda4b324de04" exitCode=0
Jul 28 04:17:46.436091 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.436027    2566 generic.go:356] "Generic (PLEG): container finished" podID="981d6a0f-d0e1-491e-803c-e6c81b0e0a40" containerID="3de83ec883b6c1be0d49cd644963cc573bb6695d513f8ce18fbceca519429433" exitCode=0
Jul 28 04:17:46.436091 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.436036    2566 generic.go:356] "Generic (PLEG): container finished" podID="981d6a0f-d0e1-491e-803c-e6c81b0e0a40" containerID="90e9428ac89fed20cb9e237f23eb39faa71f5327fc5dc06d4f29b0022511b4fb" exitCode=0
Jul 28 04:17:46.436091 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.436045    2566 generic.go:356] "Generic (PLEG): container finished" podID="981d6a0f-d0e1-491e-803c-e6c81b0e0a40" containerID="e8f713475790f4f893142e7e142ae100604b444d6ca27090bd46ed08253c8f96" exitCode=0
Jul 28 04:17:46.436091 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.436053    2566 generic.go:356] "Generic (PLEG): container finished" podID="981d6a0f-d0e1-491e-803c-e6c81b0e0a40" containerID="fe966e374a546305195a0039383a543bf32d46ebd9c69bad9ff734888c263c4d" exitCode=0
Jul 28 04:17:46.436091 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.436060    2566 generic.go:356] "Generic (PLEG): container finished" podID="981d6a0f-d0e1-491e-803c-e6c81b0e0a40" containerID="9cd004fd3e50129ed6f8b124333db1431ab0b936b7d04629d7894a9c6e4a87ec" exitCode=0
Jul 28 04:17:46.436091 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.436063    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.436091 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.436084    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"981d6a0f-d0e1-491e-803c-e6c81b0e0a40","Type":"ContainerDied","Data":"d5e65baccc4e3374550788db2407d911a0016b90c794153433c8eda4b324de04"}
Jul 28 04:17:46.436564 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.436160    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"981d6a0f-d0e1-491e-803c-e6c81b0e0a40","Type":"ContainerDied","Data":"3de83ec883b6c1be0d49cd644963cc573bb6695d513f8ce18fbceca519429433"}
Jul 28 04:17:46.436564 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.436199    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"981d6a0f-d0e1-491e-803c-e6c81b0e0a40","Type":"ContainerDied","Data":"90e9428ac89fed20cb9e237f23eb39faa71f5327fc5dc06d4f29b0022511b4fb"}
Jul 28 04:17:46.436564 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.436232    2566 scope.go:117] "RemoveContainer" containerID="d5e65baccc4e3374550788db2407d911a0016b90c794153433c8eda4b324de04"
Jul 28 04:17:46.436564 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.436240    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"981d6a0f-d0e1-491e-803c-e6c81b0e0a40","Type":"ContainerDied","Data":"e8f713475790f4f893142e7e142ae100604b444d6ca27090bd46ed08253c8f96"}
Jul 28 04:17:46.436564 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.436279    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"981d6a0f-d0e1-491e-803c-e6c81b0e0a40","Type":"ContainerDied","Data":"fe966e374a546305195a0039383a543bf32d46ebd9c69bad9ff734888c263c4d"}
Jul 28 04:17:46.436564 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.436323    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"981d6a0f-d0e1-491e-803c-e6c81b0e0a40","Type":"ContainerDied","Data":"9cd004fd3e50129ed6f8b124333db1431ab0b936b7d04629d7894a9c6e4a87ec"}
Jul 28 04:17:46.436564 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.436368    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"981d6a0f-d0e1-491e-803c-e6c81b0e0a40","Type":"ContainerDied","Data":"e8d8816deb3b0807ce9db8ca069ec4014fd117e0959f70385e90997647a11535"}
Jul 28 04:17:46.444832 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.444813    2566 scope.go:117] "RemoveContainer" containerID="3de83ec883b6c1be0d49cd644963cc573bb6695d513f8ce18fbceca519429433"
Jul 28 04:17:46.453663 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.453645    2566 scope.go:117] "RemoveContainer" containerID="90e9428ac89fed20cb9e237f23eb39faa71f5327fc5dc06d4f29b0022511b4fb"
Jul 28 04:17:46.459873 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.459850    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 04:17:46.460711 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.460696    2566 scope.go:117] "RemoveContainer" containerID="e8f713475790f4f893142e7e142ae100604b444d6ca27090bd46ed08253c8f96"
Jul 28 04:17:46.465917 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.465897    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 04:17:46.467650 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.467637    2566 scope.go:117] "RemoveContainer" containerID="fe966e374a546305195a0039383a543bf32d46ebd9c69bad9ff734888c263c4d"
Jul 28 04:17:46.474664 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.474642    2566 scope.go:117] "RemoveContainer" containerID="9cd004fd3e50129ed6f8b124333db1431ab0b936b7d04629d7894a9c6e4a87ec"
Jul 28 04:17:46.481803 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.481785    2566 scope.go:117] "RemoveContainer" containerID="57072d126e9f2f6421f153ed376900b6f1ef209ce951c824f82ed5b34a06c6dd"
Jul 28 04:17:46.489354 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.489335    2566 scope.go:117] "RemoveContainer" containerID="d5e65baccc4e3374550788db2407d911a0016b90c794153433c8eda4b324de04"
Jul 28 04:17:46.489619 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:17:46.489601    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5e65baccc4e3374550788db2407d911a0016b90c794153433c8eda4b324de04\": container with ID starting with d5e65baccc4e3374550788db2407d911a0016b90c794153433c8eda4b324de04 not found: ID does not exist" containerID="d5e65baccc4e3374550788db2407d911a0016b90c794153433c8eda4b324de04"
Jul 28 04:17:46.489716 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.489630    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5e65baccc4e3374550788db2407d911a0016b90c794153433c8eda4b324de04"} err="failed to get container status \"d5e65baccc4e3374550788db2407d911a0016b90c794153433c8eda4b324de04\": rpc error: code = NotFound desc = could not find container \"d5e65baccc4e3374550788db2407d911a0016b90c794153433c8eda4b324de04\": container with ID starting with d5e65baccc4e3374550788db2407d911a0016b90c794153433c8eda4b324de04 not found: ID does not exist"
Jul 28 04:17:46.489716 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.489653    2566 scope.go:117] "RemoveContainer" containerID="3de83ec883b6c1be0d49cd644963cc573bb6695d513f8ce18fbceca519429433"
Jul 28 04:17:46.490265 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:17:46.490234    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3de83ec883b6c1be0d49cd644963cc573bb6695d513f8ce18fbceca519429433\": container with ID starting with 3de83ec883b6c1be0d49cd644963cc573bb6695d513f8ce18fbceca519429433 not found: ID does not exist" containerID="3de83ec883b6c1be0d49cd644963cc573bb6695d513f8ce18fbceca519429433"
Jul 28 04:17:46.490336 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.490276    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3de83ec883b6c1be0d49cd644963cc573bb6695d513f8ce18fbceca519429433"} err="failed to get container status \"3de83ec883b6c1be0d49cd644963cc573bb6695d513f8ce18fbceca519429433\": rpc error: code = NotFound desc = could not find container \"3de83ec883b6c1be0d49cd644963cc573bb6695d513f8ce18fbceca519429433\": container with ID starting with 3de83ec883b6c1be0d49cd644963cc573bb6695d513f8ce18fbceca519429433 not found: ID does not exist"
Jul 28 04:17:46.490336 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.490300    2566 scope.go:117] "RemoveContainer" containerID="90e9428ac89fed20cb9e237f23eb39faa71f5327fc5dc06d4f29b0022511b4fb"
Jul 28 04:17:46.490582 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:17:46.490561    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90e9428ac89fed20cb9e237f23eb39faa71f5327fc5dc06d4f29b0022511b4fb\": container with ID starting with 90e9428ac89fed20cb9e237f23eb39faa71f5327fc5dc06d4f29b0022511b4fb not found: ID does not exist" containerID="90e9428ac89fed20cb9e237f23eb39faa71f5327fc5dc06d4f29b0022511b4fb"
Jul 28 04:17:46.490633 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.490588    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90e9428ac89fed20cb9e237f23eb39faa71f5327fc5dc06d4f29b0022511b4fb"} err="failed to get container status \"90e9428ac89fed20cb9e237f23eb39faa71f5327fc5dc06d4f29b0022511b4fb\": rpc error: code = NotFound desc = could not find container \"90e9428ac89fed20cb9e237f23eb39faa71f5327fc5dc06d4f29b0022511b4fb\": container with ID starting with 90e9428ac89fed20cb9e237f23eb39faa71f5327fc5dc06d4f29b0022511b4fb not found: ID does not exist"
Jul 28 04:17:46.490633 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.490605    2566 scope.go:117] "RemoveContainer" containerID="e8f713475790f4f893142e7e142ae100604b444d6ca27090bd46ed08253c8f96"
Jul 28 04:17:46.490888 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:17:46.490864    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8f713475790f4f893142e7e142ae100604b444d6ca27090bd46ed08253c8f96\": container with ID starting with e8f713475790f4f893142e7e142ae100604b444d6ca27090bd46ed08253c8f96 not found: ID does not exist" containerID="e8f713475790f4f893142e7e142ae100604b444d6ca27090bd46ed08253c8f96"
Jul 28 04:17:46.490929 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.490900    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8f713475790f4f893142e7e142ae100604b444d6ca27090bd46ed08253c8f96"} err="failed to get container status \"e8f713475790f4f893142e7e142ae100604b444d6ca27090bd46ed08253c8f96\": rpc error: code = NotFound desc = could not find container \"e8f713475790f4f893142e7e142ae100604b444d6ca27090bd46ed08253c8f96\": container with ID starting with e8f713475790f4f893142e7e142ae100604b444d6ca27090bd46ed08253c8f96 not found: ID does not exist"
Jul 28 04:17:46.490929 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.490924    2566 scope.go:117] "RemoveContainer" containerID="fe966e374a546305195a0039383a543bf32d46ebd9c69bad9ff734888c263c4d"
Jul 28 04:17:46.491203 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:17:46.491185    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe966e374a546305195a0039383a543bf32d46ebd9c69bad9ff734888c263c4d\": container with ID starting with fe966e374a546305195a0039383a543bf32d46ebd9c69bad9ff734888c263c4d not found: ID does not exist" containerID="fe966e374a546305195a0039383a543bf32d46ebd9c69bad9ff734888c263c4d"
Jul 28 04:17:46.491254 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.491209    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe966e374a546305195a0039383a543bf32d46ebd9c69bad9ff734888c263c4d"} err="failed to get container status \"fe966e374a546305195a0039383a543bf32d46ebd9c69bad9ff734888c263c4d\": rpc error: code = NotFound desc = could not find container \"fe966e374a546305195a0039383a543bf32d46ebd9c69bad9ff734888c263c4d\": container with ID starting with fe966e374a546305195a0039383a543bf32d46ebd9c69bad9ff734888c263c4d not found: ID does not exist"
Jul 28 04:17:46.491254 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.491229    2566 scope.go:117] "RemoveContainer" containerID="9cd004fd3e50129ed6f8b124333db1431ab0b936b7d04629d7894a9c6e4a87ec"
Jul 28 04:17:46.491488 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:17:46.491466    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cd004fd3e50129ed6f8b124333db1431ab0b936b7d04629d7894a9c6e4a87ec\": container with ID starting with 9cd004fd3e50129ed6f8b124333db1431ab0b936b7d04629d7894a9c6e4a87ec not found: ID does not exist" containerID="9cd004fd3e50129ed6f8b124333db1431ab0b936b7d04629d7894a9c6e4a87ec"
Jul 28 04:17:46.491568 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.491488    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 04:17:46.491568 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.491491    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cd004fd3e50129ed6f8b124333db1431ab0b936b7d04629d7894a9c6e4a87ec"} err="failed to get container status \"9cd004fd3e50129ed6f8b124333db1431ab0b936b7d04629d7894a9c6e4a87ec\": rpc error: code = NotFound desc = could not find container \"9cd004fd3e50129ed6f8b124333db1431ab0b936b7d04629d7894a9c6e4a87ec\": container with ID starting with 9cd004fd3e50129ed6f8b124333db1431ab0b936b7d04629d7894a9c6e4a87ec not found: ID does not exist"
Jul 28 04:17:46.491568 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.491507    2566 scope.go:117] "RemoveContainer" containerID="57072d126e9f2f6421f153ed376900b6f1ef209ce951c824f82ed5b34a06c6dd"
Jul 28 04:17:46.491735 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:17:46.491715    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57072d126e9f2f6421f153ed376900b6f1ef209ce951c824f82ed5b34a06c6dd\": container with ID starting with 57072d126e9f2f6421f153ed376900b6f1ef209ce951c824f82ed5b34a06c6dd not found: ID does not exist" containerID="57072d126e9f2f6421f153ed376900b6f1ef209ce951c824f82ed5b34a06c6dd"
Jul 28 04:17:46.491773 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.491743    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57072d126e9f2f6421f153ed376900b6f1ef209ce951c824f82ed5b34a06c6dd"} err="failed to get container status \"57072d126e9f2f6421f153ed376900b6f1ef209ce951c824f82ed5b34a06c6dd\": rpc error: code = NotFound desc = could not find container \"57072d126e9f2f6421f153ed376900b6f1ef209ce951c824f82ed5b34a06c6dd\": container with ID starting with 57072d126e9f2f6421f153ed376900b6f1ef209ce951c824f82ed5b34a06c6dd not found: ID does not exist"
Jul 28 04:17:46.491773 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.491764    2566 scope.go:117] "RemoveContainer" containerID="d5e65baccc4e3374550788db2407d911a0016b90c794153433c8eda4b324de04"
Jul 28 04:17:46.491864 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.491805    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ec31cf3-eec3-4c57-96e6-fe88637a67c3" containerName="console"
Jul 28 04:17:46.491864 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.491818    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ec31cf3-eec3-4c57-96e6-fe88637a67c3" containerName="console"
Jul 28 04:17:46.491957 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.491907    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55cd6e46-995c-4857-b7b7-c90e4e4ff93d" containerName="registry"
Jul 28 04:17:46.491957 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.491919    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="55cd6e46-995c-4857-b7b7-c90e4e4ff93d" containerName="registry"
Jul 28 04:17:46.491957 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.491945    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="981d6a0f-d0e1-491e-803c-e6c81b0e0a40" containerName="init-config-reloader"
Jul 28 04:17:46.491957 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.491952    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="981d6a0f-d0e1-491e-803c-e6c81b0e0a40" containerName="init-config-reloader"
Jul 28 04:17:46.492066 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.491966    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="981d6a0f-d0e1-491e-803c-e6c81b0e0a40" containerName="thanos-sidecar"
Jul 28 04:17:46.492066 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.491975    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="981d6a0f-d0e1-491e-803c-e6c81b0e0a40" containerName="thanos-sidecar"
Jul 28 04:17:46.492066 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.491995    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="981d6a0f-d0e1-491e-803c-e6c81b0e0a40" containerName="prometheus"
Jul 28 04:17:46.492066 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.492003    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="981d6a0f-d0e1-491e-803c-e6c81b0e0a40" containerName="prometheus"
Jul 28 04:17:46.492066 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.491999    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5e65baccc4e3374550788db2407d911a0016b90c794153433c8eda4b324de04"} err="failed to get container status \"d5e65baccc4e3374550788db2407d911a0016b90c794153433c8eda4b324de04\": rpc error: code = NotFound desc = could not find container \"d5e65baccc4e3374550788db2407d911a0016b90c794153433c8eda4b324de04\": container with ID starting with d5e65baccc4e3374550788db2407d911a0016b90c794153433c8eda4b324de04 not found: ID does not exist"
Jul 28 04:17:46.492066 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.492022    2566 scope.go:117] "RemoveContainer" containerID="3de83ec883b6c1be0d49cd644963cc573bb6695d513f8ce18fbceca519429433"
Jul 28 04:17:46.492066 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.492009    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="981d6a0f-d0e1-491e-803c-e6c81b0e0a40" containerName="config-reloader"
Jul 28 04:17:46.492359 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.492072    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="981d6a0f-d0e1-491e-803c-e6c81b0e0a40" containerName="config-reloader"
Jul 28 04:17:46.492359 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.492086    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="981d6a0f-d0e1-491e-803c-e6c81b0e0a40" containerName="kube-rbac-proxy-web"
Jul 28 04:17:46.492359 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.492092    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="981d6a0f-d0e1-491e-803c-e6c81b0e0a40" containerName="kube-rbac-proxy-web"
Jul 28 04:17:46.492359 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.492098    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="981d6a0f-d0e1-491e-803c-e6c81b0e0a40" containerName="kube-rbac-proxy"
Jul 28 04:17:46.492359 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.492104    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="981d6a0f-d0e1-491e-803c-e6c81b0e0a40" containerName="kube-rbac-proxy"
Jul 28 04:17:46.492359 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.492113    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="981d6a0f-d0e1-491e-803c-e6c81b0e0a40" containerName="kube-rbac-proxy-thanos"
Jul 28 04:17:46.492359 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.492122    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="981d6a0f-d0e1-491e-803c-e6c81b0e0a40" containerName="kube-rbac-proxy-thanos"
Jul 28 04:17:46.492359 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.492207    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="981d6a0f-d0e1-491e-803c-e6c81b0e0a40" containerName="kube-rbac-proxy"
Jul 28 04:17:46.492359 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.492218    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="981d6a0f-d0e1-491e-803c-e6c81b0e0a40" containerName="kube-rbac-proxy-web"
Jul 28 04:17:46.492359 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.492225    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="981d6a0f-d0e1-491e-803c-e6c81b0e0a40" containerName="config-reloader"
Jul 28 04:17:46.492359 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.492231    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="981d6a0f-d0e1-491e-803c-e6c81b0e0a40" containerName="kube-rbac-proxy-thanos"
Jul 28 04:17:46.492359 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.492238    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="981d6a0f-d0e1-491e-803c-e6c81b0e0a40" containerName="thanos-sidecar"
Jul 28 04:17:46.492359 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.492247    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ec31cf3-eec3-4c57-96e6-fe88637a67c3" containerName="console"
Jul 28 04:17:46.492359 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.492255    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="55cd6e46-995c-4857-b7b7-c90e4e4ff93d" containerName="registry"
Jul 28 04:17:46.492359 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.492260    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="981d6a0f-d0e1-491e-803c-e6c81b0e0a40" containerName="prometheus"
Jul 28 04:17:46.492359 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.492331    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3de83ec883b6c1be0d49cd644963cc573bb6695d513f8ce18fbceca519429433"} err="failed to get container status \"3de83ec883b6c1be0d49cd644963cc573bb6695d513f8ce18fbceca519429433\": rpc error: code = NotFound desc = could not find container \"3de83ec883b6c1be0d49cd644963cc573bb6695d513f8ce18fbceca519429433\": container with ID starting with 3de83ec883b6c1be0d49cd644963cc573bb6695d513f8ce18fbceca519429433 not found: ID does not exist"
Jul 28 04:17:46.492359 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.492368    2566 scope.go:117] "RemoveContainer" containerID="90e9428ac89fed20cb9e237f23eb39faa71f5327fc5dc06d4f29b0022511b4fb"
Jul 28 04:17:46.493138 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.492631    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90e9428ac89fed20cb9e237f23eb39faa71f5327fc5dc06d4f29b0022511b4fb"} err="failed to get container status \"90e9428ac89fed20cb9e237f23eb39faa71f5327fc5dc06d4f29b0022511b4fb\": rpc error: code = NotFound desc = could not find container \"90e9428ac89fed20cb9e237f23eb39faa71f5327fc5dc06d4f29b0022511b4fb\": container with ID starting with 90e9428ac89fed20cb9e237f23eb39faa71f5327fc5dc06d4f29b0022511b4fb not found: ID does not exist"
Jul 28 04:17:46.493138 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.492650    2566 scope.go:117] "RemoveContainer" containerID="e8f713475790f4f893142e7e142ae100604b444d6ca27090bd46ed08253c8f96"
Jul 28 04:17:46.493138 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.492936    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8f713475790f4f893142e7e142ae100604b444d6ca27090bd46ed08253c8f96"} err="failed to get container status \"e8f713475790f4f893142e7e142ae100604b444d6ca27090bd46ed08253c8f96\": rpc error: code = NotFound desc = could not find container \"e8f713475790f4f893142e7e142ae100604b444d6ca27090bd46ed08253c8f96\": container with ID starting with e8f713475790f4f893142e7e142ae100604b444d6ca27090bd46ed08253c8f96 not found: ID does not exist"
Jul 28 04:17:46.493138 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.492959    2566 scope.go:117] "RemoveContainer" containerID="fe966e374a546305195a0039383a543bf32d46ebd9c69bad9ff734888c263c4d"
Jul 28 04:17:46.493636 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.493608    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe966e374a546305195a0039383a543bf32d46ebd9c69bad9ff734888c263c4d"} err="failed to get container status \"fe966e374a546305195a0039383a543bf32d46ebd9c69bad9ff734888c263c4d\": rpc error: code = NotFound desc = could not find container \"fe966e374a546305195a0039383a543bf32d46ebd9c69bad9ff734888c263c4d\": container with ID starting with fe966e374a546305195a0039383a543bf32d46ebd9c69bad9ff734888c263c4d not found: ID does not exist"
Jul 28 04:17:46.493704 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.493640    2566 scope.go:117] "RemoveContainer" containerID="9cd004fd3e50129ed6f8b124333db1431ab0b936b7d04629d7894a9c6e4a87ec"
Jul 28 04:17:46.493917 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.493892    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cd004fd3e50129ed6f8b124333db1431ab0b936b7d04629d7894a9c6e4a87ec"} err="failed to get container status \"9cd004fd3e50129ed6f8b124333db1431ab0b936b7d04629d7894a9c6e4a87ec\": rpc error: code = NotFound desc = could not find container \"9cd004fd3e50129ed6f8b124333db1431ab0b936b7d04629d7894a9c6e4a87ec\": container with ID starting with 9cd004fd3e50129ed6f8b124333db1431ab0b936b7d04629d7894a9c6e4a87ec not found: ID does not exist"
Jul 28 04:17:46.493988 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.493918    2566 scope.go:117] "RemoveContainer" containerID="57072d126e9f2f6421f153ed376900b6f1ef209ce951c824f82ed5b34a06c6dd"
Jul 28 04:17:46.494166 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.494149    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57072d126e9f2f6421f153ed376900b6f1ef209ce951c824f82ed5b34a06c6dd"} err="failed to get container status \"57072d126e9f2f6421f153ed376900b6f1ef209ce951c824f82ed5b34a06c6dd\": rpc error: code = NotFound desc = could not find container \"57072d126e9f2f6421f153ed376900b6f1ef209ce951c824f82ed5b34a06c6dd\": container with ID starting with 57072d126e9f2f6421f153ed376900b6f1ef209ce951c824f82ed5b34a06c6dd not found: ID does not exist"
Jul 28 04:17:46.494228 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.494166    2566 scope.go:117] "RemoveContainer" containerID="d5e65baccc4e3374550788db2407d911a0016b90c794153433c8eda4b324de04"
Jul 28 04:17:46.494397 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.494378    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5e65baccc4e3374550788db2407d911a0016b90c794153433c8eda4b324de04"} err="failed to get container status \"d5e65baccc4e3374550788db2407d911a0016b90c794153433c8eda4b324de04\": rpc error: code = NotFound desc = could not find container \"d5e65baccc4e3374550788db2407d911a0016b90c794153433c8eda4b324de04\": container with ID starting with d5e65baccc4e3374550788db2407d911a0016b90c794153433c8eda4b324de04 not found: ID does not exist"
Jul 28 04:17:46.494438 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.494398    2566 scope.go:117] "RemoveContainer" containerID="3de83ec883b6c1be0d49cd644963cc573bb6695d513f8ce18fbceca519429433"
Jul 28 04:17:46.494660 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.494646    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3de83ec883b6c1be0d49cd644963cc573bb6695d513f8ce18fbceca519429433"} err="failed to get container status \"3de83ec883b6c1be0d49cd644963cc573bb6695d513f8ce18fbceca519429433\": rpc error: code = NotFound desc = could not find container \"3de83ec883b6c1be0d49cd644963cc573bb6695d513f8ce18fbceca519429433\": container with ID starting with 3de83ec883b6c1be0d49cd644963cc573bb6695d513f8ce18fbceca519429433 not found: ID does not exist"
Jul 28 04:17:46.494660 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.494660    2566 scope.go:117] "RemoveContainer" containerID="90e9428ac89fed20cb9e237f23eb39faa71f5327fc5dc06d4f29b0022511b4fb"
Jul 28 04:17:46.494909 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.494894    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90e9428ac89fed20cb9e237f23eb39faa71f5327fc5dc06d4f29b0022511b4fb"} err="failed to get container status \"90e9428ac89fed20cb9e237f23eb39faa71f5327fc5dc06d4f29b0022511b4fb\": rpc error: code = NotFound desc = could not find container \"90e9428ac89fed20cb9e237f23eb39faa71f5327fc5dc06d4f29b0022511b4fb\": container with ID starting with 90e9428ac89fed20cb9e237f23eb39faa71f5327fc5dc06d4f29b0022511b4fb not found: ID does not exist"
Jul 28 04:17:46.494958 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.494910    2566 scope.go:117] "RemoveContainer" containerID="e8f713475790f4f893142e7e142ae100604b444d6ca27090bd46ed08253c8f96"
Jul 28 04:17:46.495126 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.495101    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8f713475790f4f893142e7e142ae100604b444d6ca27090bd46ed08253c8f96"} err="failed to get container status \"e8f713475790f4f893142e7e142ae100604b444d6ca27090bd46ed08253c8f96\": rpc error: code = NotFound desc = could not find container \"e8f713475790f4f893142e7e142ae100604b444d6ca27090bd46ed08253c8f96\": container with ID starting with e8f713475790f4f893142e7e142ae100604b444d6ca27090bd46ed08253c8f96 not found: ID does not exist"
Jul 28 04:17:46.495126 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.495121    2566 scope.go:117] "RemoveContainer" containerID="fe966e374a546305195a0039383a543bf32d46ebd9c69bad9ff734888c263c4d"
Jul 28 04:17:46.495348 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.495332    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe966e374a546305195a0039383a543bf32d46ebd9c69bad9ff734888c263c4d"} err="failed to get container status \"fe966e374a546305195a0039383a543bf32d46ebd9c69bad9ff734888c263c4d\": rpc error: code = NotFound desc = could not find container \"fe966e374a546305195a0039383a543bf32d46ebd9c69bad9ff734888c263c4d\": container with ID starting with fe966e374a546305195a0039383a543bf32d46ebd9c69bad9ff734888c263c4d not found: ID does not exist"
Jul 28 04:17:46.495408 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.495350    2566 scope.go:117] "RemoveContainer" containerID="9cd004fd3e50129ed6f8b124333db1431ab0b936b7d04629d7894a9c6e4a87ec"
Jul 28 04:17:46.495561 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.495544    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cd004fd3e50129ed6f8b124333db1431ab0b936b7d04629d7894a9c6e4a87ec"} err="failed to get container status \"9cd004fd3e50129ed6f8b124333db1431ab0b936b7d04629d7894a9c6e4a87ec\": rpc error: code = NotFound desc = could not find container \"9cd004fd3e50129ed6f8b124333db1431ab0b936b7d04629d7894a9c6e4a87ec\": container with ID starting with 9cd004fd3e50129ed6f8b124333db1431ab0b936b7d04629d7894a9c6e4a87ec not found: ID does not exist"
Jul 28 04:17:46.495612 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.495561    2566 scope.go:117] "RemoveContainer" containerID="57072d126e9f2f6421f153ed376900b6f1ef209ce951c824f82ed5b34a06c6dd"
Jul 28 04:17:46.495819 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.495795    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57072d126e9f2f6421f153ed376900b6f1ef209ce951c824f82ed5b34a06c6dd"} err="failed to get container status \"57072d126e9f2f6421f153ed376900b6f1ef209ce951c824f82ed5b34a06c6dd\": rpc error: code = NotFound desc = could not find container \"57072d126e9f2f6421f153ed376900b6f1ef209ce951c824f82ed5b34a06c6dd\": container with ID starting with 57072d126e9f2f6421f153ed376900b6f1ef209ce951c824f82ed5b34a06c6dd not found: ID does not exist"
Jul 28 04:17:46.495904 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.495821    2566 scope.go:117] "RemoveContainer" containerID="d5e65baccc4e3374550788db2407d911a0016b90c794153433c8eda4b324de04"
Jul 28 04:17:46.496077 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.496055    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5e65baccc4e3374550788db2407d911a0016b90c794153433c8eda4b324de04"} err="failed to get container status \"d5e65baccc4e3374550788db2407d911a0016b90c794153433c8eda4b324de04\": rpc error: code = NotFound desc = could not find container \"d5e65baccc4e3374550788db2407d911a0016b90c794153433c8eda4b324de04\": container with ID starting with d5e65baccc4e3374550788db2407d911a0016b90c794153433c8eda4b324de04 not found: ID does not exist"
Jul 28 04:17:46.496077 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.496077    2566 scope.go:117] "RemoveContainer" containerID="3de83ec883b6c1be0d49cd644963cc573bb6695d513f8ce18fbceca519429433"
Jul 28 04:17:46.496204 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.496112    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.496589 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.496310    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3de83ec883b6c1be0d49cd644963cc573bb6695d513f8ce18fbceca519429433"} err="failed to get container status \"3de83ec883b6c1be0d49cd644963cc573bb6695d513f8ce18fbceca519429433\": rpc error: code = NotFound desc = could not find container \"3de83ec883b6c1be0d49cd644963cc573bb6695d513f8ce18fbceca519429433\": container with ID starting with 3de83ec883b6c1be0d49cd644963cc573bb6695d513f8ce18fbceca519429433 not found: ID does not exist"
Jul 28 04:17:46.496589 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.496333    2566 scope.go:117] "RemoveContainer" containerID="90e9428ac89fed20cb9e237f23eb39faa71f5327fc5dc06d4f29b0022511b4fb"
Jul 28 04:17:46.496863 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.496818    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90e9428ac89fed20cb9e237f23eb39faa71f5327fc5dc06d4f29b0022511b4fb"} err="failed to get container status \"90e9428ac89fed20cb9e237f23eb39faa71f5327fc5dc06d4f29b0022511b4fb\": rpc error: code = NotFound desc = could not find container \"90e9428ac89fed20cb9e237f23eb39faa71f5327fc5dc06d4f29b0022511b4fb\": container with ID starting with 90e9428ac89fed20cb9e237f23eb39faa71f5327fc5dc06d4f29b0022511b4fb not found: ID does not exist"
Jul 28 04:17:46.496945 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.496867    2566 scope.go:117] "RemoveContainer" containerID="e8f713475790f4f893142e7e142ae100604b444d6ca27090bd46ed08253c8f96"
Jul 28 04:17:46.497182 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.497152    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8f713475790f4f893142e7e142ae100604b444d6ca27090bd46ed08253c8f96"} err="failed to get container status \"e8f713475790f4f893142e7e142ae100604b444d6ca27090bd46ed08253c8f96\": rpc error: code = NotFound desc = could not find container \"e8f713475790f4f893142e7e142ae100604b444d6ca27090bd46ed08253c8f96\": container with ID starting with e8f713475790f4f893142e7e142ae100604b444d6ca27090bd46ed08253c8f96 not found: ID does not exist"
Jul 28 04:17:46.497182 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.497184    2566 scope.go:117] "RemoveContainer" containerID="fe966e374a546305195a0039383a543bf32d46ebd9c69bad9ff734888c263c4d"
Jul 28 04:17:46.497486 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.497460    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe966e374a546305195a0039383a543bf32d46ebd9c69bad9ff734888c263c4d"} err="failed to get container status \"fe966e374a546305195a0039383a543bf32d46ebd9c69bad9ff734888c263c4d\": rpc error: code = NotFound desc = could not find container \"fe966e374a546305195a0039383a543bf32d46ebd9c69bad9ff734888c263c4d\": container with ID starting with fe966e374a546305195a0039383a543bf32d46ebd9c69bad9ff734888c263c4d not found: ID does not exist"
Jul 28 04:17:46.497546 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.497488    2566 scope.go:117] "RemoveContainer" containerID="9cd004fd3e50129ed6f8b124333db1431ab0b936b7d04629d7894a9c6e4a87ec"
Jul 28 04:17:46.497738 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.497715    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cd004fd3e50129ed6f8b124333db1431ab0b936b7d04629d7894a9c6e4a87ec"} err="failed to get container status \"9cd004fd3e50129ed6f8b124333db1431ab0b936b7d04629d7894a9c6e4a87ec\": rpc error: code = NotFound desc = could not find container \"9cd004fd3e50129ed6f8b124333db1431ab0b936b7d04629d7894a9c6e4a87ec\": container with ID starting with 9cd004fd3e50129ed6f8b124333db1431ab0b936b7d04629d7894a9c6e4a87ec not found: ID does not exist"
Jul 28 04:17:46.497738 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.497737    2566 scope.go:117] "RemoveContainer" containerID="57072d126e9f2f6421f153ed376900b6f1ef209ce951c824f82ed5b34a06c6dd"
Jul 28 04:17:46.498034 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.498005    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57072d126e9f2f6421f153ed376900b6f1ef209ce951c824f82ed5b34a06c6dd"} err="failed to get container status \"57072d126e9f2f6421f153ed376900b6f1ef209ce951c824f82ed5b34a06c6dd\": rpc error: code = NotFound desc = could not find container \"57072d126e9f2f6421f153ed376900b6f1ef209ce951c824f82ed5b34a06c6dd\": container with ID starting with 57072d126e9f2f6421f153ed376900b6f1ef209ce951c824f82ed5b34a06c6dd not found: ID does not exist"
Jul 28 04:17:46.498034 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.498033    2566 scope.go:117] "RemoveContainer" containerID="d5e65baccc4e3374550788db2407d911a0016b90c794153433c8eda4b324de04"
Jul 28 04:17:46.498315 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.498291    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5e65baccc4e3374550788db2407d911a0016b90c794153433c8eda4b324de04"} err="failed to get container status \"d5e65baccc4e3374550788db2407d911a0016b90c794153433c8eda4b324de04\": rpc error: code = NotFound desc = could not find container \"d5e65baccc4e3374550788db2407d911a0016b90c794153433c8eda4b324de04\": container with ID starting with d5e65baccc4e3374550788db2407d911a0016b90c794153433c8eda4b324de04 not found: ID does not exist"
Jul 28 04:17:46.498377 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.498320    2566 scope.go:117] "RemoveContainer" containerID="3de83ec883b6c1be0d49cd644963cc573bb6695d513f8ce18fbceca519429433"
Jul 28 04:17:46.498630 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.498611    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3de83ec883b6c1be0d49cd644963cc573bb6695d513f8ce18fbceca519429433"} err="failed to get container status \"3de83ec883b6c1be0d49cd644963cc573bb6695d513f8ce18fbceca519429433\": rpc error: code = NotFound desc = could not find container \"3de83ec883b6c1be0d49cd644963cc573bb6695d513f8ce18fbceca519429433\": container with ID starting with 3de83ec883b6c1be0d49cd644963cc573bb6695d513f8ce18fbceca519429433 not found: ID does not exist"
Jul 28 04:17:46.498630 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.498630    2566 scope.go:117] "RemoveContainer" containerID="90e9428ac89fed20cb9e237f23eb39faa71f5327fc5dc06d4f29b0022511b4fb"
Jul 28 04:17:46.498762 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.498672    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 28 04:17:46.498933 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.498913    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90e9428ac89fed20cb9e237f23eb39faa71f5327fc5dc06d4f29b0022511b4fb"} err="failed to get container status \"90e9428ac89fed20cb9e237f23eb39faa71f5327fc5dc06d4f29b0022511b4fb\": rpc error: code = NotFound desc = could not find container \"90e9428ac89fed20cb9e237f23eb39faa71f5327fc5dc06d4f29b0022511b4fb\": container with ID starting with 90e9428ac89fed20cb9e237f23eb39faa71f5327fc5dc06d4f29b0022511b4fb not found: ID does not exist"
Jul 28 04:17:46.499003 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.498934    2566 scope.go:117] "RemoveContainer" containerID="e8f713475790f4f893142e7e142ae100604b444d6ca27090bd46ed08253c8f96"
Jul 28 04:17:46.499055 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.499001    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 28 04:17:46.499055 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.499041    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 28 04:17:46.499204 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.499074    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 28 04:17:46.499204 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.499085    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 28 04:17:46.499204 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.499108    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dmmeekb5lj9gh\""
Jul 28 04:17:46.500072 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.499486    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 28 04:17:46.500072 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.499661    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8f713475790f4f893142e7e142ae100604b444d6ca27090bd46ed08253c8f96"} err="failed to get container status \"e8f713475790f4f893142e7e142ae100604b444d6ca27090bd46ed08253c8f96\": rpc error: code = NotFound desc = could not find container \"e8f713475790f4f893142e7e142ae100604b444d6ca27090bd46ed08253c8f96\": container with ID starting with e8f713475790f4f893142e7e142ae100604b444d6ca27090bd46ed08253c8f96 not found: ID does not exist"
Jul 28 04:17:46.500072 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.499691    2566 scope.go:117] "RemoveContainer" containerID="fe966e374a546305195a0039383a543bf32d46ebd9c69bad9ff734888c263c4d"
Jul 28 04:17:46.500072 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.499881    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 28 04:17:46.500072 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.500025    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-mvl8j\""
Jul 28 04:17:46.500349 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.500141    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 28 04:17:46.503027 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.500709    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 28 04:17:46.503027 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.501328    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 28 04:17:46.503027 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.502190    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe966e374a546305195a0039383a543bf32d46ebd9c69bad9ff734888c263c4d"} err="failed to get container status \"fe966e374a546305195a0039383a543bf32d46ebd9c69bad9ff734888c263c4d\": rpc error: code = NotFound desc = could not find container \"fe966e374a546305195a0039383a543bf32d46ebd9c69bad9ff734888c263c4d\": container with ID starting with fe966e374a546305195a0039383a543bf32d46ebd9c69bad9ff734888c263c4d not found: ID does not exist"
Jul 28 04:17:46.503027 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.502217    2566 scope.go:117] "RemoveContainer" containerID="9cd004fd3e50129ed6f8b124333db1431ab0b936b7d04629d7894a9c6e4a87ec"
Jul 28 04:17:46.503027 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.502240    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 28 04:17:46.503027 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.502599    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cd004fd3e50129ed6f8b124333db1431ab0b936b7d04629d7894a9c6e4a87ec"} err="failed to get container status \"9cd004fd3e50129ed6f8b124333db1431ab0b936b7d04629d7894a9c6e4a87ec\": rpc error: code = NotFound desc = could not find container \"9cd004fd3e50129ed6f8b124333db1431ab0b936b7d04629d7894a9c6e4a87ec\": container with ID starting with 9cd004fd3e50129ed6f8b124333db1431ab0b936b7d04629d7894a9c6e4a87ec not found: ID does not exist"
Jul 28 04:17:46.503027 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.502625    2566 scope.go:117] "RemoveContainer" containerID="57072d126e9f2f6421f153ed376900b6f1ef209ce951c824f82ed5b34a06c6dd"
Jul 28 04:17:46.503027 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.502895    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 28 04:17:46.504904 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.504575    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57072d126e9f2f6421f153ed376900b6f1ef209ce951c824f82ed5b34a06c6dd"} err="failed to get container status \"57072d126e9f2f6421f153ed376900b6f1ef209ce951c824f82ed5b34a06c6dd\": rpc error: code = NotFound desc = could not find container \"57072d126e9f2f6421f153ed376900b6f1ef209ce951c824f82ed5b34a06c6dd\": container with ID starting with 57072d126e9f2f6421f153ed376900b6f1ef209ce951c824f82ed5b34a06c6dd not found: ID does not exist"
Jul 28 04:17:46.504904 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.504603    2566 scope.go:117] "RemoveContainer" containerID="d5e65baccc4e3374550788db2407d911a0016b90c794153433c8eda4b324de04"
Jul 28 04:17:46.508107 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.506884    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 28 04:17:46.508754 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.508546    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5e65baccc4e3374550788db2407d911a0016b90c794153433c8eda4b324de04"} err="failed to get container status \"d5e65baccc4e3374550788db2407d911a0016b90c794153433c8eda4b324de04\": rpc error: code = NotFound desc = could not find container \"d5e65baccc4e3374550788db2407d911a0016b90c794153433c8eda4b324de04\": container with ID starting with d5e65baccc4e3374550788db2407d911a0016b90c794153433c8eda4b324de04 not found: ID does not exist"
Jul 28 04:17:46.508865 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.508763    2566 scope.go:117] "RemoveContainer" containerID="3de83ec883b6c1be0d49cd644963cc573bb6695d513f8ce18fbceca519429433"
Jul 28 04:17:46.509436 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.509316    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3de83ec883b6c1be0d49cd644963cc573bb6695d513f8ce18fbceca519429433"} err="failed to get container status \"3de83ec883b6c1be0d49cd644963cc573bb6695d513f8ce18fbceca519429433\": rpc error: code = NotFound desc = could not find container \"3de83ec883b6c1be0d49cd644963cc573bb6695d513f8ce18fbceca519429433\": container with ID starting with 3de83ec883b6c1be0d49cd644963cc573bb6695d513f8ce18fbceca519429433 not found: ID does not exist"
Jul 28 04:17:46.509436 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.509345    2566 scope.go:117] "RemoveContainer" containerID="90e9428ac89fed20cb9e237f23eb39faa71f5327fc5dc06d4f29b0022511b4fb"
Jul 28 04:17:46.509890 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.509865    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90e9428ac89fed20cb9e237f23eb39faa71f5327fc5dc06d4f29b0022511b4fb"} err="failed to get container status \"90e9428ac89fed20cb9e237f23eb39faa71f5327fc5dc06d4f29b0022511b4fb\": rpc error: code = NotFound desc = could not find container \"90e9428ac89fed20cb9e237f23eb39faa71f5327fc5dc06d4f29b0022511b4fb\": container with ID starting with 90e9428ac89fed20cb9e237f23eb39faa71f5327fc5dc06d4f29b0022511b4fb not found: ID does not exist"
Jul 28 04:17:46.510166 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.509891    2566 scope.go:117] "RemoveContainer" containerID="e8f713475790f4f893142e7e142ae100604b444d6ca27090bd46ed08253c8f96"
Jul 28 04:17:46.510705 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.510475    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8f713475790f4f893142e7e142ae100604b444d6ca27090bd46ed08253c8f96"} err="failed to get container status \"e8f713475790f4f893142e7e142ae100604b444d6ca27090bd46ed08253c8f96\": rpc error: code = NotFound desc = could not find container \"e8f713475790f4f893142e7e142ae100604b444d6ca27090bd46ed08253c8f96\": container with ID starting with e8f713475790f4f893142e7e142ae100604b444d6ca27090bd46ed08253c8f96 not found: ID does not exist"
Jul 28 04:17:46.510705 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.510510    2566 scope.go:117] "RemoveContainer" containerID="fe966e374a546305195a0039383a543bf32d46ebd9c69bad9ff734888c263c4d"
Jul 28 04:17:46.511251 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.511094    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 28 04:17:46.511251 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.511133    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe966e374a546305195a0039383a543bf32d46ebd9c69bad9ff734888c263c4d"} err="failed to get container status \"fe966e374a546305195a0039383a543bf32d46ebd9c69bad9ff734888c263c4d\": rpc error: code = NotFound desc = could not find container \"fe966e374a546305195a0039383a543bf32d46ebd9c69bad9ff734888c263c4d\": container with ID starting with fe966e374a546305195a0039383a543bf32d46ebd9c69bad9ff734888c263c4d not found: ID does not exist"
Jul 28 04:17:46.511251 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.511158    2566 scope.go:117] "RemoveContainer" containerID="9cd004fd3e50129ed6f8b124333db1431ab0b936b7d04629d7894a9c6e4a87ec"
Jul 28 04:17:46.514173 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.514049    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cd004fd3e50129ed6f8b124333db1431ab0b936b7d04629d7894a9c6e4a87ec"} err="failed to get container status \"9cd004fd3e50129ed6f8b124333db1431ab0b936b7d04629d7894a9c6e4a87ec\": rpc error: code = NotFound desc = could not find container \"9cd004fd3e50129ed6f8b124333db1431ab0b936b7d04629d7894a9c6e4a87ec\": container with ID starting with 9cd004fd3e50129ed6f8b124333db1431ab0b936b7d04629d7894a9c6e4a87ec not found: ID does not exist"
Jul 28 04:17:46.514173 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.514095    2566 scope.go:117] "RemoveContainer" containerID="57072d126e9f2f6421f153ed376900b6f1ef209ce951c824f82ed5b34a06c6dd"
Jul 28 04:17:46.514173 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.514145    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 04:17:46.514487 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.514415    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57072d126e9f2f6421f153ed376900b6f1ef209ce951c824f82ed5b34a06c6dd"} err="failed to get container status \"57072d126e9f2f6421f153ed376900b6f1ef209ce951c824f82ed5b34a06c6dd\": rpc error: code = NotFound desc = could not find container \"57072d126e9f2f6421f153ed376900b6f1ef209ce951c824f82ed5b34a06c6dd\": container with ID starting with 57072d126e9f2f6421f153ed376900b6f1ef209ce951c824f82ed5b34a06c6dd not found: ID does not exist"
Jul 28 04:17:46.589523 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.589497    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/834f4106-f9fe-43f0-99e6-79ed625abab3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.589620 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.589532    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/834f4106-f9fe-43f0-99e6-79ed625abab3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.589620 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.589548    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/834f4106-f9fe-43f0-99e6-79ed625abab3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.589620 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.589565    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/834f4106-f9fe-43f0-99e6-79ed625abab3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.589620 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.589594    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/834f4106-f9fe-43f0-99e6-79ed625abab3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.589757 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.589640    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/834f4106-f9fe-43f0-99e6-79ed625abab3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.589757 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.589659    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/834f4106-f9fe-43f0-99e6-79ed625abab3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.589757 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.589684    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/834f4106-f9fe-43f0-99e6-79ed625abab3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.589757 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.589700    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs5pl\" (UniqueName: \"kubernetes.io/projected/834f4106-f9fe-43f0-99e6-79ed625abab3-kube-api-access-vs5pl\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.589757 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.589730    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/834f4106-f9fe-43f0-99e6-79ed625abab3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.589757 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.589754    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/834f4106-f9fe-43f0-99e6-79ed625abab3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.589972 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.589778    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/834f4106-f9fe-43f0-99e6-79ed625abab3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.589972 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.589800    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/834f4106-f9fe-43f0-99e6-79ed625abab3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.589972 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.589819    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/834f4106-f9fe-43f0-99e6-79ed625abab3-config\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.589972 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.589832    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/834f4106-f9fe-43f0-99e6-79ed625abab3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.589972 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.589873    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/834f4106-f9fe-43f0-99e6-79ed625abab3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.589972 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.589891    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/834f4106-f9fe-43f0-99e6-79ed625abab3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.589972 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.589906    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/834f4106-f9fe-43f0-99e6-79ed625abab3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.589972 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.589960    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/834f4106-f9fe-43f0-99e6-79ed625abab3-config-out\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.590218 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.590010    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/834f4106-f9fe-43f0-99e6-79ed625abab3-web-config\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.690828 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.690748    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs5pl\" (UniqueName: \"kubernetes.io/projected/834f4106-f9fe-43f0-99e6-79ed625abab3-kube-api-access-vs5pl\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.690828 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.690788    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/834f4106-f9fe-43f0-99e6-79ed625abab3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.690828 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.690808    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/834f4106-f9fe-43f0-99e6-79ed625abab3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.690828 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.690832    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/834f4106-f9fe-43f0-99e6-79ed625abab3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.691145 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.690864    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/834f4106-f9fe-43f0-99e6-79ed625abab3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.691145 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.690889    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/834f4106-f9fe-43f0-99e6-79ed625abab3-config\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.691145 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.690911    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/834f4106-f9fe-43f0-99e6-79ed625abab3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.691145 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.690933    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/834f4106-f9fe-43f0-99e6-79ed625abab3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.691145 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.690958    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/834f4106-f9fe-43f0-99e6-79ed625abab3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.691145 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.690981    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/834f4106-f9fe-43f0-99e6-79ed625abab3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.691145 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.691022    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/834f4106-f9fe-43f0-99e6-79ed625abab3-config-out\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.691145 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.691054    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/834f4106-f9fe-43f0-99e6-79ed625abab3-web-config\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.691145 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.691121    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/834f4106-f9fe-43f0-99e6-79ed625abab3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.691613 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.691164    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/834f4106-f9fe-43f0-99e6-79ed625abab3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.691613 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.691192    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/834f4106-f9fe-43f0-99e6-79ed625abab3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.691613 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.691218    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/834f4106-f9fe-43f0-99e6-79ed625abab3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.691613 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.691243    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/834f4106-f9fe-43f0-99e6-79ed625abab3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.691613 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.691278    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/834f4106-f9fe-43f0-99e6-79ed625abab3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.691613 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.691310    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/834f4106-f9fe-43f0-99e6-79ed625abab3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.691613 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.691340    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/834f4106-f9fe-43f0-99e6-79ed625abab3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.691988 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.691622    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/834f4106-f9fe-43f0-99e6-79ed625abab3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.693330 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.692280    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/834f4106-f9fe-43f0-99e6-79ed625abab3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.693330 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.692377    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/834f4106-f9fe-43f0-99e6-79ed625abab3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.693330 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.692997    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/834f4106-f9fe-43f0-99e6-79ed625abab3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.694310 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.693953    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/834f4106-f9fe-43f0-99e6-79ed625abab3-config\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.694310 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.693952    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/834f4106-f9fe-43f0-99e6-79ed625abab3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.694310 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.694042    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/834f4106-f9fe-43f0-99e6-79ed625abab3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.694310 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.694116    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/834f4106-f9fe-43f0-99e6-79ed625abab3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.694310 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.694148    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/834f4106-f9fe-43f0-99e6-79ed625abab3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.694675 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.694387    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/834f4106-f9fe-43f0-99e6-79ed625abab3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.695081 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.695059    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/834f4106-f9fe-43f0-99e6-79ed625abab3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.695205 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.695163    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/834f4106-f9fe-43f0-99e6-79ed625abab3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.695479 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.695460    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/834f4106-f9fe-43f0-99e6-79ed625abab3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.695658 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.695635    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/834f4106-f9fe-43f0-99e6-79ed625abab3-config-out\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.695750 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.695725    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/834f4106-f9fe-43f0-99e6-79ed625abab3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.696469 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.696448    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/834f4106-f9fe-43f0-99e6-79ed625abab3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.696690 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.696670    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/834f4106-f9fe-43f0-99e6-79ed625abab3-web-config\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.696730 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.696720    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/834f4106-f9fe-43f0-99e6-79ed625abab3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.696790 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.696774    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/834f4106-f9fe-43f0-99e6-79ed625abab3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.701902 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.701881    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs5pl\" (UniqueName: \"kubernetes.io/projected/834f4106-f9fe-43f0-99e6-79ed625abab3-kube-api-access-vs5pl\") pod \"prometheus-k8s-0\" (UID: \"834f4106-f9fe-43f0-99e6-79ed625abab3\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.817705 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.817682    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:46.951054 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:46.950801    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 04:17:46.953325 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:17:46.953300    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod834f4106_f9fe_43f0_99e6_79ed625abab3.slice/crio-0d13250ae09b231c962eaa5b8a9c4ab96c4fae325e2ac7f68f444a93fc79f49f WatchSource:0}: Error finding container 0d13250ae09b231c962eaa5b8a9c4ab96c4fae325e2ac7f68f444a93fc79f49f: Status 404 returned error can't find the container with id 0d13250ae09b231c962eaa5b8a9c4ab96c4fae325e2ac7f68f444a93fc79f49f
Jul 28 04:17:47.441572 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:47.441542    2566 generic.go:356] "Generic (PLEG): container finished" podID="834f4106-f9fe-43f0-99e6-79ed625abab3" containerID="87d02cd1ad6fc28d152bb3612f2ac665d7980dc397706da3876497de748996d2" exitCode=0
Jul 28 04:17:47.441733 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:47.441622    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"834f4106-f9fe-43f0-99e6-79ed625abab3","Type":"ContainerDied","Data":"87d02cd1ad6fc28d152bb3612f2ac665d7980dc397706da3876497de748996d2"}
Jul 28 04:17:47.441733 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:47.441692    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"834f4106-f9fe-43f0-99e6-79ed625abab3","Type":"ContainerStarted","Data":"0d13250ae09b231c962eaa5b8a9c4ab96c4fae325e2ac7f68f444a93fc79f49f"}
Jul 28 04:17:48.087625 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:48.087590    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="981d6a0f-d0e1-491e-803c-e6c81b0e0a40" path="/var/lib/kubelet/pods/981d6a0f-d0e1-491e-803c-e6c81b0e0a40/volumes"
Jul 28 04:17:48.449386 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:48.449290    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"834f4106-f9fe-43f0-99e6-79ed625abab3","Type":"ContainerStarted","Data":"a4599fdf0dcfc9a51d61e55dd261255c73eb1aab9847e32f4b4e4d68e55cbbba"}
Jul 28 04:17:48.449386 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:48.449356    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"834f4106-f9fe-43f0-99e6-79ed625abab3","Type":"ContainerStarted","Data":"de5bd09a9c57b5265dcd488e765c92048fdc3091a6b64001d9a80934437026d9"}
Jul 28 04:17:48.449593 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:48.449391    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"834f4106-f9fe-43f0-99e6-79ed625abab3","Type":"ContainerStarted","Data":"779126437a48538abdc5ad5861593248be5b721a40793dbc18e8d361c0863014"}
Jul 28 04:17:48.449593 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:48.449419    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"834f4106-f9fe-43f0-99e6-79ed625abab3","Type":"ContainerStarted","Data":"c8d134655412067bd6f090e2b2add20d1473156e4fe81d8e745cb20065abd2cc"}
Jul 28 04:17:48.449593 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:48.449442    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"834f4106-f9fe-43f0-99e6-79ed625abab3","Type":"ContainerStarted","Data":"47d442020f856b9a01c4a6f71e99cae0b75b4ee905e21149fbf3d2849d270dab"}
Jul 28 04:17:48.449593 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:48.449477    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"834f4106-f9fe-43f0-99e6-79ed625abab3","Type":"ContainerStarted","Data":"ddb53b4e9d396b0636bbe2d44e5f38800b49ff1cfc827bf27e2b52e700fb37dd"}
Jul 28 04:17:48.477257 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:48.477207    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.457579675 podStartE2EDuration="2.477192332s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.612657ms" isStatefulPod=false podCreationTimestamp="2026-07-28 04:17:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:17:48.474973502 +0000 UTC m=+368.865149523" watchObservedRunningTime="2026-07-28 04:17:48.477192332 +0000 UTC m=+368.867368352"
Jul 28 04:17:51.818248 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:51.818213    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:17:54.051970 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:54.051924    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6107883-62a6-4a82-9e04-ed46057a6b0e-metrics-certs\") pod \"network-metrics-daemon-l6p44\" (UID: \"e6107883-62a6-4a82-9e04-ed46057a6b0e\") " pod="openshift-multus/network-metrics-daemon-l6p44"
Jul 28 04:17:54.054313 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:54.054279    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6107883-62a6-4a82-9e04-ed46057a6b0e-metrics-certs\") pod \"network-metrics-daemon-l6p44\" (UID: \"e6107883-62a6-4a82-9e04-ed46057a6b0e\") " pod="openshift-multus/network-metrics-daemon-l6p44"
Jul 28 04:17:54.186251 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:54.186229    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x5zc2\""
Jul 28 04:17:54.194222 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:54.194203    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l6p44"
Jul 28 04:17:54.319799 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:54.319774    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l6p44"]
Jul 28 04:17:54.321905 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:17:54.321880    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6107883_62a6_4a82_9e04_ed46057a6b0e.slice/crio-91b93419e9d9de3cef751ffead5caf9a863cbaa6f3dc101abce4808adbde6028 WatchSource:0}: Error finding container 91b93419e9d9de3cef751ffead5caf9a863cbaa6f3dc101abce4808adbde6028: Status 404 returned error can't find the container with id 91b93419e9d9de3cef751ffead5caf9a863cbaa6f3dc101abce4808adbde6028
Jul 28 04:17:54.474321 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:54.474289    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l6p44" event={"ID":"e6107883-62a6-4a82-9e04-ed46057a6b0e","Type":"ContainerStarted","Data":"91b93419e9d9de3cef751ffead5caf9a863cbaa6f3dc101abce4808adbde6028"}
Jul 28 04:17:55.480184 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:55.480093    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l6p44" event={"ID":"e6107883-62a6-4a82-9e04-ed46057a6b0e","Type":"ContainerStarted","Data":"72db1b2deb555d9deae5fed95a7326df7292c33328d5afd34b7ac5e5ec3cdec2"}
Jul 28 04:17:55.480184 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:55.480156    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l6p44" event={"ID":"e6107883-62a6-4a82-9e04-ed46057a6b0e","Type":"ContainerStarted","Data":"6730ead528aa634e86f47b4ac6ab27970aced2789217b67318bee7d05507e4fa"}
Jul 28 04:17:55.500857 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:17:55.496485    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-l6p44" podStartSLOduration=373.695285711 podStartE2EDuration="6m14.496463729s" totalImagesPullingTime="801.178018ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:11:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:17:55.494381386 +0000 UTC m=+375.884557407" watchObservedRunningTime="2026-07-28 04:17:55.496463729 +0000 UTC m=+375.886639751"
Jul 28 04:18:46.818893 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:18:46.818831    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:18:46.834552 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:18:46.834530    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:18:47.689383 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:18:47.689358    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 04:21:22.242259 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:22.242225    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-69d65f9dd8-lgq4p"]
Jul 28 04:21:22.245660 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:22.245644    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-69d65f9dd8-lgq4p"
Jul 28 04:21:22.248308 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:22.248284    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 28 04:21:22.248308 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:22.248299    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 28 04:21:22.248508 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:22.248340    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 28 04:21:22.248508 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:22.248341    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 28 04:21:22.248508 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:22.248401    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-9c6v4\""
Jul 28 04:21:22.263257 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:22.263231    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-69d65f9dd8-lgq4p"]
Jul 28 04:21:22.343108 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:22.343077    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r624t\" (UniqueName: \"kubernetes.io/projected/3eb1967e-94c7-450b-82d7-076c52b46af1-kube-api-access-r624t\") pod \"opendatahub-operator-controller-manager-69d65f9dd8-lgq4p\" (UID: \"3eb1967e-94c7-450b-82d7-076c52b46af1\") " pod="opendatahub/opendatahub-operator-controller-manager-69d65f9dd8-lgq4p"
Jul 28 04:21:22.343254 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:22.343117    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3eb1967e-94c7-450b-82d7-076c52b46af1-tmp\") pod \"opendatahub-operator-controller-manager-69d65f9dd8-lgq4p\" (UID: \"3eb1967e-94c7-450b-82d7-076c52b46af1\") " pod="opendatahub/opendatahub-operator-controller-manager-69d65f9dd8-lgq4p"
Jul 28 04:21:22.343254 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:22.343146    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3eb1967e-94c7-450b-82d7-076c52b46af1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69d65f9dd8-lgq4p\" (UID: \"3eb1967e-94c7-450b-82d7-076c52b46af1\") " pod="opendatahub/opendatahub-operator-controller-manager-69d65f9dd8-lgq4p"
Jul 28 04:21:22.343254 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:22.343188    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3eb1967e-94c7-450b-82d7-076c52b46af1-manifests\") pod \"opendatahub-operator-controller-manager-69d65f9dd8-lgq4p\" (UID: \"3eb1967e-94c7-450b-82d7-076c52b46af1\") " pod="opendatahub/opendatahub-operator-controller-manager-69d65f9dd8-lgq4p"
Jul 28 04:21:22.343254 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:22.343204    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3eb1967e-94c7-450b-82d7-076c52b46af1-webhook-cert\") pod \"opendatahub-operator-controller-manager-69d65f9dd8-lgq4p\" (UID: \"3eb1967e-94c7-450b-82d7-076c52b46af1\") " pod="opendatahub/opendatahub-operator-controller-manager-69d65f9dd8-lgq4p"
Jul 28 04:21:22.444039 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:22.444010    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3eb1967e-94c7-450b-82d7-076c52b46af1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69d65f9dd8-lgq4p\" (UID: \"3eb1967e-94c7-450b-82d7-076c52b46af1\") " pod="opendatahub/opendatahub-operator-controller-manager-69d65f9dd8-lgq4p"
Jul 28 04:21:22.444198 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:22.444050    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3eb1967e-94c7-450b-82d7-076c52b46af1-manifests\") pod \"opendatahub-operator-controller-manager-69d65f9dd8-lgq4p\" (UID: \"3eb1967e-94c7-450b-82d7-076c52b46af1\") " pod="opendatahub/opendatahub-operator-controller-manager-69d65f9dd8-lgq4p"
Jul 28 04:21:22.444198 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:22.444066    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3eb1967e-94c7-450b-82d7-076c52b46af1-webhook-cert\") pod \"opendatahub-operator-controller-manager-69d65f9dd8-lgq4p\" (UID: \"3eb1967e-94c7-450b-82d7-076c52b46af1\") " pod="opendatahub/opendatahub-operator-controller-manager-69d65f9dd8-lgq4p"
Jul 28 04:21:22.444198 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:22.444106    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r624t\" (UniqueName: \"kubernetes.io/projected/3eb1967e-94c7-450b-82d7-076c52b46af1-kube-api-access-r624t\") pod \"opendatahub-operator-controller-manager-69d65f9dd8-lgq4p\" (UID: \"3eb1967e-94c7-450b-82d7-076c52b46af1\") " pod="opendatahub/opendatahub-operator-controller-manager-69d65f9dd8-lgq4p"
Jul 28 04:21:22.444350 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:22.444228    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3eb1967e-94c7-450b-82d7-076c52b46af1-tmp\") pod \"opendatahub-operator-controller-manager-69d65f9dd8-lgq4p\" (UID: \"3eb1967e-94c7-450b-82d7-076c52b46af1\") " pod="opendatahub/opendatahub-operator-controller-manager-69d65f9dd8-lgq4p"
Jul 28 04:21:22.444526 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:22.444506    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3eb1967e-94c7-450b-82d7-076c52b46af1-manifests\") pod \"opendatahub-operator-controller-manager-69d65f9dd8-lgq4p\" (UID: \"3eb1967e-94c7-450b-82d7-076c52b46af1\") " pod="opendatahub/opendatahub-operator-controller-manager-69d65f9dd8-lgq4p"
Jul 28 04:21:22.446486 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:22.446461    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3eb1967e-94c7-450b-82d7-076c52b46af1-tmp\") pod \"opendatahub-operator-controller-manager-69d65f9dd8-lgq4p\" (UID: \"3eb1967e-94c7-450b-82d7-076c52b46af1\") " pod="opendatahub/opendatahub-operator-controller-manager-69d65f9dd8-lgq4p"
Jul 28 04:21:22.446601 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:22.446583    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3eb1967e-94c7-450b-82d7-076c52b46af1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69d65f9dd8-lgq4p\" (UID: \"3eb1967e-94c7-450b-82d7-076c52b46af1\") " pod="opendatahub/opendatahub-operator-controller-manager-69d65f9dd8-lgq4p"
Jul 28 04:21:22.446649 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:22.446619    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3eb1967e-94c7-450b-82d7-076c52b46af1-webhook-cert\") pod \"opendatahub-operator-controller-manager-69d65f9dd8-lgq4p\" (UID: \"3eb1967e-94c7-450b-82d7-076c52b46af1\") " pod="opendatahub/opendatahub-operator-controller-manager-69d65f9dd8-lgq4p"
Jul 28 04:21:22.455072 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:22.455050    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r624t\" (UniqueName: \"kubernetes.io/projected/3eb1967e-94c7-450b-82d7-076c52b46af1-kube-api-access-r624t\") pod \"opendatahub-operator-controller-manager-69d65f9dd8-lgq4p\" (UID: \"3eb1967e-94c7-450b-82d7-076c52b46af1\") " pod="opendatahub/opendatahub-operator-controller-manager-69d65f9dd8-lgq4p"
Jul 28 04:21:22.555937 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:22.555913    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-69d65f9dd8-lgq4p"
Jul 28 04:21:22.689142 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:22.689101    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-69d65f9dd8-lgq4p"]
Jul 28 04:21:22.691526 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:21:22.691503    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eb1967e_94c7_450b_82d7_076c52b46af1.slice/crio-33eec767bf96fd5d2a074023ad6ea88dc2c8635e616e283883834fa8a8ba2a3c WatchSource:0}: Error finding container 33eec767bf96fd5d2a074023ad6ea88dc2c8635e616e283883834fa8a8ba2a3c: Status 404 returned error can't find the container with id 33eec767bf96fd5d2a074023ad6ea88dc2c8635e616e283883834fa8a8ba2a3c
Jul 28 04:21:23.275972 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:23.275901    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-69d65f9dd8-lgq4p" event={"ID":"3eb1967e-94c7-450b-82d7-076c52b46af1","Type":"ContainerStarted","Data":"33eec767bf96fd5d2a074023ad6ea88dc2c8635e616e283883834fa8a8ba2a3c"}
Jul 28 04:21:26.292311 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:26.292270    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-69d65f9dd8-lgq4p" event={"ID":"3eb1967e-94c7-450b-82d7-076c52b46af1","Type":"ContainerStarted","Data":"65caaa865f07453b26ef6d7864deea0170e601b257bc97c97ce44118de167029"}
Jul 28 04:21:27.304514 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:27.304479    2566 generic.go:356] "Generic (PLEG): container finished" podID="3eb1967e-94c7-450b-82d7-076c52b46af1" containerID="65caaa865f07453b26ef6d7864deea0170e601b257bc97c97ce44118de167029" exitCode=0
Jul 28 04:21:27.304888 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:27.304542    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-69d65f9dd8-lgq4p" event={"ID":"3eb1967e-94c7-450b-82d7-076c52b46af1","Type":"ContainerDied","Data":"65caaa865f07453b26ef6d7864deea0170e601b257bc97c97ce44118de167029"}
Jul 28 04:21:28.310317 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:28.310281    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-69d65f9dd8-lgq4p" event={"ID":"3eb1967e-94c7-450b-82d7-076c52b46af1","Type":"ContainerStarted","Data":"11fdd48e87455e8c9d820e2e66c6a5bec852b6b4eba0f31719120b681f36a678"}
Jul 28 04:21:28.310783 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:28.310402    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-69d65f9dd8-lgq4p"
Jul 28 04:21:28.333573 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:28.333526    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-69d65f9dd8-lgq4p" podStartSLOduration=2.0425384 podStartE2EDuration="6.333511668s" totalImagesPullingTime="3.274077469s" totalInitContainerRuntime="1.016895799s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:21:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:21:28.331228076 +0000 UTC m=+588.721404097" watchObservedRunningTime="2026-07-28 04:21:28.333511668 +0000 UTC m=+588.723687688"
Jul 28 04:21:39.316583 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:39.316541    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-69d65f9dd8-lgq4p"
Jul 28 04:21:40.679730 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:40.679700    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7796f8d956-tv25r"]
Jul 28 04:21:40.685537 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:40.685517    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7796f8d956-tv25r"
Jul 28 04:21:40.688662 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:40.688642    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 28 04:21:40.689738 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:40.689716    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 28 04:21:40.689887 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:40.689767    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-f4c7r\""
Jul 28 04:21:40.689887 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:40.689781    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 28 04:21:40.689887 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:40.689828    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 28 04:21:40.689887 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:40.689886    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 28 04:21:40.691037 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:40.691017    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ktcn\" (UniqueName: \"kubernetes.io/projected/3abde1d4-6845-4a1e-8ec4-3746b5c7d5fe-kube-api-access-6ktcn\") pod \"lws-controller-manager-7796f8d956-tv25r\" (UID: \"3abde1d4-6845-4a1e-8ec4-3746b5c7d5fe\") " pod="openshift-lws-operator/lws-controller-manager-7796f8d956-tv25r"
Jul 28 04:21:40.691150 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:40.691133    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3abde1d4-6845-4a1e-8ec4-3746b5c7d5fe-manager-config\") pod \"lws-controller-manager-7796f8d956-tv25r\" (UID: \"3abde1d4-6845-4a1e-8ec4-3746b5c7d5fe\") " pod="openshift-lws-operator/lws-controller-manager-7796f8d956-tv25r"
Jul 28 04:21:40.691194 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:40.691180    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3abde1d4-6845-4a1e-8ec4-3746b5c7d5fe-cert\") pod \"lws-controller-manager-7796f8d956-tv25r\" (UID: \"3abde1d4-6845-4a1e-8ec4-3746b5c7d5fe\") " pod="openshift-lws-operator/lws-controller-manager-7796f8d956-tv25r"
Jul 28 04:21:40.691229 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:40.691206    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3abde1d4-6845-4a1e-8ec4-3746b5c7d5fe-metrics-cert\") pod \"lws-controller-manager-7796f8d956-tv25r\" (UID: \"3abde1d4-6845-4a1e-8ec4-3746b5c7d5fe\") " pod="openshift-lws-operator/lws-controller-manager-7796f8d956-tv25r"
Jul 28 04:21:40.694424 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:40.694404    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7796f8d956-tv25r"]
Jul 28 04:21:40.792082 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:40.792050    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3abde1d4-6845-4a1e-8ec4-3746b5c7d5fe-manager-config\") pod \"lws-controller-manager-7796f8d956-tv25r\" (UID: \"3abde1d4-6845-4a1e-8ec4-3746b5c7d5fe\") " pod="openshift-lws-operator/lws-controller-manager-7796f8d956-tv25r"
Jul 28 04:21:40.792263 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:40.792088    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3abde1d4-6845-4a1e-8ec4-3746b5c7d5fe-cert\") pod \"lws-controller-manager-7796f8d956-tv25r\" (UID: \"3abde1d4-6845-4a1e-8ec4-3746b5c7d5fe\") " pod="openshift-lws-operator/lws-controller-manager-7796f8d956-tv25r"
Jul 28 04:21:40.792263 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:40.792109    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3abde1d4-6845-4a1e-8ec4-3746b5c7d5fe-metrics-cert\") pod \"lws-controller-manager-7796f8d956-tv25r\" (UID: \"3abde1d4-6845-4a1e-8ec4-3746b5c7d5fe\") " pod="openshift-lws-operator/lws-controller-manager-7796f8d956-tv25r"
Jul 28 04:21:40.792263 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:40.792140    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ktcn\" (UniqueName: \"kubernetes.io/projected/3abde1d4-6845-4a1e-8ec4-3746b5c7d5fe-kube-api-access-6ktcn\") pod \"lws-controller-manager-7796f8d956-tv25r\" (UID: \"3abde1d4-6845-4a1e-8ec4-3746b5c7d5fe\") " pod="openshift-lws-operator/lws-controller-manager-7796f8d956-tv25r"
Jul 28 04:21:40.792718 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:40.792697    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/3abde1d4-6845-4a1e-8ec4-3746b5c7d5fe-manager-config\") pod \"lws-controller-manager-7796f8d956-tv25r\" (UID: \"3abde1d4-6845-4a1e-8ec4-3746b5c7d5fe\") " pod="openshift-lws-operator/lws-controller-manager-7796f8d956-tv25r"
Jul 28 04:21:40.794599 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:40.794580    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3abde1d4-6845-4a1e-8ec4-3746b5c7d5fe-cert\") pod \"lws-controller-manager-7796f8d956-tv25r\" (UID: \"3abde1d4-6845-4a1e-8ec4-3746b5c7d5fe\") " pod="openshift-lws-operator/lws-controller-manager-7796f8d956-tv25r"
Jul 28 04:21:40.794649 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:40.794601    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3abde1d4-6845-4a1e-8ec4-3746b5c7d5fe-metrics-cert\") pod \"lws-controller-manager-7796f8d956-tv25r\" (UID: \"3abde1d4-6845-4a1e-8ec4-3746b5c7d5fe\") " pod="openshift-lws-operator/lws-controller-manager-7796f8d956-tv25r"
Jul 28 04:21:40.809584 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:40.809554    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ktcn\" (UniqueName: \"kubernetes.io/projected/3abde1d4-6845-4a1e-8ec4-3746b5c7d5fe-kube-api-access-6ktcn\") pod \"lws-controller-manager-7796f8d956-tv25r\" (UID: \"3abde1d4-6845-4a1e-8ec4-3746b5c7d5fe\") " pod="openshift-lws-operator/lws-controller-manager-7796f8d956-tv25r"
Jul 28 04:21:40.995473 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:40.995386    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7796f8d956-tv25r"
Jul 28 04:21:41.139353 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:41.139329    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7796f8d956-tv25r"]
Jul 28 04:21:41.141003 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:21:41.140971    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3abde1d4_6845_4a1e_8ec4_3746b5c7d5fe.slice/crio-1ffd71248b05959c705eb579bd3f6ddcaa27d0e7914a3d25ddfb4c7e1f8d8b0e WatchSource:0}: Error finding container 1ffd71248b05959c705eb579bd3f6ddcaa27d0e7914a3d25ddfb4c7e1f8d8b0e: Status 404 returned error can't find the container with id 1ffd71248b05959c705eb579bd3f6ddcaa27d0e7914a3d25ddfb4c7e1f8d8b0e
Jul 28 04:21:41.363131 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:41.363095    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7796f8d956-tv25r" event={"ID":"3abde1d4-6845-4a1e-8ec4-3746b5c7d5fe","Type":"ContainerStarted","Data":"1ffd71248b05959c705eb579bd3f6ddcaa27d0e7914a3d25ddfb4c7e1f8d8b0e"}
Jul 28 04:21:43.544857 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:43.544814    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4glpr_bfd2d3d8-f287-42d8-83c7-f82185c96735/console-operator/2.log"
Jul 28 04:21:43.544857 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:43.544830    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4glpr_bfd2d3d8-f287-42d8-83c7-f82185c96735/console-operator/2.log"
Jul 28 04:21:44.379692 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:44.379653    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7796f8d956-tv25r" event={"ID":"3abde1d4-6845-4a1e-8ec4-3746b5c7d5fe","Type":"ContainerStarted","Data":"3bef3c24ba9e55e0c82fac7eefb14ead621e23ad479dbefc1fcb15f8e88c01c1"}
Jul 28 04:21:44.379906 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:44.379779    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7796f8d956-tv25r"
Jul 28 04:21:44.395097 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:44.395045    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7796f8d956-tv25r" podStartSLOduration=1.947498592 podStartE2EDuration="4.39502707s" totalImagesPullingTime="2.447528478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:21:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:21:44.393543035 +0000 UTC m=+604.783719055" watchObservedRunningTime="2026-07-28 04:21:44.39502707 +0000 UTC m=+604.785203094"
Jul 28 04:21:51.981038 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:51.981002    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-58d69b594d-jqh6r"]
Jul 28 04:21:51.984382 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:51.984366    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-58d69b594d-jqh6r"
Jul 28 04:21:51.987127 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:51.987101    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 28 04:21:51.987247 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:51.987110    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 28 04:21:51.987247 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:51.987110    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-c94xc\""
Jul 28 04:21:51.994749 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:51.994728    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-58d69b594d-jqh6r"]
Jul 28 04:21:52.082793 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:52.082761    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baf9fb3c-ce25-4d8c-a06d-ce4a4aa666c5-tls-certs\") pod \"kube-auth-proxy-58d69b594d-jqh6r\" (UID: \"baf9fb3c-ce25-4d8c-a06d-ce4a4aa666c5\") " pod="openshift-ingress/kube-auth-proxy-58d69b594d-jqh6r"
Jul 28 04:21:52.082959 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:52.082812    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86qnm\" (UniqueName: \"kubernetes.io/projected/baf9fb3c-ce25-4d8c-a06d-ce4a4aa666c5-kube-api-access-86qnm\") pod \"kube-auth-proxy-58d69b594d-jqh6r\" (UID: \"baf9fb3c-ce25-4d8c-a06d-ce4a4aa666c5\") " pod="openshift-ingress/kube-auth-proxy-58d69b594d-jqh6r"
Jul 28 04:21:52.082959 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:52.082892    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/baf9fb3c-ce25-4d8c-a06d-ce4a4aa666c5-tmp\") pod \"kube-auth-proxy-58d69b594d-jqh6r\" (UID: \"baf9fb3c-ce25-4d8c-a06d-ce4a4aa666c5\") " pod="openshift-ingress/kube-auth-proxy-58d69b594d-jqh6r"
Jul 28 04:21:52.184109 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:52.184070    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baf9fb3c-ce25-4d8c-a06d-ce4a4aa666c5-tls-certs\") pod \"kube-auth-proxy-58d69b594d-jqh6r\" (UID: \"baf9fb3c-ce25-4d8c-a06d-ce4a4aa666c5\") " pod="openshift-ingress/kube-auth-proxy-58d69b594d-jqh6r"
Jul 28 04:21:52.184287 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:52.184133    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86qnm\" (UniqueName: \"kubernetes.io/projected/baf9fb3c-ce25-4d8c-a06d-ce4a4aa666c5-kube-api-access-86qnm\") pod \"kube-auth-proxy-58d69b594d-jqh6r\" (UID: \"baf9fb3c-ce25-4d8c-a06d-ce4a4aa666c5\") " pod="openshift-ingress/kube-auth-proxy-58d69b594d-jqh6r"
Jul 28 04:21:52.184287 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:52.184192    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/baf9fb3c-ce25-4d8c-a06d-ce4a4aa666c5-tmp\") pod \"kube-auth-proxy-58d69b594d-jqh6r\" (UID: \"baf9fb3c-ce25-4d8c-a06d-ce4a4aa666c5\") " pod="openshift-ingress/kube-auth-proxy-58d69b594d-jqh6r"
Jul 28 04:21:52.184287 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:21:52.184212    2566 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 28 04:21:52.184287 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:21:52.184270    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/baf9fb3c-ce25-4d8c-a06d-ce4a4aa666c5-tls-certs podName:baf9fb3c-ce25-4d8c-a06d-ce4a4aa666c5 nodeName:}" failed. No retries permitted until 2026-07-28 04:21:52.684254574 +0000 UTC m=+613.074430573 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/baf9fb3c-ce25-4d8c-a06d-ce4a4aa666c5-tls-certs") pod "kube-auth-proxy-58d69b594d-jqh6r" (UID: "baf9fb3c-ce25-4d8c-a06d-ce4a4aa666c5") : secret "kube-auth-proxy-tls" not found
Jul 28 04:21:52.186515 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:52.186493    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/baf9fb3c-ce25-4d8c-a06d-ce4a4aa666c5-tmp\") pod \"kube-auth-proxy-58d69b594d-jqh6r\" (UID: \"baf9fb3c-ce25-4d8c-a06d-ce4a4aa666c5\") " pod="openshift-ingress/kube-auth-proxy-58d69b594d-jqh6r"
Jul 28 04:21:52.195728 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:52.195704    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86qnm\" (UniqueName: \"kubernetes.io/projected/baf9fb3c-ce25-4d8c-a06d-ce4a4aa666c5-kube-api-access-86qnm\") pod \"kube-auth-proxy-58d69b594d-jqh6r\" (UID: \"baf9fb3c-ce25-4d8c-a06d-ce4a4aa666c5\") " pod="openshift-ingress/kube-auth-proxy-58d69b594d-jqh6r"
Jul 28 04:21:52.688414 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:52.688380    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baf9fb3c-ce25-4d8c-a06d-ce4a4aa666c5-tls-certs\") pod \"kube-auth-proxy-58d69b594d-jqh6r\" (UID: \"baf9fb3c-ce25-4d8c-a06d-ce4a4aa666c5\") " pod="openshift-ingress/kube-auth-proxy-58d69b594d-jqh6r"
Jul 28 04:21:52.690754 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:52.690724    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/baf9fb3c-ce25-4d8c-a06d-ce4a4aa666c5-tls-certs\") pod \"kube-auth-proxy-58d69b594d-jqh6r\" (UID: \"baf9fb3c-ce25-4d8c-a06d-ce4a4aa666c5\") " pod="openshift-ingress/kube-auth-proxy-58d69b594d-jqh6r"
Jul 28 04:21:52.894099 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:52.894069    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-58d69b594d-jqh6r"
Jul 28 04:21:53.052519 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:53.052495    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-58d69b594d-jqh6r"]
Jul 28 04:21:53.054827 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:21:53.054802    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaf9fb3c_ce25_4d8c_a06d_ce4a4aa666c5.slice/crio-018ab87c7ed75be016463680e3a3a0652533caeca68384558fc67fb6f5127f93 WatchSource:0}: Error finding container 018ab87c7ed75be016463680e3a3a0652533caeca68384558fc67fb6f5127f93: Status 404 returned error can't find the container with id 018ab87c7ed75be016463680e3a3a0652533caeca68384558fc67fb6f5127f93
Jul 28 04:21:53.056567 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:53.056551    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 04:21:53.419509 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:53.419473    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-58d69b594d-jqh6r" event={"ID":"baf9fb3c-ce25-4d8c-a06d-ce4a4aa666c5","Type":"ContainerStarted","Data":"018ab87c7ed75be016463680e3a3a0652533caeca68384558fc67fb6f5127f93"}
Jul 28 04:21:55.386808 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:55.386772    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7796f8d956-tv25r"
Jul 28 04:21:56.434306 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:56.434270    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-58d69b594d-jqh6r" event={"ID":"baf9fb3c-ce25-4d8c-a06d-ce4a4aa666c5","Type":"ContainerStarted","Data":"138b5dd2a1ea93aa9531cc8266ad99c9b4fc010760c6ec2ad8fbcb35c8980ec9"}
Jul 28 04:21:56.452108 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:21:56.452063    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-58d69b594d-jqh6r" podStartSLOduration=2.7419521209999997 podStartE2EDuration="5.452049572s" totalImagesPullingTime="2.710097451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:21:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:21:56.450606054 +0000 UTC m=+616.840782076" watchObservedRunningTime="2026-07-28 04:21:56.452049572 +0000 UTC m=+616.842225593"
Jul 28 04:23:19.890214 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:19.890173    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7bd48fb7cf-wbkqp"]
Jul 28 04:23:19.893757 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:19.893719    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bd48fb7cf-wbkqp"
Jul 28 04:23:19.896635 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:19.896608    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 28 04:23:19.897720 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:19.897699    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7n5sw\""
Jul 28 04:23:19.897810 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:19.897754    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 28 04:23:19.897810 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:19.897766    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 28 04:23:19.897932 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:19.897704    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 28 04:23:19.898101 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:19.898084    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 28 04:23:19.902242 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:19.902219    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 28 04:23:19.904286 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:19.904262    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bd48fb7cf-wbkqp"]
Jul 28 04:23:20.000816 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:20.000786    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c0a0e084-e243-4fd1-84d0-f429ae783778-console-oauth-config\") pod \"console-7bd48fb7cf-wbkqp\" (UID: \"c0a0e084-e243-4fd1-84d0-f429ae783778\") " pod="openshift-console/console-7bd48fb7cf-wbkqp"
Jul 28 04:23:20.000974 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:20.000856    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c0a0e084-e243-4fd1-84d0-f429ae783778-console-config\") pod \"console-7bd48fb7cf-wbkqp\" (UID: \"c0a0e084-e243-4fd1-84d0-f429ae783778\") " pod="openshift-console/console-7bd48fb7cf-wbkqp"
Jul 28 04:23:20.000974 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:20.000874    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bs2j\" (UniqueName: \"kubernetes.io/projected/c0a0e084-e243-4fd1-84d0-f429ae783778-kube-api-access-5bs2j\") pod \"console-7bd48fb7cf-wbkqp\" (UID: \"c0a0e084-e243-4fd1-84d0-f429ae783778\") " pod="openshift-console/console-7bd48fb7cf-wbkqp"
Jul 28 04:23:20.000974 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:20.000893    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c0a0e084-e243-4fd1-84d0-f429ae783778-console-serving-cert\") pod \"console-7bd48fb7cf-wbkqp\" (UID: \"c0a0e084-e243-4fd1-84d0-f429ae783778\") " pod="openshift-console/console-7bd48fb7cf-wbkqp"
Jul 28 04:23:20.000974 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:20.000909    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c0a0e084-e243-4fd1-84d0-f429ae783778-service-ca\") pod \"console-7bd48fb7cf-wbkqp\" (UID: \"c0a0e084-e243-4fd1-84d0-f429ae783778\") " pod="openshift-console/console-7bd48fb7cf-wbkqp"
Jul 28 04:23:20.001108 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:20.000984    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c0a0e084-e243-4fd1-84d0-f429ae783778-oauth-serving-cert\") pod \"console-7bd48fb7cf-wbkqp\" (UID: \"c0a0e084-e243-4fd1-84d0-f429ae783778\") " pod="openshift-console/console-7bd48fb7cf-wbkqp"
Jul 28 04:23:20.001108 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:20.001031    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0a0e084-e243-4fd1-84d0-f429ae783778-trusted-ca-bundle\") pod \"console-7bd48fb7cf-wbkqp\" (UID: \"c0a0e084-e243-4fd1-84d0-f429ae783778\") " pod="openshift-console/console-7bd48fb7cf-wbkqp"
Jul 28 04:23:20.102136 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:20.102107    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c0a0e084-e243-4fd1-84d0-f429ae783778-oauth-serving-cert\") pod \"console-7bd48fb7cf-wbkqp\" (UID: \"c0a0e084-e243-4fd1-84d0-f429ae783778\") " pod="openshift-console/console-7bd48fb7cf-wbkqp"
Jul 28 04:23:20.102290 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:20.102143    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0a0e084-e243-4fd1-84d0-f429ae783778-trusted-ca-bundle\") pod \"console-7bd48fb7cf-wbkqp\" (UID: \"c0a0e084-e243-4fd1-84d0-f429ae783778\") " pod="openshift-console/console-7bd48fb7cf-wbkqp"
Jul 28 04:23:20.102290 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:20.102171    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c0a0e084-e243-4fd1-84d0-f429ae783778-console-oauth-config\") pod \"console-7bd48fb7cf-wbkqp\" (UID: \"c0a0e084-e243-4fd1-84d0-f429ae783778\") " pod="openshift-console/console-7bd48fb7cf-wbkqp"
Jul 28 04:23:20.102290 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:20.102216    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c0a0e084-e243-4fd1-84d0-f429ae783778-console-config\") pod \"console-7bd48fb7cf-wbkqp\" (UID: \"c0a0e084-e243-4fd1-84d0-f429ae783778\") " pod="openshift-console/console-7bd48fb7cf-wbkqp"
Jul 28 04:23:20.102290 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:20.102232    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bs2j\" (UniqueName: \"kubernetes.io/projected/c0a0e084-e243-4fd1-84d0-f429ae783778-kube-api-access-5bs2j\") pod \"console-7bd48fb7cf-wbkqp\" (UID: \"c0a0e084-e243-4fd1-84d0-f429ae783778\") " pod="openshift-console/console-7bd48fb7cf-wbkqp"
Jul 28 04:23:20.102290 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:20.102249    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c0a0e084-e243-4fd1-84d0-f429ae783778-console-serving-cert\") pod \"console-7bd48fb7cf-wbkqp\" (UID: \"c0a0e084-e243-4fd1-84d0-f429ae783778\") " pod="openshift-console/console-7bd48fb7cf-wbkqp"
Jul 28 04:23:20.102290 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:20.102270    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c0a0e084-e243-4fd1-84d0-f429ae783778-service-ca\") pod \"console-7bd48fb7cf-wbkqp\" (UID: \"c0a0e084-e243-4fd1-84d0-f429ae783778\") " pod="openshift-console/console-7bd48fb7cf-wbkqp"
Jul 28 04:23:20.103030 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:20.103005    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c0a0e084-e243-4fd1-84d0-f429ae783778-oauth-serving-cert\") pod \"console-7bd48fb7cf-wbkqp\" (UID: \"c0a0e084-e243-4fd1-84d0-f429ae783778\") " pod="openshift-console/console-7bd48fb7cf-wbkqp"
Jul 28 04:23:20.103171 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:20.103147    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c0a0e084-e243-4fd1-84d0-f429ae783778-console-config\") pod \"console-7bd48fb7cf-wbkqp\" (UID: \"c0a0e084-e243-4fd1-84d0-f429ae783778\") " pod="openshift-console/console-7bd48fb7cf-wbkqp"
Jul 28 04:23:20.103266 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:20.103008    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c0a0e084-e243-4fd1-84d0-f429ae783778-service-ca\") pod \"console-7bd48fb7cf-wbkqp\" (UID: \"c0a0e084-e243-4fd1-84d0-f429ae783778\") " pod="openshift-console/console-7bd48fb7cf-wbkqp"
Jul 28 04:23:20.103266 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:20.103044    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c0a0e084-e243-4fd1-84d0-f429ae783778-trusted-ca-bundle\") pod \"console-7bd48fb7cf-wbkqp\" (UID: \"c0a0e084-e243-4fd1-84d0-f429ae783778\") " pod="openshift-console/console-7bd48fb7cf-wbkqp"
Jul 28 04:23:20.104652 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:20.104630    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c0a0e084-e243-4fd1-84d0-f429ae783778-console-oauth-config\") pod \"console-7bd48fb7cf-wbkqp\" (UID: \"c0a0e084-e243-4fd1-84d0-f429ae783778\") " pod="openshift-console/console-7bd48fb7cf-wbkqp"
Jul 28 04:23:20.104777 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:20.104759    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c0a0e084-e243-4fd1-84d0-f429ae783778-console-serving-cert\") pod \"console-7bd48fb7cf-wbkqp\" (UID: \"c0a0e084-e243-4fd1-84d0-f429ae783778\") " pod="openshift-console/console-7bd48fb7cf-wbkqp"
Jul 28 04:23:20.117277 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:20.117256    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bs2j\" (UniqueName: \"kubernetes.io/projected/c0a0e084-e243-4fd1-84d0-f429ae783778-kube-api-access-5bs2j\") pod \"console-7bd48fb7cf-wbkqp\" (UID: \"c0a0e084-e243-4fd1-84d0-f429ae783778\") " pod="openshift-console/console-7bd48fb7cf-wbkqp"
Jul 28 04:23:20.205936 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:20.205877    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bd48fb7cf-wbkqp"
Jul 28 04:23:20.335207 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:20.335183    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bd48fb7cf-wbkqp"]
Jul 28 04:23:20.336943 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:23:20.336920    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0a0e084_e243_4fd1_84d0_f429ae783778.slice/crio-fed7f55aac020b93df58f1c0aeb0fb502a35b19dd5a3643d1a3d5b8b7be21eb7 WatchSource:0}: Error finding container fed7f55aac020b93df58f1c0aeb0fb502a35b19dd5a3643d1a3d5b8b7be21eb7: Status 404 returned error can't find the container with id fed7f55aac020b93df58f1c0aeb0fb502a35b19dd5a3643d1a3d5b8b7be21eb7
Jul 28 04:23:20.791781 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:20.791743    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bd48fb7cf-wbkqp" event={"ID":"c0a0e084-e243-4fd1-84d0-f429ae783778","Type":"ContainerStarted","Data":"cf89fc6b1109cf0808c81509b850ea3bf70cbda6ce5edade60acceae182bfd24"}
Jul 28 04:23:20.791981 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:20.791791    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bd48fb7cf-wbkqp" event={"ID":"c0a0e084-e243-4fd1-84d0-f429ae783778","Type":"ContainerStarted","Data":"fed7f55aac020b93df58f1c0aeb0fb502a35b19dd5a3643d1a3d5b8b7be21eb7"}
Jul 28 04:23:20.814366 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:20.814317    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7bd48fb7cf-wbkqp" podStartSLOduration=1.814300597 podStartE2EDuration="1.814300597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:23:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:23:20.813586171 +0000 UTC m=+701.203762204" watchObservedRunningTime="2026-07-28 04:23:20.814300597 +0000 UTC m=+701.204476618"
Jul 28 04:23:21.343863 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:21.343811    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-chdx9"]
Jul 28 04:23:21.347158 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:21.347142    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-chdx9"
Jul 28 04:23:21.350037 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:21.350012    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 28 04:23:21.350183 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:21.350042    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 28 04:23:21.350183 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:21.350022    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-72lbj\""
Jul 28 04:23:21.351319 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:21.351305    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 28 04:23:21.358591 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:21.358572    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-chdx9"]
Jul 28 04:23:21.513372 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:21.513340    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwjvl\" (UniqueName: \"kubernetes.io/projected/58a62ded-ca6b-41f7-9e33-16f1702ecdf7-kube-api-access-hwjvl\") pod \"dns-operator-controller-manager-864dd78c4c-chdx9\" (UID: \"58a62ded-ca6b-41f7-9e33-16f1702ecdf7\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-chdx9"
Jul 28 04:23:21.614135 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:21.614076    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwjvl\" (UniqueName: \"kubernetes.io/projected/58a62ded-ca6b-41f7-9e33-16f1702ecdf7-kube-api-access-hwjvl\") pod \"dns-operator-controller-manager-864dd78c4c-chdx9\" (UID: \"58a62ded-ca6b-41f7-9e33-16f1702ecdf7\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-chdx9"
Jul 28 04:23:21.626524 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:21.626494    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwjvl\" (UniqueName: \"kubernetes.io/projected/58a62ded-ca6b-41f7-9e33-16f1702ecdf7-kube-api-access-hwjvl\") pod \"dns-operator-controller-manager-864dd78c4c-chdx9\" (UID: \"58a62ded-ca6b-41f7-9e33-16f1702ecdf7\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-chdx9"
Jul 28 04:23:21.657306 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:21.657284    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-chdx9"
Jul 28 04:23:21.797719 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:21.797695    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-chdx9"]
Jul 28 04:23:21.799271 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:23:21.799245    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a62ded_ca6b_41f7_9e33_16f1702ecdf7.slice/crio-5ccac1281bae8cfd669cd2507ba63019a99caa0773fe25f01174be045c1d060e WatchSource:0}: Error finding container 5ccac1281bae8cfd669cd2507ba63019a99caa0773fe25f01174be045c1d060e: Status 404 returned error can't find the container with id 5ccac1281bae8cfd669cd2507ba63019a99caa0773fe25f01174be045c1d060e
Jul 28 04:23:22.805883 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:22.805824    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-chdx9" event={"ID":"58a62ded-ca6b-41f7-9e33-16f1702ecdf7","Type":"ContainerStarted","Data":"5ccac1281bae8cfd669cd2507ba63019a99caa0773fe25f01174be045c1d060e"}
Jul 28 04:23:23.556096 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:23.556058    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-92pxx"]
Jul 28 04:23:23.559967 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:23.559944    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-92pxx"
Jul 28 04:23:23.562690 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:23.562667    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-xt2pg\""
Jul 28 04:23:23.567303 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:23.567277    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-92pxx"]
Jul 28 04:23:23.732348 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:23.732302    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b82p8\" (UniqueName: \"kubernetes.io/projected/3ff263e0-3528-40ba-8fb7-978cba5c3eff-kube-api-access-b82p8\") pod \"limitador-operator-controller-manager-859d7b78bd-92pxx\" (UID: \"3ff263e0-3528-40ba-8fb7-978cba5c3eff\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-92pxx"
Jul 28 04:23:23.832964 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:23.832882    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b82p8\" (UniqueName: \"kubernetes.io/projected/3ff263e0-3528-40ba-8fb7-978cba5c3eff-kube-api-access-b82p8\") pod \"limitador-operator-controller-manager-859d7b78bd-92pxx\" (UID: \"3ff263e0-3528-40ba-8fb7-978cba5c3eff\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-92pxx"
Jul 28 04:23:23.841482 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:23.841451    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b82p8\" (UniqueName: \"kubernetes.io/projected/3ff263e0-3528-40ba-8fb7-978cba5c3eff-kube-api-access-b82p8\") pod \"limitador-operator-controller-manager-859d7b78bd-92pxx\" (UID: \"3ff263e0-3528-40ba-8fb7-978cba5c3eff\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-92pxx"
Jul 28 04:23:23.873775 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:23.873744    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-92pxx"
Jul 28 04:23:24.025023 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:24.024998    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-92pxx"]
Jul 28 04:23:24.026973 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:23:24.026944    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ff263e0_3528_40ba_8fb7_978cba5c3eff.slice/crio-a50902f5b420df433d6af238290741254761dc082d0401473f33ef1c8d9eaf1b WatchSource:0}: Error finding container a50902f5b420df433d6af238290741254761dc082d0401473f33ef1c8d9eaf1b: Status 404 returned error can't find the container with id a50902f5b420df433d6af238290741254761dc082d0401473f33ef1c8d9eaf1b
Jul 28 04:23:24.817416 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:24.817373    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-chdx9" event={"ID":"58a62ded-ca6b-41f7-9e33-16f1702ecdf7","Type":"ContainerStarted","Data":"d36fddb22d4ac637ceb582453b48feb76a12dbb11388f01cd0bfd0f3261259e2"}
Jul 28 04:23:24.817599 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:24.817481    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-chdx9"
Jul 28 04:23:24.818566 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:24.818540    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-92pxx" event={"ID":"3ff263e0-3528-40ba-8fb7-978cba5c3eff","Type":"ContainerStarted","Data":"a50902f5b420df433d6af238290741254761dc082d0401473f33ef1c8d9eaf1b"}
Jul 28 04:23:24.833696 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:24.833652    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-chdx9" podStartSLOduration=1.733289463 podStartE2EDuration="3.833638221s" totalImagesPullingTime="2.100348758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:23:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:23:24.832502157 +0000 UTC m=+705.222678179" watchObservedRunningTime="2026-07-28 04:23:24.833638221 +0000 UTC m=+705.223814295"
Jul 28 04:23:26.830600 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:26.830565    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-92pxx" event={"ID":"3ff263e0-3528-40ba-8fb7-978cba5c3eff","Type":"ContainerStarted","Data":"28eb8f091f66493f75a1123ed47cc0213dbe463f6af01151aa214ef4badb1b22"}
Jul 28 04:23:26.831005 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:26.830676    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-92pxx"
Jul 28 04:23:26.846665 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:26.846616    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-92pxx" podStartSLOduration=2.095079779 podStartE2EDuration="3.846602736s" totalImagesPullingTime="1.751522957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:23:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:23:26.845992816 +0000 UTC m=+707.236168860" watchObservedRunningTime="2026-07-28 04:23:26.846602736 +0000 UTC m=+707.236778757"
Jul 28 04:23:30.206395 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:30.206362    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7bd48fb7cf-wbkqp"
Jul 28 04:23:30.206794 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:30.206429    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7bd48fb7cf-wbkqp"
Jul 28 04:23:30.211279 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:30.211258    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7bd48fb7cf-wbkqp"
Jul 28 04:23:30.852323 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:30.852295    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7bd48fb7cf-wbkqp"
Jul 28 04:23:35.031862 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:35.031763    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-fh5dt"]
Jul 28 04:23:35.035324 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:35.035301    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-fh5dt"
Jul 28 04:23:35.038055 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:35.038032    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-s2lcg\""
Jul 28 04:23:35.051165 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:35.051144    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-fh5dt"]
Jul 28 04:23:35.125731 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:35.125697    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6slt\" (UniqueName: \"kubernetes.io/projected/a56b2e50-c54d-45a3-afb1-ef088b99b4ac-kube-api-access-l6slt\") pod \"kuadrant-operator-controller-manager-55467d9f7-fh5dt\" (UID: \"a56b2e50-c54d-45a3-afb1-ef088b99b4ac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-fh5dt"
Jul 28 04:23:35.125888 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:35.125745    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a56b2e50-c54d-45a3-afb1-ef088b99b4ac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-fh5dt\" (UID: \"a56b2e50-c54d-45a3-afb1-ef088b99b4ac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-fh5dt"
Jul 28 04:23:35.226333 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:35.226298    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6slt\" (UniqueName: \"kubernetes.io/projected/a56b2e50-c54d-45a3-afb1-ef088b99b4ac-kube-api-access-l6slt\") pod \"kuadrant-operator-controller-manager-55467d9f7-fh5dt\" (UID: \"a56b2e50-c54d-45a3-afb1-ef088b99b4ac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-fh5dt"
Jul 28 04:23:35.226497 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:35.226340    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a56b2e50-c54d-45a3-afb1-ef088b99b4ac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-fh5dt\" (UID: \"a56b2e50-c54d-45a3-afb1-ef088b99b4ac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-fh5dt"
Jul 28 04:23:35.226661 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:35.226644    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a56b2e50-c54d-45a3-afb1-ef088b99b4ac-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-fh5dt\" (UID: \"a56b2e50-c54d-45a3-afb1-ef088b99b4ac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-fh5dt"
Jul 28 04:23:35.238759 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:35.238734    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6slt\" (UniqueName: \"kubernetes.io/projected/a56b2e50-c54d-45a3-afb1-ef088b99b4ac-kube-api-access-l6slt\") pod \"kuadrant-operator-controller-manager-55467d9f7-fh5dt\" (UID: \"a56b2e50-c54d-45a3-afb1-ef088b99b4ac\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-fh5dt"
Jul 28 04:23:35.344532 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:35.344467    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-fh5dt"
Jul 28 04:23:35.353612 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:35.353583    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-fh5dt"]
Jul 28 04:23:35.360879 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:35.360816    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-fh5dt"]
Jul 28 04:23:35.383455 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:35.383427    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gsh7p"]
Jul 28 04:23:35.386979 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:35.386959    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gsh7p"
Jul 28 04:23:35.401526 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:35.401504    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gsh7p"]
Jul 28 04:23:35.428249 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:35.428222    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzx6b\" (UniqueName: \"kubernetes.io/projected/90d5f987-8063-4cb4-9aa2-48a38f9fd84e-kube-api-access-jzx6b\") pod \"kuadrant-operator-controller-manager-55467d9f7-gsh7p\" (UID: \"90d5f987-8063-4cb4-9aa2-48a38f9fd84e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gsh7p"
Jul 28 04:23:35.428350 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:35.428311    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/90d5f987-8063-4cb4-9aa2-48a38f9fd84e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-gsh7p\" (UID: \"90d5f987-8063-4cb4-9aa2-48a38f9fd84e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gsh7p"
Jul 28 04:23:35.432566 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:35.432542    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-92pxx"]
Jul 28 04:23:35.432743 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:35.432722    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-92pxx" podUID="3ff263e0-3528-40ba-8fb7-978cba5c3eff" containerName="manager" containerID="cri-o://28eb8f091f66493f75a1123ed47cc0213dbe463f6af01151aa214ef4badb1b22" gracePeriod=2
Jul 28 04:23:35.434768 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:35.434744    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-92pxx"
Jul 28 04:23:35.435169 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:23:35.435144    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-92pxx\" is forbidden: User \"system:node:ip-10-0-129-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-187.ec2.internal' and this object" podUID="3ff263e0-3528-40ba-8fb7-978cba5c3eff" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-92pxx"
Jul 28 04:23:35.437140 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:35.437056    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-92pxx"]
Jul 28 04:23:35.437356 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:23:35.437326    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-92pxx\" is forbidden: User \"system:node:ip-10-0-129-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-187.ec2.internal' and this object" podUID="3ff263e0-3528-40ba-8fb7-978cba5c3eff" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-92pxx"
Jul 28 04:23:35.529682 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:35.529653    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/90d5f987-8063-4cb4-9aa2-48a38f9fd84e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-gsh7p\" (UID: \"90d5f987-8063-4cb4-9aa2-48a38f9fd84e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gsh7p"
Jul 28 04:23:35.529816 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:35.529728    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzx6b\" (UniqueName: \"kubernetes.io/projected/90d5f987-8063-4cb4-9aa2-48a38f9fd84e-kube-api-access-jzx6b\") pod \"kuadrant-operator-controller-manager-55467d9f7-gsh7p\" (UID: \"90d5f987-8063-4cb4-9aa2-48a38f9fd84e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gsh7p"
Jul 28 04:23:35.530038 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:35.530018    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/90d5f987-8063-4cb4-9aa2-48a38f9fd84e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-gsh7p\" (UID: \"90d5f987-8063-4cb4-9aa2-48a38f9fd84e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gsh7p"
Jul 28 04:23:35.544303 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:35.544276    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzx6b\" (UniqueName: \"kubernetes.io/projected/90d5f987-8063-4cb4-9aa2-48a38f9fd84e-kube-api-access-jzx6b\") pod \"kuadrant-operator-controller-manager-55467d9f7-gsh7p\" (UID: \"90d5f987-8063-4cb4-9aa2-48a38f9fd84e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gsh7p"
Jul 28 04:23:35.657900 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:35.657873    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-92pxx"
Jul 28 04:23:35.660261 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:23:35.660237    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-92pxx\" is forbidden: User \"system:node:ip-10-0-129-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-187.ec2.internal' and this object" podUID="3ff263e0-3528-40ba-8fb7-978cba5c3eff" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-92pxx"
Jul 28 04:23:35.697899 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:35.697875    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gsh7p"
Jul 28 04:23:35.730886 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:35.730859    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b82p8\" (UniqueName: \"kubernetes.io/projected/3ff263e0-3528-40ba-8fb7-978cba5c3eff-kube-api-access-b82p8\") pod \"3ff263e0-3528-40ba-8fb7-978cba5c3eff\" (UID: \"3ff263e0-3528-40ba-8fb7-978cba5c3eff\") "
Jul 28 04:23:35.732880 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:35.732849    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ff263e0-3528-40ba-8fb7-978cba5c3eff-kube-api-access-b82p8" (OuterVolumeSpecName: "kube-api-access-b82p8") pod "3ff263e0-3528-40ba-8fb7-978cba5c3eff" (UID: "3ff263e0-3528-40ba-8fb7-978cba5c3eff"). InnerVolumeSpecName "kube-api-access-b82p8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:23:35.825423 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:35.825395    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gsh7p"]
Jul 28 04:23:35.826855 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:35.826819    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-chdx9"
Jul 28 04:23:35.827231 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:23:35.827209    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90d5f987_8063_4cb4_9aa2_48a38f9fd84e.slice/crio-c48e4f0b8487450ee16e24103aebdaefb693b38ba0719182a61b5928afe09333 WatchSource:0}: Error finding container c48e4f0b8487450ee16e24103aebdaefb693b38ba0719182a61b5928afe09333: Status 404 returned error can't find the container with id c48e4f0b8487450ee16e24103aebdaefb693b38ba0719182a61b5928afe09333
Jul 28 04:23:35.829123 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:23:35.829098    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-92pxx\" is forbidden: User \"system:node:ip-10-0-129-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-187.ec2.internal' and this object" podUID="3ff263e0-3528-40ba-8fb7-978cba5c3eff" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-92pxx"
Jul 28 04:23:35.831577 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:35.831554    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b82p8\" (UniqueName: \"kubernetes.io/projected/3ff263e0-3528-40ba-8fb7-978cba5c3eff-kube-api-access-b82p8\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:23:35.873572 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:35.873498    2566 generic.go:356] "Generic (PLEG): container finished" podID="3ff263e0-3528-40ba-8fb7-978cba5c3eff" containerID="28eb8f091f66493f75a1123ed47cc0213dbe463f6af01151aa214ef4badb1b22" exitCode=0
Jul 28 04:23:35.873572 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:35.873545    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-92pxx"
Jul 28 04:23:35.873742 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:35.873592    2566 scope.go:117] "RemoveContainer" containerID="28eb8f091f66493f75a1123ed47cc0213dbe463f6af01151aa214ef4badb1b22"
Jul 28 04:23:35.875145 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:35.875119    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gsh7p" event={"ID":"90d5f987-8063-4cb4-9aa2-48a38f9fd84e","Type":"ContainerStarted","Data":"c48e4f0b8487450ee16e24103aebdaefb693b38ba0719182a61b5928afe09333"}
Jul 28 04:23:35.876054 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:23:35.876026    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-92pxx\" is forbidden: User \"system:node:ip-10-0-129-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-187.ec2.internal' and this object" podUID="3ff263e0-3528-40ba-8fb7-978cba5c3eff" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-92pxx"
Jul 28 04:23:35.882444 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:35.882333    2566 scope.go:117] "RemoveContainer" containerID="28eb8f091f66493f75a1123ed47cc0213dbe463f6af01151aa214ef4badb1b22"
Jul 28 04:23:35.882758 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:23:35.882653    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28eb8f091f66493f75a1123ed47cc0213dbe463f6af01151aa214ef4badb1b22\": container with ID starting with 28eb8f091f66493f75a1123ed47cc0213dbe463f6af01151aa214ef4badb1b22 not found: ID does not exist" containerID="28eb8f091f66493f75a1123ed47cc0213dbe463f6af01151aa214ef4badb1b22"
Jul 28 04:23:35.882758 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:35.882690    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28eb8f091f66493f75a1123ed47cc0213dbe463f6af01151aa214ef4badb1b22"} err="failed to get container status \"28eb8f091f66493f75a1123ed47cc0213dbe463f6af01151aa214ef4badb1b22\": rpc error: code = NotFound desc = could not find container \"28eb8f091f66493f75a1123ed47cc0213dbe463f6af01151aa214ef4badb1b22\": container with ID starting with 28eb8f091f66493f75a1123ed47cc0213dbe463f6af01151aa214ef4badb1b22 not found: ID does not exist"
Jul 28 04:23:35.885435 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:23:35.885411    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-92pxx\" is forbidden: User \"system:node:ip-10-0-129-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-187.ec2.internal' and this object" podUID="3ff263e0-3528-40ba-8fb7-978cba5c3eff" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-92pxx"
Jul 28 04:23:36.087933 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:36.087904    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ff263e0-3528-40ba-8fb7-978cba5c3eff" path="/var/lib/kubelet/pods/3ff263e0-3528-40ba-8fb7-978cba5c3eff/volumes"
Jul 28 04:23:37.891525 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:23:37.891486    2566 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-fh5dt_kuadrant-system_a56b2e50-c54d-45a3-afb1-ef088b99b4ac_0(f284d06220d2115ff2340795ac93437c30a17ea8cf2b62b6dd40c73a91f0d46e): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-fh5dt to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}"
Jul 28 04:23:37.891822 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:23:37.891548    2566 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-fh5dt_kuadrant-system_a56b2e50-c54d-45a3-afb1-ef088b99b4ac_0(f284d06220d2115ff2340795ac93437c30a17ea8cf2b62b6dd40c73a91f0d46e): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-fh5dt to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-fh5dt"
Jul 28 04:23:38.891494 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:38.891465    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-fh5dt"
Jul 28 04:23:38.898878 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:38.898852    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-fh5dt"
Jul 28 04:23:38.901301 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:23:38.901272    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-fh5dt\" is forbidden: User \"system:node:ip-10-0-129-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-187.ec2.internal' and this object" podUID="a56b2e50-c54d-45a3-afb1-ef088b99b4ac" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-fh5dt"
Jul 28 04:23:38.961167 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:38.961135    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6slt\" (UniqueName: \"kubernetes.io/projected/a56b2e50-c54d-45a3-afb1-ef088b99b4ac-kube-api-access-l6slt\") pod \"a56b2e50-c54d-45a3-afb1-ef088b99b4ac\" (UID: \"a56b2e50-c54d-45a3-afb1-ef088b99b4ac\") "
Jul 28 04:23:38.961328 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:38.961310    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a56b2e50-c54d-45a3-afb1-ef088b99b4ac-extensions-socket-volume\") pod \"a56b2e50-c54d-45a3-afb1-ef088b99b4ac\" (UID: \"a56b2e50-c54d-45a3-afb1-ef088b99b4ac\") "
Jul 28 04:23:38.961584 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:38.961551    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a56b2e50-c54d-45a3-afb1-ef088b99b4ac-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a56b2e50-c54d-45a3-afb1-ef088b99b4ac" (UID: "a56b2e50-c54d-45a3-afb1-ef088b99b4ac"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:23:38.963498 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:38.963464    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a56b2e50-c54d-45a3-afb1-ef088b99b4ac-kube-api-access-l6slt" (OuterVolumeSpecName: "kube-api-access-l6slt") pod "a56b2e50-c54d-45a3-afb1-ef088b99b4ac" (UID: "a56b2e50-c54d-45a3-afb1-ef088b99b4ac"). InnerVolumeSpecName "kube-api-access-l6slt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:23:39.062557 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:39.062519    2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a56b2e50-c54d-45a3-afb1-ef088b99b4ac-extensions-socket-volume\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:23:39.062557 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:39.062550    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6slt\" (UniqueName: \"kubernetes.io/projected/a56b2e50-c54d-45a3-afb1-ef088b99b4ac-kube-api-access-l6slt\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:23:39.897204 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:39.897173    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-fh5dt"
Jul 28 04:23:39.897369 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:39.897175    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gsh7p" event={"ID":"90d5f987-8063-4cb4-9aa2-48a38f9fd84e","Type":"ContainerStarted","Data":"5f86bbc7304c7052e25256a7be7b79c25101ee11641cc73b95748d78f4346537"}
Jul 28 04:23:39.897448 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:39.897428    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gsh7p"
Jul 28 04:23:39.917761 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:39.917717    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gsh7p" podStartSLOduration=1.047079669 podStartE2EDuration="4.917700908s" totalImagesPullingTime="3.870621239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:23:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:23:39.916521906 +0000 UTC m=+720.306697945" watchObservedRunningTime="2026-07-28 04:23:39.917700908 +0000 UTC m=+720.307876930"
Jul 28 04:23:39.918662 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:23:39.918628    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-fh5dt\" is forbidden: User \"system:node:ip-10-0-129-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-187.ec2.internal' and this object" podUID="a56b2e50-c54d-45a3-afb1-ef088b99b4ac" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-fh5dt"
Jul 28 04:23:39.921873 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:23:39.921819    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-fh5dt\" is forbidden: User \"system:node:ip-10-0-129-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-187.ec2.internal' and this object" podUID="a56b2e50-c54d-45a3-afb1-ef088b99b4ac" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-fh5dt"
Jul 28 04:23:40.086145 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:23:40.086094    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-fh5dt\" is forbidden: User \"system:node:ip-10-0-129-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-187.ec2.internal' and this object" podUID="a56b2e50-c54d-45a3-afb1-ef088b99b4ac" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-fh5dt"
Jul 28 04:23:40.087297 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:40.087274    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a56b2e50-c54d-45a3-afb1-ef088b99b4ac" path="/var/lib/kubelet/pods/a56b2e50-c54d-45a3-afb1-ef088b99b4ac/volumes"
Jul 28 04:23:51.908425 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:51.908397    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gsh7p"
Jul 28 04:23:52.242944 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:52.242914    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-klr2g"]
Jul 28 04:23:52.243247 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:52.243235    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ff263e0-3528-40ba-8fb7-978cba5c3eff" containerName="manager"
Jul 28 04:23:52.243290 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:52.243249    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ff263e0-3528-40ba-8fb7-978cba5c3eff" containerName="manager"
Jul 28 04:23:52.243326 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:52.243301    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ff263e0-3528-40ba-8fb7-978cba5c3eff" containerName="manager"
Jul 28 04:23:52.251334 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:52.251315    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-klr2g"
Jul 28 04:23:52.260366 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:52.260345    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-klr2g"]
Jul 28 04:23:52.375957 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:52.375928    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3631aa8e-ef0c-45eb-aaf7-b9e2d2e344e2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-klr2g\" (UID: \"3631aa8e-ef0c-45eb-aaf7-b9e2d2e344e2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-klr2g"
Jul 28 04:23:52.376112 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:52.375979    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x59l4\" (UniqueName: \"kubernetes.io/projected/3631aa8e-ef0c-45eb-aaf7-b9e2d2e344e2-kube-api-access-x59l4\") pod \"kuadrant-operator-controller-manager-86878c769f-klr2g\" (UID: \"3631aa8e-ef0c-45eb-aaf7-b9e2d2e344e2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-klr2g"
Jul 28 04:23:52.476702 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:52.476619    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3631aa8e-ef0c-45eb-aaf7-b9e2d2e344e2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-klr2g\" (UID: \"3631aa8e-ef0c-45eb-aaf7-b9e2d2e344e2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-klr2g"
Jul 28 04:23:52.476885 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:52.476749    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x59l4\" (UniqueName: \"kubernetes.io/projected/3631aa8e-ef0c-45eb-aaf7-b9e2d2e344e2-kube-api-access-x59l4\") pod \"kuadrant-operator-controller-manager-86878c769f-klr2g\" (UID: \"3631aa8e-ef0c-45eb-aaf7-b9e2d2e344e2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-klr2g"
Jul 28 04:23:52.477018 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:52.476987    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3631aa8e-ef0c-45eb-aaf7-b9e2d2e344e2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-klr2g\" (UID: \"3631aa8e-ef0c-45eb-aaf7-b9e2d2e344e2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-klr2g"
Jul 28 04:23:52.487155 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:52.487135    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x59l4\" (UniqueName: \"kubernetes.io/projected/3631aa8e-ef0c-45eb-aaf7-b9e2d2e344e2-kube-api-access-x59l4\") pod \"kuadrant-operator-controller-manager-86878c769f-klr2g\" (UID: \"3631aa8e-ef0c-45eb-aaf7-b9e2d2e344e2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-klr2g"
Jul 28 04:23:52.561719 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:52.561694    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-klr2g"
Jul 28 04:23:52.693120 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:52.693097    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-klr2g"]
Jul 28 04:23:52.695721 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:23:52.695692    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3631aa8e_ef0c_45eb_aaf7_b9e2d2e344e2.slice/crio-d4c85146b4dd83a883eed402ff32eadec1c98306f718a329e3f43c295c76e42f WatchSource:0}: Error finding container d4c85146b4dd83a883eed402ff32eadec1c98306f718a329e3f43c295c76e42f: Status 404 returned error can't find the container with id d4c85146b4dd83a883eed402ff32eadec1c98306f718a329e3f43c295c76e42f
Jul 28 04:23:52.955421 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:52.955380    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-klr2g" event={"ID":"3631aa8e-ef0c-45eb-aaf7-b9e2d2e344e2","Type":"ContainerStarted","Data":"ceaa31864e77c813bcf5b9c57637ef3f896125d1cd4e1534d83a1d54c986ad68"}
Jul 28 04:23:52.955813 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:52.955432    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-klr2g" event={"ID":"3631aa8e-ef0c-45eb-aaf7-b9e2d2e344e2","Type":"ContainerStarted","Data":"d4c85146b4dd83a883eed402ff32eadec1c98306f718a329e3f43c295c76e42f"}
Jul 28 04:23:52.955813 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:52.955546    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-klr2g"
Jul 28 04:23:52.982560 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:23:52.982514    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-klr2g" podStartSLOduration=0.982500312 podStartE2EDuration="982.500312ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:23:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:23:52.98195871 +0000 UTC m=+733.372134754" watchObservedRunningTime="2026-07-28 04:23:52.982500312 +0000 UTC m=+733.372676333"
Jul 28 04:24:03.963334 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:03.963298    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-klr2g"
Jul 28 04:24:04.008345 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:04.008313    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gsh7p"]
Jul 28 04:24:04.008568 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:04.008543    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gsh7p" podUID="90d5f987-8063-4cb4-9aa2-48a38f9fd84e" containerName="manager" containerID="cri-o://5f86bbc7304c7052e25256a7be7b79c25101ee11641cc73b95748d78f4346537" gracePeriod=10
Jul 28 04:24:04.263760 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:04.263736    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gsh7p"
Jul 28 04:24:04.383898 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:04.383873    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/90d5f987-8063-4cb4-9aa2-48a38f9fd84e-extensions-socket-volume\") pod \"90d5f987-8063-4cb4-9aa2-48a38f9fd84e\" (UID: \"90d5f987-8063-4cb4-9aa2-48a38f9fd84e\") "
Jul 28 04:24:04.384092 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:04.383994    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzx6b\" (UniqueName: \"kubernetes.io/projected/90d5f987-8063-4cb4-9aa2-48a38f9fd84e-kube-api-access-jzx6b\") pod \"90d5f987-8063-4cb4-9aa2-48a38f9fd84e\" (UID: \"90d5f987-8063-4cb4-9aa2-48a38f9fd84e\") "
Jul 28 04:24:04.384346 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:04.384322    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d5f987-8063-4cb4-9aa2-48a38f9fd84e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "90d5f987-8063-4cb4-9aa2-48a38f9fd84e" (UID: "90d5f987-8063-4cb4-9aa2-48a38f9fd84e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:24:04.386014 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:04.385998    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90d5f987-8063-4cb4-9aa2-48a38f9fd84e-kube-api-access-jzx6b" (OuterVolumeSpecName: "kube-api-access-jzx6b") pod "90d5f987-8063-4cb4-9aa2-48a38f9fd84e" (UID: "90d5f987-8063-4cb4-9aa2-48a38f9fd84e"). InnerVolumeSpecName "kube-api-access-jzx6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:24:04.485572 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:04.485540    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzx6b\" (UniqueName: \"kubernetes.io/projected/90d5f987-8063-4cb4-9aa2-48a38f9fd84e-kube-api-access-jzx6b\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:24:04.485572 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:04.485565    2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/90d5f987-8063-4cb4-9aa2-48a38f9fd84e-extensions-socket-volume\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:24:05.010433 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:05.010396    2566 generic.go:356] "Generic (PLEG): container finished" podID="90d5f987-8063-4cb4-9aa2-48a38f9fd84e" containerID="5f86bbc7304c7052e25256a7be7b79c25101ee11641cc73b95748d78f4346537" exitCode=0
Jul 28 04:24:05.010904 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:05.010440    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gsh7p"
Jul 28 04:24:05.010904 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:05.010463    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gsh7p" event={"ID":"90d5f987-8063-4cb4-9aa2-48a38f9fd84e","Type":"ContainerDied","Data":"5f86bbc7304c7052e25256a7be7b79c25101ee11641cc73b95748d78f4346537"}
Jul 28 04:24:05.010904 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:05.010506    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gsh7p" event={"ID":"90d5f987-8063-4cb4-9aa2-48a38f9fd84e","Type":"ContainerDied","Data":"c48e4f0b8487450ee16e24103aebdaefb693b38ba0719182a61b5928afe09333"}
Jul 28 04:24:05.010904 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:05.010527    2566 scope.go:117] "RemoveContainer" containerID="5f86bbc7304c7052e25256a7be7b79c25101ee11641cc73b95748d78f4346537"
Jul 28 04:24:05.019521 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:05.019507    2566 scope.go:117] "RemoveContainer" containerID="5f86bbc7304c7052e25256a7be7b79c25101ee11641cc73b95748d78f4346537"
Jul 28 04:24:05.019762 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:24:05.019746    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f86bbc7304c7052e25256a7be7b79c25101ee11641cc73b95748d78f4346537\": container with ID starting with 5f86bbc7304c7052e25256a7be7b79c25101ee11641cc73b95748d78f4346537 not found: ID does not exist" containerID="5f86bbc7304c7052e25256a7be7b79c25101ee11641cc73b95748d78f4346537"
Jul 28 04:24:05.019801 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:05.019770    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f86bbc7304c7052e25256a7be7b79c25101ee11641cc73b95748d78f4346537"} err="failed to get container status \"5f86bbc7304c7052e25256a7be7b79c25101ee11641cc73b95748d78f4346537\": rpc error: code = NotFound desc = could not find container \"5f86bbc7304c7052e25256a7be7b79c25101ee11641cc73b95748d78f4346537\": container with ID starting with 5f86bbc7304c7052e25256a7be7b79c25101ee11641cc73b95748d78f4346537 not found: ID does not exist"
Jul 28 04:24:05.032665 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:05.032644    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gsh7p"]
Jul 28 04:24:05.034160 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:05.034139    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-gsh7p"]
Jul 28 04:24:06.087966 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:06.087936    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90d5f987-8063-4cb4-9aa2-48a38f9fd84e" path="/var/lib/kubelet/pods/90d5f987-8063-4cb4-9aa2-48a38f9fd84e/volumes"
Jul 28 04:24:24.644643 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:24.644598    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-5tb48"]
Jul 28 04:24:24.645150 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:24.644950    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90d5f987-8063-4cb4-9aa2-48a38f9fd84e" containerName="manager"
Jul 28 04:24:24.645150 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:24.644962    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="90d5f987-8063-4cb4-9aa2-48a38f9fd84e" containerName="manager"
Jul 28 04:24:24.645150 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:24.645019    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="90d5f987-8063-4cb4-9aa2-48a38f9fd84e" containerName="manager"
Jul 28 04:24:24.649613 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:24.649591    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-5tb48"
Jul 28 04:24:24.652046 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:24.652018    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9jn4d\""
Jul 28 04:24:24.654368 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:24.654348    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-5tb48"]
Jul 28 04:24:24.750027 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:24.749995    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-975kk\" (UniqueName: \"kubernetes.io/projected/078e3aaf-a23f-4f19-b68a-c83e8dfb4b92-kube-api-access-975kk\") pod \"authorino-58cd955675-5tb48\" (UID: \"078e3aaf-a23f-4f19-b68a-c83e8dfb4b92\") " pod="kuadrant-system/authorino-58cd955675-5tb48"
Jul 28 04:24:24.851424 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:24.851392    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-975kk\" (UniqueName: \"kubernetes.io/projected/078e3aaf-a23f-4f19-b68a-c83e8dfb4b92-kube-api-access-975kk\") pod \"authorino-58cd955675-5tb48\" (UID: \"078e3aaf-a23f-4f19-b68a-c83e8dfb4b92\") " pod="kuadrant-system/authorino-58cd955675-5tb48"
Jul 28 04:24:24.859166 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:24.859136    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-975kk\" (UniqueName: \"kubernetes.io/projected/078e3aaf-a23f-4f19-b68a-c83e8dfb4b92-kube-api-access-975kk\") pod \"authorino-58cd955675-5tb48\" (UID: \"078e3aaf-a23f-4f19-b68a-c83e8dfb4b92\") " pod="kuadrant-system/authorino-58cd955675-5tb48"
Jul 28 04:24:24.979046 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:24.978978    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-5tb48"
Jul 28 04:24:25.105914 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:25.105877    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-5tb48"]
Jul 28 04:24:25.110598 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:24:25.110573    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod078e3aaf_a23f_4f19_b68a_c83e8dfb4b92.slice/crio-4cbb4368ed54a791c78b11ba873b979f0955ce5c78b190b75158f59f4d487c85 WatchSource:0}: Error finding container 4cbb4368ed54a791c78b11ba873b979f0955ce5c78b190b75158f59f4d487c85: Status 404 returned error can't find the container with id 4cbb4368ed54a791c78b11ba873b979f0955ce5c78b190b75158f59f4d487c85
Jul 28 04:24:26.118517 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:26.118476    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-5tb48" event={"ID":"078e3aaf-a23f-4f19-b68a-c83e8dfb4b92","Type":"ContainerStarted","Data":"4cbb4368ed54a791c78b11ba873b979f0955ce5c78b190b75158f59f4d487c85"}
Jul 28 04:24:29.138451 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:29.138409    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-5tb48" event={"ID":"078e3aaf-a23f-4f19-b68a-c83e8dfb4b92","Type":"ContainerStarted","Data":"873d65d789bcc2c129982ae5b79b2e3f2a3e66a85f90eaf8a8c4ae7ad2e56dc0"}
Jul 28 04:24:29.153025 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:29.152981    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-5tb48" podStartSLOduration=2.059001732 podStartE2EDuration="5.152967183s" totalImagesPullingTime="3.093965451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:24:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:24:29.151671676 +0000 UTC m=+769.541847709" watchObservedRunningTime="2026-07-28 04:24:29.152967183 +0000 UTC m=+769.543143204"
Jul 28 04:24:29.966498 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:29.966465    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-5tb48"]
Jul 28 04:24:31.147215 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:31.147159    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-5tb48" podUID="078e3aaf-a23f-4f19-b68a-c83e8dfb4b92" containerName="authorino" containerID="cri-o://873d65d789bcc2c129982ae5b79b2e3f2a3e66a85f90eaf8a8c4ae7ad2e56dc0" gracePeriod=30
Jul 28 04:24:31.403592 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:31.403533    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-5tb48"
Jul 28 04:24:31.507218 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:31.507187    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-975kk\" (UniqueName: \"kubernetes.io/projected/078e3aaf-a23f-4f19-b68a-c83e8dfb4b92-kube-api-access-975kk\") pod \"078e3aaf-a23f-4f19-b68a-c83e8dfb4b92\" (UID: \"078e3aaf-a23f-4f19-b68a-c83e8dfb4b92\") "
Jul 28 04:24:31.509143 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:31.509115    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/078e3aaf-a23f-4f19-b68a-c83e8dfb4b92-kube-api-access-975kk" (OuterVolumeSpecName: "kube-api-access-975kk") pod "078e3aaf-a23f-4f19-b68a-c83e8dfb4b92" (UID: "078e3aaf-a23f-4f19-b68a-c83e8dfb4b92"). InnerVolumeSpecName "kube-api-access-975kk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:24:31.608351 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:31.608314    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-975kk\" (UniqueName: \"kubernetes.io/projected/078e3aaf-a23f-4f19-b68a-c83e8dfb4b92-kube-api-access-975kk\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:24:32.152916 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:32.152883    2566 generic.go:356] "Generic (PLEG): container finished" podID="078e3aaf-a23f-4f19-b68a-c83e8dfb4b92" containerID="873d65d789bcc2c129982ae5b79b2e3f2a3e66a85f90eaf8a8c4ae7ad2e56dc0" exitCode=0
Jul 28 04:24:32.153331 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:32.152920    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-5tb48"
Jul 28 04:24:32.153331 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:32.152971    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-5tb48" event={"ID":"078e3aaf-a23f-4f19-b68a-c83e8dfb4b92","Type":"ContainerDied","Data":"873d65d789bcc2c129982ae5b79b2e3f2a3e66a85f90eaf8a8c4ae7ad2e56dc0"}
Jul 28 04:24:32.153331 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:32.153018    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-5tb48" event={"ID":"078e3aaf-a23f-4f19-b68a-c83e8dfb4b92","Type":"ContainerDied","Data":"4cbb4368ed54a791c78b11ba873b979f0955ce5c78b190b75158f59f4d487c85"}
Jul 28 04:24:32.153331 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:32.153044    2566 scope.go:117] "RemoveContainer" containerID="873d65d789bcc2c129982ae5b79b2e3f2a3e66a85f90eaf8a8c4ae7ad2e56dc0"
Jul 28 04:24:32.161501 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:32.161487    2566 scope.go:117] "RemoveContainer" containerID="873d65d789bcc2c129982ae5b79b2e3f2a3e66a85f90eaf8a8c4ae7ad2e56dc0"
Jul 28 04:24:32.161772 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:24:32.161755    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"873d65d789bcc2c129982ae5b79b2e3f2a3e66a85f90eaf8a8c4ae7ad2e56dc0\": container with ID starting with 873d65d789bcc2c129982ae5b79b2e3f2a3e66a85f90eaf8a8c4ae7ad2e56dc0 not found: ID does not exist" containerID="873d65d789bcc2c129982ae5b79b2e3f2a3e66a85f90eaf8a8c4ae7ad2e56dc0"
Jul 28 04:24:32.161833 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:32.161791    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"873d65d789bcc2c129982ae5b79b2e3f2a3e66a85f90eaf8a8c4ae7ad2e56dc0"} err="failed to get container status \"873d65d789bcc2c129982ae5b79b2e3f2a3e66a85f90eaf8a8c4ae7ad2e56dc0\": rpc error: code = NotFound desc = could not find container \"873d65d789bcc2c129982ae5b79b2e3f2a3e66a85f90eaf8a8c4ae7ad2e56dc0\": container with ID starting with 873d65d789bcc2c129982ae5b79b2e3f2a3e66a85f90eaf8a8c4ae7ad2e56dc0 not found: ID does not exist"
Jul 28 04:24:32.167998 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:32.167976    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-5tb48"]
Jul 28 04:24:32.171237 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:32.171218    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-5tb48"]
Jul 28 04:24:34.087896 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:34.087859    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="078e3aaf-a23f-4f19-b68a-c83e8dfb4b92" path="/var/lib/kubelet/pods/078e3aaf-a23f-4f19-b68a-c83e8dfb4b92/volumes"
Jul 28 04:24:55.625935 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:55.625898    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-s9ks8"]
Jul 28 04:24:55.626380 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:55.626272    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="078e3aaf-a23f-4f19-b68a-c83e8dfb4b92" containerName="authorino"
Jul 28 04:24:55.626380 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:55.626285    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="078e3aaf-a23f-4f19-b68a-c83e8dfb4b92" containerName="authorino"
Jul 28 04:24:55.626380 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:55.626345    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="078e3aaf-a23f-4f19-b68a-c83e8dfb4b92" containerName="authorino"
Jul 28 04:24:55.633430 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:55.633408    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-s9ks8"
Jul 28 04:24:55.635046 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:55.635021    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-s9ks8"]
Jul 28 04:24:55.635790 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:55.635771    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9jn4d\""
Jul 28 04:24:55.800335 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:55.800296    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-s9ks8"]
Jul 28 04:24:55.800605 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:24:55.800583    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-xghmj], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-s9ks8" podUID="88acb457-0cbc-4e33-9ffc-a6766bb11a7a"
Jul 28 04:24:55.807889 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:55.807851    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xghmj\" (UniqueName: \"kubernetes.io/projected/88acb457-0cbc-4e33-9ffc-a6766bb11a7a-kube-api-access-xghmj\") pod \"authorino-5d4b67c446-s9ks8\" (UID: \"88acb457-0cbc-4e33-9ffc-a6766bb11a7a\") " pod="kuadrant-system/authorino-5d4b67c446-s9ks8"
Jul 28 04:24:55.825236 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:55.825210    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76877994-n4rrv"]
Jul 28 04:24:55.828702 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:55.828683    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-n4rrv"
Jul 28 04:24:55.831413 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:55.831390    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 28 04:24:55.834424 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:55.834399    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76877994-n4rrv"]
Jul 28 04:24:55.899870 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:55.899770    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-n4rrv"]
Jul 28 04:24:55.900063 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:24:55.900042    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-jwq8x tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-76877994-n4rrv" podUID="6a20cf53-4f76-4db0-9135-48c8166ec6b6"
Jul 28 04:24:55.908467 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:55.908445    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6a20cf53-4f76-4db0-9135-48c8166ec6b6-tls-cert\") pod \"authorino-76877994-n4rrv\" (UID: \"6a20cf53-4f76-4db0-9135-48c8166ec6b6\") " pod="kuadrant-system/authorino-76877994-n4rrv"
Jul 28 04:24:55.908535 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:55.908474    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwq8x\" (UniqueName: \"kubernetes.io/projected/6a20cf53-4f76-4db0-9135-48c8166ec6b6-kube-api-access-jwq8x\") pod \"authorino-76877994-n4rrv\" (UID: \"6a20cf53-4f76-4db0-9135-48c8166ec6b6\") " pod="kuadrant-system/authorino-76877994-n4rrv"
Jul 28 04:24:55.908535 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:55.908517    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xghmj\" (UniqueName: \"kubernetes.io/projected/88acb457-0cbc-4e33-9ffc-a6766bb11a7a-kube-api-access-xghmj\") pod \"authorino-5d4b67c446-s9ks8\" (UID: \"88acb457-0cbc-4e33-9ffc-a6766bb11a7a\") " pod="kuadrant-system/authorino-5d4b67c446-s9ks8"
Jul 28 04:24:55.920402 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:55.920371    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xghmj\" (UniqueName: \"kubernetes.io/projected/88acb457-0cbc-4e33-9ffc-a6766bb11a7a-kube-api-access-xghmj\") pod \"authorino-5d4b67c446-s9ks8\" (UID: \"88acb457-0cbc-4e33-9ffc-a6766bb11a7a\") " pod="kuadrant-system/authorino-5d4b67c446-s9ks8"
Jul 28 04:24:56.009246 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:56.009214    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6a20cf53-4f76-4db0-9135-48c8166ec6b6-tls-cert\") pod \"authorino-76877994-n4rrv\" (UID: \"6a20cf53-4f76-4db0-9135-48c8166ec6b6\") " pod="kuadrant-system/authorino-76877994-n4rrv"
Jul 28 04:24:56.009246 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:56.009248    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwq8x\" (UniqueName: \"kubernetes.io/projected/6a20cf53-4f76-4db0-9135-48c8166ec6b6-kube-api-access-jwq8x\") pod \"authorino-76877994-n4rrv\" (UID: \"6a20cf53-4f76-4db0-9135-48c8166ec6b6\") " pod="kuadrant-system/authorino-76877994-n4rrv"
Jul 28 04:24:56.011526 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:56.011501    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6a20cf53-4f76-4db0-9135-48c8166ec6b6-tls-cert\") pod \"authorino-76877994-n4rrv\" (UID: \"6a20cf53-4f76-4db0-9135-48c8166ec6b6\") " pod="kuadrant-system/authorino-76877994-n4rrv"
Jul 28 04:24:56.016833 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:56.016809    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwq8x\" (UniqueName: \"kubernetes.io/projected/6a20cf53-4f76-4db0-9135-48c8166ec6b6-kube-api-access-jwq8x\") pod \"authorino-76877994-n4rrv\" (UID: \"6a20cf53-4f76-4db0-9135-48c8166ec6b6\") " pod="kuadrant-system/authorino-76877994-n4rrv"
Jul 28 04:24:56.263775 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:56.263689    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-s9ks8"
Jul 28 04:24:56.263775 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:56.263701    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-n4rrv"
Jul 28 04:24:56.286630 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:56.286601    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-s9ks8"
Jul 28 04:24:56.289408 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:56.289387    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-n4rrv"
Jul 28 04:24:56.412049 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:56.412017    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xghmj\" (UniqueName: \"kubernetes.io/projected/88acb457-0cbc-4e33-9ffc-a6766bb11a7a-kube-api-access-xghmj\") pod \"88acb457-0cbc-4e33-9ffc-a6766bb11a7a\" (UID: \"88acb457-0cbc-4e33-9ffc-a6766bb11a7a\") "
Jul 28 04:24:56.412225 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:56.412089    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwq8x\" (UniqueName: \"kubernetes.io/projected/6a20cf53-4f76-4db0-9135-48c8166ec6b6-kube-api-access-jwq8x\") pod \"6a20cf53-4f76-4db0-9135-48c8166ec6b6\" (UID: \"6a20cf53-4f76-4db0-9135-48c8166ec6b6\") "
Jul 28 04:24:56.412292 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:56.412255    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6a20cf53-4f76-4db0-9135-48c8166ec6b6-tls-cert\") pod \"6a20cf53-4f76-4db0-9135-48c8166ec6b6\" (UID: \"6a20cf53-4f76-4db0-9135-48c8166ec6b6\") "
Jul 28 04:24:56.414135 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:56.414101    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88acb457-0cbc-4e33-9ffc-a6766bb11a7a-kube-api-access-xghmj" (OuterVolumeSpecName: "kube-api-access-xghmj") pod "88acb457-0cbc-4e33-9ffc-a6766bb11a7a" (UID: "88acb457-0cbc-4e33-9ffc-a6766bb11a7a"). InnerVolumeSpecName "kube-api-access-xghmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:24:56.414242 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:56.414169    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a20cf53-4f76-4db0-9135-48c8166ec6b6-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "6a20cf53-4f76-4db0-9135-48c8166ec6b6" (UID: "6a20cf53-4f76-4db0-9135-48c8166ec6b6"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 04:24:56.414242 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:56.414181    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a20cf53-4f76-4db0-9135-48c8166ec6b6-kube-api-access-jwq8x" (OuterVolumeSpecName: "kube-api-access-jwq8x") pod "6a20cf53-4f76-4db0-9135-48c8166ec6b6" (UID: "6a20cf53-4f76-4db0-9135-48c8166ec6b6"). InnerVolumeSpecName "kube-api-access-jwq8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:24:56.513646 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:56.513611    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xghmj\" (UniqueName: \"kubernetes.io/projected/88acb457-0cbc-4e33-9ffc-a6766bb11a7a-kube-api-access-xghmj\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:24:56.513646 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:56.513640    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwq8x\" (UniqueName: \"kubernetes.io/projected/6a20cf53-4f76-4db0-9135-48c8166ec6b6-kube-api-access-jwq8x\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:24:56.513646 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:56.513651    2566 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6a20cf53-4f76-4db0-9135-48c8166ec6b6-tls-cert\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:24:57.268441 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:57.268410    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-s9ks8"
Jul 28 04:24:57.268818 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:57.268410    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-n4rrv"
Jul 28 04:24:57.299043 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:57.299017    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-s9ks8"]
Jul 28 04:24:57.302078 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:57.302053    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-s9ks8"]
Jul 28 04:24:57.321345 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:57.321320    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-n4rrv"]
Jul 28 04:24:57.323076 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:57.323049    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76877994-n4rrv"]
Jul 28 04:24:58.089908 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:58.089871    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a20cf53-4f76-4db0-9135-48c8166ec6b6" path="/var/lib/kubelet/pods/6a20cf53-4f76-4db0-9135-48c8166ec6b6/volumes"
Jul 28 04:24:58.090217 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:24:58.090197    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88acb457-0cbc-4e33-9ffc-a6766bb11a7a" path="/var/lib/kubelet/pods/88acb457-0cbc-4e33-9ffc-a6766bb11a7a/volumes"
Jul 28 04:25:03.045021 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:03.044945    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-fdb466f56-wtq8f"]
Jul 28 04:25:03.048684 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:03.048664    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fdb466f56-wtq8f"
Jul 28 04:25:03.051232 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:03.051204    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 28 04:25:03.051232 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:03.051224    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 28 04:25:03.051366 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:03.051211    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wxk49\""
Jul 28 04:25:03.054961 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:03.054859    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-fdb466f56-wtq8f"]
Jul 28 04:25:03.173030 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:03.172993    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwsrx\" (UniqueName: \"kubernetes.io/projected/92313a5c-1dda-464a-b36d-75ac6509e13f-kube-api-access-qwsrx\") pod \"maas-controller-fdb466f56-wtq8f\" (UID: \"92313a5c-1dda-464a-b36d-75ac6509e13f\") " pod="opendatahub/maas-controller-fdb466f56-wtq8f"
Jul 28 04:25:03.173030 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:03.173033    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92313a5c-1dda-464a-b36d-75ac6509e13f-webhook-cert\") pod \"maas-controller-fdb466f56-wtq8f\" (UID: \"92313a5c-1dda-464a-b36d-75ac6509e13f\") " pod="opendatahub/maas-controller-fdb466f56-wtq8f"
Jul 28 04:25:03.225833 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:03.225757    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7847888f58-gc6c8"]
Jul 28 04:25:03.229662 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:03.229641    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7847888f58-gc6c8"
Jul 28 04:25:03.238390 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:03.238338    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7847888f58-gc6c8"]
Jul 28 04:25:03.273949 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:03.273910    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwsrx\" (UniqueName: \"kubernetes.io/projected/92313a5c-1dda-464a-b36d-75ac6509e13f-kube-api-access-qwsrx\") pod \"maas-controller-fdb466f56-wtq8f\" (UID: \"92313a5c-1dda-464a-b36d-75ac6509e13f\") " pod="opendatahub/maas-controller-fdb466f56-wtq8f"
Jul 28 04:25:03.274128 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:03.273963    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92313a5c-1dda-464a-b36d-75ac6509e13f-webhook-cert\") pod \"maas-controller-fdb466f56-wtq8f\" (UID: \"92313a5c-1dda-464a-b36d-75ac6509e13f\") " pod="opendatahub/maas-controller-fdb466f56-wtq8f"
Jul 28 04:25:03.274128 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:25:03.274106    2566 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Jul 28 04:25:03.274238 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:25:03.274167    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92313a5c-1dda-464a-b36d-75ac6509e13f-webhook-cert podName:92313a5c-1dda-464a-b36d-75ac6509e13f nodeName:}" failed. No retries permitted until 2026-07-28 04:25:03.774150509 +0000 UTC m=+804.164326511 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/92313a5c-1dda-464a-b36d-75ac6509e13f-webhook-cert") pod "maas-controller-fdb466f56-wtq8f" (UID: "92313a5c-1dda-464a-b36d-75ac6509e13f") : secret "maas-controller-webhook-cert" not found
Jul 28 04:25:03.282678 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:03.282652    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwsrx\" (UniqueName: \"kubernetes.io/projected/92313a5c-1dda-464a-b36d-75ac6509e13f-kube-api-access-qwsrx\") pod \"maas-controller-fdb466f56-wtq8f\" (UID: \"92313a5c-1dda-464a-b36d-75ac6509e13f\") " pod="opendatahub/maas-controller-fdb466f56-wtq8f"
Jul 28 04:25:03.375225 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:03.375130    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0208cc05-10d1-4d71-b063-24b462ebfa0b-webhook-cert\") pod \"maas-controller-7847888f58-gc6c8\" (UID: \"0208cc05-10d1-4d71-b063-24b462ebfa0b\") " pod="opendatahub/maas-controller-7847888f58-gc6c8"
Jul 28 04:25:03.375376 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:03.375312    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvtmt\" (UniqueName: \"kubernetes.io/projected/0208cc05-10d1-4d71-b063-24b462ebfa0b-kube-api-access-gvtmt\") pod \"maas-controller-7847888f58-gc6c8\" (UID: \"0208cc05-10d1-4d71-b063-24b462ebfa0b\") " pod="opendatahub/maas-controller-7847888f58-gc6c8"
Jul 28 04:25:03.476292 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:03.476256    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvtmt\" (UniqueName: \"kubernetes.io/projected/0208cc05-10d1-4d71-b063-24b462ebfa0b-kube-api-access-gvtmt\") pod \"maas-controller-7847888f58-gc6c8\" (UID: \"0208cc05-10d1-4d71-b063-24b462ebfa0b\") " pod="opendatahub/maas-controller-7847888f58-gc6c8"
Jul 28 04:25:03.476462 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:03.476308    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0208cc05-10d1-4d71-b063-24b462ebfa0b-webhook-cert\") pod \"maas-controller-7847888f58-gc6c8\" (UID: \"0208cc05-10d1-4d71-b063-24b462ebfa0b\") " pod="opendatahub/maas-controller-7847888f58-gc6c8"
Jul 28 04:25:03.478639 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:03.478611    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0208cc05-10d1-4d71-b063-24b462ebfa0b-webhook-cert\") pod \"maas-controller-7847888f58-gc6c8\" (UID: \"0208cc05-10d1-4d71-b063-24b462ebfa0b\") " pod="opendatahub/maas-controller-7847888f58-gc6c8"
Jul 28 04:25:03.484036 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:03.484011    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvtmt\" (UniqueName: \"kubernetes.io/projected/0208cc05-10d1-4d71-b063-24b462ebfa0b-kube-api-access-gvtmt\") pod \"maas-controller-7847888f58-gc6c8\" (UID: \"0208cc05-10d1-4d71-b063-24b462ebfa0b\") " pod="opendatahub/maas-controller-7847888f58-gc6c8"
Jul 28 04:25:03.541263 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:03.541232    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7847888f58-gc6c8"
Jul 28 04:25:03.668260 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:03.668220    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7847888f58-gc6c8"]
Jul 28 04:25:03.670631 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:25:03.670591    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0208cc05_10d1_4d71_b063_24b462ebfa0b.slice/crio-5b90a34098eaaf3f2d057e26aec75cbf07689ee892e0f4951a77a1071a1c8345 WatchSource:0}: Error finding container 5b90a34098eaaf3f2d057e26aec75cbf07689ee892e0f4951a77a1071a1c8345: Status 404 returned error can't find the container with id 5b90a34098eaaf3f2d057e26aec75cbf07689ee892e0f4951a77a1071a1c8345
Jul 28 04:25:03.779134 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:03.779093    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92313a5c-1dda-464a-b36d-75ac6509e13f-webhook-cert\") pod \"maas-controller-fdb466f56-wtq8f\" (UID: \"92313a5c-1dda-464a-b36d-75ac6509e13f\") " pod="opendatahub/maas-controller-fdb466f56-wtq8f"
Jul 28 04:25:03.781478 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:03.781453    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92313a5c-1dda-464a-b36d-75ac6509e13f-webhook-cert\") pod \"maas-controller-fdb466f56-wtq8f\" (UID: \"92313a5c-1dda-464a-b36d-75ac6509e13f\") " pod="opendatahub/maas-controller-fdb466f56-wtq8f"
Jul 28 04:25:03.960709 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:03.960619    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fdb466f56-wtq8f"
Jul 28 04:25:04.090198 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:04.090172    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-fdb466f56-wtq8f"]
Jul 28 04:25:04.091850 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:25:04.091815    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92313a5c_1dda_464a_b36d_75ac6509e13f.slice/crio-973721583a1138aeb4120b246bc36453b36894bd358148be4ca3391bae6b873b WatchSource:0}: Error finding container 973721583a1138aeb4120b246bc36453b36894bd358148be4ca3391bae6b873b: Status 404 returned error can't find the container with id 973721583a1138aeb4120b246bc36453b36894bd358148be4ca3391bae6b873b
Jul 28 04:25:04.302011 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:04.301974    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7847888f58-gc6c8" event={"ID":"0208cc05-10d1-4d71-b063-24b462ebfa0b","Type":"ContainerStarted","Data":"5b90a34098eaaf3f2d057e26aec75cbf07689ee892e0f4951a77a1071a1c8345"}
Jul 28 04:25:04.303045 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:04.303015    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fdb466f56-wtq8f" event={"ID":"92313a5c-1dda-464a-b36d-75ac6509e13f","Type":"ContainerStarted","Data":"973721583a1138aeb4120b246bc36453b36894bd358148be4ca3391bae6b873b"}
Jul 28 04:25:07.320927 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:07.320887    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7847888f58-gc6c8" event={"ID":"0208cc05-10d1-4d71-b063-24b462ebfa0b","Type":"ContainerStarted","Data":"be6fd6c0492058a8e675dd9289920994f62a1db1b4483488efc6e46eaec3a123"}
Jul 28 04:25:07.321480 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:07.320973    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7847888f58-gc6c8"
Jul 28 04:25:07.322400 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:07.322377    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fdb466f56-wtq8f" event={"ID":"92313a5c-1dda-464a-b36d-75ac6509e13f","Type":"ContainerStarted","Data":"346563167b4a3fb92c60f047e96bbb8a3f83a30408e75b552954183bfa814184"}
Jul 28 04:25:07.322518 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:07.322504    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-fdb466f56-wtq8f"
Jul 28 04:25:07.340411 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:07.340350    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7847888f58-gc6c8" podStartSLOduration=1.5201604199999998 podStartE2EDuration="4.340330301s" totalImagesPullingTime="2.820169881s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:25:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:25:07.340102647 +0000 UTC m=+807.730278669" watchObservedRunningTime="2026-07-28 04:25:07.340330301 +0000 UTC m=+807.730506325"
Jul 28 04:25:07.357510 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:07.357447    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-fdb466f56-wtq8f" podStartSLOduration=1.95148886 podStartE2EDuration="4.357425985s" totalImagesPullingTime="2.405937125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:25:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:25:07.355419585 +0000 UTC m=+807.745595617" watchObservedRunningTime="2026-07-28 04:25:07.357425985 +0000 UTC m=+807.747602007"
Jul 28 04:25:18.331484 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:18.331451    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7847888f58-gc6c8"
Jul 28 04:25:18.331977 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:18.331614    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-fdb466f56-wtq8f"
Jul 28 04:25:18.389129 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:18.389090    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-fdb466f56-wtq8f"]
Jul 28 04:25:18.389385 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:18.389359    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-fdb466f56-wtq8f" podUID="92313a5c-1dda-464a-b36d-75ac6509e13f" containerName="manager" containerID="cri-o://346563167b4a3fb92c60f047e96bbb8a3f83a30408e75b552954183bfa814184" gracePeriod=30
Jul 28 04:25:18.632899 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:18.632875    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fdb466f56-wtq8f"
Jul 28 04:25:18.713171 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:18.713142    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwsrx\" (UniqueName: \"kubernetes.io/projected/92313a5c-1dda-464a-b36d-75ac6509e13f-kube-api-access-qwsrx\") pod \"92313a5c-1dda-464a-b36d-75ac6509e13f\" (UID: \"92313a5c-1dda-464a-b36d-75ac6509e13f\") "
Jul 28 04:25:18.713171 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:18.713177    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92313a5c-1dda-464a-b36d-75ac6509e13f-webhook-cert\") pod \"92313a5c-1dda-464a-b36d-75ac6509e13f\" (UID: \"92313a5c-1dda-464a-b36d-75ac6509e13f\") "
Jul 28 04:25:18.715285 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:18.715256    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92313a5c-1dda-464a-b36d-75ac6509e13f-kube-api-access-qwsrx" (OuterVolumeSpecName: "kube-api-access-qwsrx") pod "92313a5c-1dda-464a-b36d-75ac6509e13f" (UID: "92313a5c-1dda-464a-b36d-75ac6509e13f"). InnerVolumeSpecName "kube-api-access-qwsrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:25:18.715396 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:18.715317    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92313a5c-1dda-464a-b36d-75ac6509e13f-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "92313a5c-1dda-464a-b36d-75ac6509e13f" (UID: "92313a5c-1dda-464a-b36d-75ac6509e13f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 04:25:18.814595 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:18.814563    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwsrx\" (UniqueName: \"kubernetes.io/projected/92313a5c-1dda-464a-b36d-75ac6509e13f-kube-api-access-qwsrx\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:25:18.814595 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:18.814590    2566 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/92313a5c-1dda-464a-b36d-75ac6509e13f-webhook-cert\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:25:19.382484 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:19.382451    2566 generic.go:356] "Generic (PLEG): container finished" podID="92313a5c-1dda-464a-b36d-75ac6509e13f" containerID="346563167b4a3fb92c60f047e96bbb8a3f83a30408e75b552954183bfa814184" exitCode=0
Jul 28 04:25:19.382902 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:19.382496    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-fdb466f56-wtq8f"
Jul 28 04:25:19.382902 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:19.382526    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fdb466f56-wtq8f" event={"ID":"92313a5c-1dda-464a-b36d-75ac6509e13f","Type":"ContainerDied","Data":"346563167b4a3fb92c60f047e96bbb8a3f83a30408e75b552954183bfa814184"}
Jul 28 04:25:19.382902 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:19.382579    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-fdb466f56-wtq8f" event={"ID":"92313a5c-1dda-464a-b36d-75ac6509e13f","Type":"ContainerDied","Data":"973721583a1138aeb4120b246bc36453b36894bd358148be4ca3391bae6b873b"}
Jul 28 04:25:19.382902 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:19.382601    2566 scope.go:117] "RemoveContainer" containerID="346563167b4a3fb92c60f047e96bbb8a3f83a30408e75b552954183bfa814184"
Jul 28 04:25:19.391815 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:19.391702    2566 scope.go:117] "RemoveContainer" containerID="346563167b4a3fb92c60f047e96bbb8a3f83a30408e75b552954183bfa814184"
Jul 28 04:25:19.392025 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:25:19.392001    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"346563167b4a3fb92c60f047e96bbb8a3f83a30408e75b552954183bfa814184\": container with ID starting with 346563167b4a3fb92c60f047e96bbb8a3f83a30408e75b552954183bfa814184 not found: ID does not exist" containerID="346563167b4a3fb92c60f047e96bbb8a3f83a30408e75b552954183bfa814184"
Jul 28 04:25:19.392076 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:19.392034    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"346563167b4a3fb92c60f047e96bbb8a3f83a30408e75b552954183bfa814184"} err="failed to get container status \"346563167b4a3fb92c60f047e96bbb8a3f83a30408e75b552954183bfa814184\": rpc error: code = NotFound desc = could not find container \"346563167b4a3fb92c60f047e96bbb8a3f83a30408e75b552954183bfa814184\": container with ID starting with 346563167b4a3fb92c60f047e96bbb8a3f83a30408e75b552954183bfa814184 not found: ID does not exist"
Jul 28 04:25:19.403238 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:19.403216    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-fdb466f56-wtq8f"]
Jul 28 04:25:19.407228 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:19.407203    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-fdb466f56-wtq8f"]
Jul 28 04:25:20.088084 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:20.088051    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92313a5c-1dda-464a-b36d-75ac6509e13f" path="/var/lib/kubelet/pods/92313a5c-1dda-464a-b36d-75ac6509e13f/volumes"
Jul 28 04:25:40.530894 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:40.530853    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-kncv4"]
Jul 28 04:25:40.531332 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:40.531191    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92313a5c-1dda-464a-b36d-75ac6509e13f" containerName="manager"
Jul 28 04:25:40.531332 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:40.531201    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="92313a5c-1dda-464a-b36d-75ac6509e13f" containerName="manager"
Jul 28 04:25:40.531332 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:40.531259    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="92313a5c-1dda-464a-b36d-75ac6509e13f" containerName="manager"
Jul 28 04:25:40.535422 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:40.535400    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-kncv4"
Jul 28 04:25:40.537927 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:40.537901    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 28 04:25:40.541868 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:40.538413    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-hvzdq\""
Jul 28 04:25:40.546609 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:40.546583    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-kncv4"]
Jul 28 04:25:40.716341 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:40.716310    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmg6r\" (UniqueName: \"kubernetes.io/projected/fc1e8560-ba15-4aeb-8b1c-760913971991-kube-api-access-gmg6r\") pod \"payload-pre-processing-687dbb6cf6-kncv4\" (UID: \"fc1e8560-ba15-4aeb-8b1c-760913971991\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-kncv4"
Jul 28 04:25:40.716341 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:40.716349    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc1e8560-ba15-4aeb-8b1c-760913971991-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-kncv4\" (UID: \"fc1e8560-ba15-4aeb-8b1c-760913971991\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-kncv4"
Jul 28 04:25:40.817238 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:40.817143    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmg6r\" (UniqueName: \"kubernetes.io/projected/fc1e8560-ba15-4aeb-8b1c-760913971991-kube-api-access-gmg6r\") pod \"payload-pre-processing-687dbb6cf6-kncv4\" (UID: \"fc1e8560-ba15-4aeb-8b1c-760913971991\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-kncv4"
Jul 28 04:25:40.817238 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:40.817187    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc1e8560-ba15-4aeb-8b1c-760913971991-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-kncv4\" (UID: \"fc1e8560-ba15-4aeb-8b1c-760913971991\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-kncv4"
Jul 28 04:25:40.817871 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:40.817852    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc1e8560-ba15-4aeb-8b1c-760913971991-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-kncv4\" (UID: \"fc1e8560-ba15-4aeb-8b1c-760913971991\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-kncv4"
Jul 28 04:25:40.824850 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:40.824813    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmg6r\" (UniqueName: \"kubernetes.io/projected/fc1e8560-ba15-4aeb-8b1c-760913971991-kube-api-access-gmg6r\") pod \"payload-pre-processing-687dbb6cf6-kncv4\" (UID: \"fc1e8560-ba15-4aeb-8b1c-760913971991\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-kncv4"
Jul 28 04:25:40.852063 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:40.852036    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-kncv4"
Jul 28 04:25:40.984990 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:40.984962    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-kncv4"]
Jul 28 04:25:40.987058 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:25:40.987031    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc1e8560_ba15_4aeb_8b1c_760913971991.slice/crio-83daaaf3cf4670d5889ea2b5710e1b1885f442c5f603ee379da13bcc08c09c64 WatchSource:0}: Error finding container 83daaaf3cf4670d5889ea2b5710e1b1885f442c5f603ee379da13bcc08c09c64: Status 404 returned error can't find the container with id 83daaaf3cf4670d5889ea2b5710e1b1885f442c5f603ee379da13bcc08c09c64
Jul 28 04:25:41.487758 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:41.487721    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-kncv4" event={"ID":"fc1e8560-ba15-4aeb-8b1c-760913971991","Type":"ContainerStarted","Data":"83daaaf3cf4670d5889ea2b5710e1b1885f442c5f603ee379da13bcc08c09c64"}
Jul 28 04:25:43.498934 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:43.498903    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-kncv4" event={"ID":"fc1e8560-ba15-4aeb-8b1c-760913971991","Type":"ContainerStarted","Data":"39828ae44d4402c454e98483e49cbe663a8ded89177ee33d0fed3aa0edb0eef2"}
Jul 28 04:25:43.499336 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:43.499022    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-kncv4"
Jul 28 04:25:43.514325 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:43.514278    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-kncv4" podStartSLOduration=1.135007352 podStartE2EDuration="3.514261786s" totalImagesPullingTime="2.379254434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:25:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:25:43.512526561 +0000 UTC m=+843.902702592" watchObservedRunningTime="2026-07-28 04:25:43.514261786 +0000 UTC m=+843.904437807"
Jul 28 04:25:54.506022 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:25:54.505992    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-kncv4"
Jul 28 04:26:09.674559 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:09.674518    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ngc2f"]
Jul 28 04:26:09.678601 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:09.678576    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ngc2f"
Jul 28 04:26:09.682272 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:09.682250    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 28 04:26:09.682389 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:09.682324    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 28 04:26:09.682389 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:09.682336    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 28 04:26:09.682505 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:09.682252    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-gmrkq\""
Jul 28 04:26:09.687260 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:09.687202    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ngc2f"]
Jul 28 04:26:09.765817 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:09.765786    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2c9cd70-f242-4eb1-8fa3-5ed2a9e9ad70-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ngc2f\" (UID: \"d2c9cd70-f242-4eb1-8fa3-5ed2a9e9ad70\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ngc2f"
Jul 28 04:26:09.765995 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:09.765860    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2c9cd70-f242-4eb1-8fa3-5ed2a9e9ad70-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ngc2f\" (UID: \"d2c9cd70-f242-4eb1-8fa3-5ed2a9e9ad70\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ngc2f"
Jul 28 04:26:09.765995 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:09.765898    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g66h4\" (UniqueName: \"kubernetes.io/projected/d2c9cd70-f242-4eb1-8fa3-5ed2a9e9ad70-kube-api-access-g66h4\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ngc2f\" (UID: \"d2c9cd70-f242-4eb1-8fa3-5ed2a9e9ad70\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ngc2f"
Jul 28 04:26:09.765995 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:09.765950    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2c9cd70-f242-4eb1-8fa3-5ed2a9e9ad70-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ngc2f\" (UID: \"d2c9cd70-f242-4eb1-8fa3-5ed2a9e9ad70\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ngc2f"
Jul 28 04:26:09.765995 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:09.765986    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2c9cd70-f242-4eb1-8fa3-5ed2a9e9ad70-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ngc2f\" (UID: \"d2c9cd70-f242-4eb1-8fa3-5ed2a9e9ad70\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ngc2f"
Jul 28 04:26:09.766146 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:09.766037    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2c9cd70-f242-4eb1-8fa3-5ed2a9e9ad70-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ngc2f\" (UID: \"d2c9cd70-f242-4eb1-8fa3-5ed2a9e9ad70\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ngc2f"
Jul 28 04:26:09.867363 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:09.867323    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2c9cd70-f242-4eb1-8fa3-5ed2a9e9ad70-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ngc2f\" (UID: \"d2c9cd70-f242-4eb1-8fa3-5ed2a9e9ad70\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ngc2f"
Jul 28 04:26:09.867363 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:09.867362    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g66h4\" (UniqueName: \"kubernetes.io/projected/d2c9cd70-f242-4eb1-8fa3-5ed2a9e9ad70-kube-api-access-g66h4\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ngc2f\" (UID: \"d2c9cd70-f242-4eb1-8fa3-5ed2a9e9ad70\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ngc2f"
Jul 28 04:26:09.867608 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:09.867384    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2c9cd70-f242-4eb1-8fa3-5ed2a9e9ad70-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ngc2f\" (UID: \"d2c9cd70-f242-4eb1-8fa3-5ed2a9e9ad70\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ngc2f"
Jul 28 04:26:09.867608 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:09.867413    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2c9cd70-f242-4eb1-8fa3-5ed2a9e9ad70-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ngc2f\" (UID: \"d2c9cd70-f242-4eb1-8fa3-5ed2a9e9ad70\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ngc2f"
Jul 28 04:26:09.867608 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:09.867468    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2c9cd70-f242-4eb1-8fa3-5ed2a9e9ad70-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ngc2f\" (UID: \"d2c9cd70-f242-4eb1-8fa3-5ed2a9e9ad70\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ngc2f"
Jul 28 04:26:09.867608 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:09.867507    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2c9cd70-f242-4eb1-8fa3-5ed2a9e9ad70-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ngc2f\" (UID: \"d2c9cd70-f242-4eb1-8fa3-5ed2a9e9ad70\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ngc2f"
Jul 28 04:26:09.867826 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:09.867800    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2c9cd70-f242-4eb1-8fa3-5ed2a9e9ad70-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ngc2f\" (UID: \"d2c9cd70-f242-4eb1-8fa3-5ed2a9e9ad70\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ngc2f"
Jul 28 04:26:09.868006 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:09.867981    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2c9cd70-f242-4eb1-8fa3-5ed2a9e9ad70-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ngc2f\" (UID: \"d2c9cd70-f242-4eb1-8fa3-5ed2a9e9ad70\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ngc2f"
Jul 28 04:26:09.868071 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:09.868003    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2c9cd70-f242-4eb1-8fa3-5ed2a9e9ad70-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ngc2f\" (UID: \"d2c9cd70-f242-4eb1-8fa3-5ed2a9e9ad70\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ngc2f"
Jul 28 04:26:09.869854 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:09.869817    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2c9cd70-f242-4eb1-8fa3-5ed2a9e9ad70-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ngc2f\" (UID: \"d2c9cd70-f242-4eb1-8fa3-5ed2a9e9ad70\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ngc2f"
Jul 28 04:26:09.869992 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:09.869976    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2c9cd70-f242-4eb1-8fa3-5ed2a9e9ad70-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ngc2f\" (UID: \"d2c9cd70-f242-4eb1-8fa3-5ed2a9e9ad70\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ngc2f"
Jul 28 04:26:09.874771 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:09.874752    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g66h4\" (UniqueName: \"kubernetes.io/projected/d2c9cd70-f242-4eb1-8fa3-5ed2a9e9ad70-kube-api-access-g66h4\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-ngc2f\" (UID: \"d2c9cd70-f242-4eb1-8fa3-5ed2a9e9ad70\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ngc2f"
Jul 28 04:26:09.989789 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:09.989690    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ngc2f"
Jul 28 04:26:10.128691 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:10.128667    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ngc2f"]
Jul 28 04:26:10.131852 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:26:10.131810    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2c9cd70_f242_4eb1_8fa3_5ed2a9e9ad70.slice/crio-2014f786a9ccf0978382c0a1900c35fbeba63922ce1a4d055339f0d0b12113d9 WatchSource:0}: Error finding container 2014f786a9ccf0978382c0a1900c35fbeba63922ce1a4d055339f0d0b12113d9: Status 404 returned error can't find the container with id 2014f786a9ccf0978382c0a1900c35fbeba63922ce1a4d055339f0d0b12113d9
Jul 28 04:26:10.628627 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:10.628588    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ngc2f" event={"ID":"d2c9cd70-f242-4eb1-8fa3-5ed2a9e9ad70","Type":"ContainerStarted","Data":"2014f786a9ccf0978382c0a1900c35fbeba63922ce1a4d055339f0d0b12113d9"}
Jul 28 04:26:11.634513 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:11.634428    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ngc2f" event={"ID":"d2c9cd70-f242-4eb1-8fa3-5ed2a9e9ad70","Type":"ContainerStarted","Data":"f58092ce70950b6d5d736051a48ba9721ea682438cae52117301cf5f3d1a9b42"}
Jul 28 04:26:11.652654 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:11.652608    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ngc2f" podStartSLOduration=1.505526167 podStartE2EDuration="2.652594419s" totalImagesPullingTime="1.147068252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:26:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:26:11.651869344 +0000 UTC m=+872.042045574" watchObservedRunningTime="2026-07-28 04:26:11.652594419 +0000 UTC m=+872.042770440"
Jul 28 04:26:19.990817 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:19.990779    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ngc2f"
Jul 28 04:26:19.991227 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:19.990876    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ngc2f"
Jul 28 04:26:20.003896 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:20.003865    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ngc2f"
Jul 28 04:26:20.684606 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:20.684576    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-ngc2f"
Jul 28 04:26:22.184175 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:22.184142    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-qqcd4"]
Jul 28 04:26:22.192718 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:22.192692    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qqcd4"
Jul 28 04:26:22.196425 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:22.196401    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 28 04:26:22.197862 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:22.197822    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-qqcd4"]
Jul 28 04:26:22.284353 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:22.284315    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc87f1c9-5ac5-41ff-b653-701ce5e0b760-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qqcd4\" (UID: \"dc87f1c9-5ac5-41ff-b653-701ce5e0b760\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qqcd4"
Jul 28 04:26:22.284514 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:22.284372    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc87f1c9-5ac5-41ff-b653-701ce5e0b760-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qqcd4\" (UID: \"dc87f1c9-5ac5-41ff-b653-701ce5e0b760\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qqcd4"
Jul 28 04:26:22.284514 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:22.284413    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc87f1c9-5ac5-41ff-b653-701ce5e0b760-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qqcd4\" (UID: \"dc87f1c9-5ac5-41ff-b653-701ce5e0b760\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qqcd4"
Jul 28 04:26:22.284514 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:22.284448    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc87f1c9-5ac5-41ff-b653-701ce5e0b760-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qqcd4\" (UID: \"dc87f1c9-5ac5-41ff-b653-701ce5e0b760\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qqcd4"
Jul 28 04:26:22.284514 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:22.284499    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc87f1c9-5ac5-41ff-b653-701ce5e0b760-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qqcd4\" (UID: \"dc87f1c9-5ac5-41ff-b653-701ce5e0b760\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qqcd4"
Jul 28 04:26:22.284696 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:22.284535    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8zzx\" (UniqueName: \"kubernetes.io/projected/dc87f1c9-5ac5-41ff-b653-701ce5e0b760-kube-api-access-r8zzx\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qqcd4\" (UID: \"dc87f1c9-5ac5-41ff-b653-701ce5e0b760\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qqcd4"
Jul 28 04:26:22.385322 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:22.385284    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc87f1c9-5ac5-41ff-b653-701ce5e0b760-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qqcd4\" (UID: \"dc87f1c9-5ac5-41ff-b653-701ce5e0b760\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qqcd4"
Jul 28 04:26:22.385502 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:22.385339    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc87f1c9-5ac5-41ff-b653-701ce5e0b760-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qqcd4\" (UID: \"dc87f1c9-5ac5-41ff-b653-701ce5e0b760\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qqcd4"
Jul 28 04:26:22.385502 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:22.385362    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc87f1c9-5ac5-41ff-b653-701ce5e0b760-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qqcd4\" (UID: \"dc87f1c9-5ac5-41ff-b653-701ce5e0b760\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qqcd4"
Jul 28 04:26:22.385502 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:22.385396    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc87f1c9-5ac5-41ff-b653-701ce5e0b760-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qqcd4\" (UID: \"dc87f1c9-5ac5-41ff-b653-701ce5e0b760\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qqcd4"
Jul 28 04:26:22.385502 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:22.385416    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc87f1c9-5ac5-41ff-b653-701ce5e0b760-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qqcd4\" (UID: \"dc87f1c9-5ac5-41ff-b653-701ce5e0b760\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qqcd4"
Jul 28 04:26:22.385502 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:22.385438    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8zzx\" (UniqueName: \"kubernetes.io/projected/dc87f1c9-5ac5-41ff-b653-701ce5e0b760-kube-api-access-r8zzx\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qqcd4\" (UID: \"dc87f1c9-5ac5-41ff-b653-701ce5e0b760\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qqcd4"
Jul 28 04:26:22.385774 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:22.385752    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dc87f1c9-5ac5-41ff-b653-701ce5e0b760-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qqcd4\" (UID: \"dc87f1c9-5ac5-41ff-b653-701ce5e0b760\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qqcd4"
Jul 28 04:26:22.385862 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:22.385769    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc87f1c9-5ac5-41ff-b653-701ce5e0b760-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qqcd4\" (UID: \"dc87f1c9-5ac5-41ff-b653-701ce5e0b760\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qqcd4"
Jul 28 04:26:22.385862 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:22.385808    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dc87f1c9-5ac5-41ff-b653-701ce5e0b760-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qqcd4\" (UID: \"dc87f1c9-5ac5-41ff-b653-701ce5e0b760\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qqcd4"
Jul 28 04:26:22.387708 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:22.387685    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dc87f1c9-5ac5-41ff-b653-701ce5e0b760-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qqcd4\" (UID: \"dc87f1c9-5ac5-41ff-b653-701ce5e0b760\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qqcd4"
Jul 28 04:26:22.387865 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:22.387831    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dc87f1c9-5ac5-41ff-b653-701ce5e0b760-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qqcd4\" (UID: \"dc87f1c9-5ac5-41ff-b653-701ce5e0b760\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qqcd4"
Jul 28 04:26:22.392786 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:22.392760    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8zzx\" (UniqueName: \"kubernetes.io/projected/dc87f1c9-5ac5-41ff-b653-701ce5e0b760-kube-api-access-r8zzx\") pod \"e2e-trlp-test-simulated-kserve-598466548d-qqcd4\" (UID: \"dc87f1c9-5ac5-41ff-b653-701ce5e0b760\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qqcd4"
Jul 28 04:26:22.505636 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:22.505547    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qqcd4"
Jul 28 04:26:22.644513 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:22.644482    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-qqcd4"]
Jul 28 04:26:22.646077 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:26:22.646049    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc87f1c9_5ac5_41ff_b653_701ce5e0b760.slice/crio-0b20dbc2db2bf7b2cf943a9baded3589292732f3aa293e15c80ec3802958d8af WatchSource:0}: Error finding container 0b20dbc2db2bf7b2cf943a9baded3589292732f3aa293e15c80ec3802958d8af: Status 404 returned error can't find the container with id 0b20dbc2db2bf7b2cf943a9baded3589292732f3aa293e15c80ec3802958d8af
Jul 28 04:26:22.683859 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:22.683813    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qqcd4" event={"ID":"dc87f1c9-5ac5-41ff-b653-701ce5e0b760","Type":"ContainerStarted","Data":"0b20dbc2db2bf7b2cf943a9baded3589292732f3aa293e15c80ec3802958d8af"}
Jul 28 04:26:23.695270 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:23.695231    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qqcd4" event={"ID":"dc87f1c9-5ac5-41ff-b653-701ce5e0b760","Type":"ContainerStarted","Data":"4ffe673e4cda5662efb50f90ad3c4a9f9d739e20b72d3861d48acfb6bfca3de6"}
Jul 28 04:26:23.715069 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:23.714991    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qqcd4" podStartSLOduration=1.5307292239999999 podStartE2EDuration="1.714970926s" totalImagesPullingTime="184.241702ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:26:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:26:23.714411011 +0000 UTC m=+884.104587033" watchObservedRunningTime="2026-07-28 04:26:23.714970926 +0000 UTC m=+884.105146947"
Jul 28 04:26:32.506678 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:32.506598    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qqcd4"
Jul 28 04:26:32.506678 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:32.506677    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qqcd4"
Jul 28 04:26:32.519289 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:32.519265    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qqcd4"
Jul 28 04:26:32.748547 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:32.748521    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-qqcd4"
Jul 28 04:26:35.585652 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:35.585613    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-tdmg6"]
Jul 28 04:26:35.589594 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:35.589570    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdmg6"
Jul 28 04:26:35.592407 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:35.592383    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 28 04:26:35.599618 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:35.599586    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-tdmg6"]
Jul 28 04:26:35.690590 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:35.690552    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/173039f3-efce-40f1-b977-4a0c921c0bfb-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tdmg6\" (UID: \"173039f3-efce-40f1-b977-4a0c921c0bfb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdmg6"
Jul 28 04:26:35.690590 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:35.690591    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/173039f3-efce-40f1-b977-4a0c921c0bfb-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tdmg6\" (UID: \"173039f3-efce-40f1-b977-4a0c921c0bfb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdmg6"
Jul 28 04:26:35.690800 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:35.690621    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/173039f3-efce-40f1-b977-4a0c921c0bfb-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tdmg6\" (UID: \"173039f3-efce-40f1-b977-4a0c921c0bfb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdmg6"
Jul 28 04:26:35.690800 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:35.690674    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/173039f3-efce-40f1-b977-4a0c921c0bfb-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tdmg6\" (UID: \"173039f3-efce-40f1-b977-4a0c921c0bfb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdmg6"
Jul 28 04:26:35.690800 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:35.690717    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/173039f3-efce-40f1-b977-4a0c921c0bfb-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tdmg6\" (UID: \"173039f3-efce-40f1-b977-4a0c921c0bfb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdmg6"
Jul 28 04:26:35.690800 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:35.690741    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwvp6\" (UniqueName: \"kubernetes.io/projected/173039f3-efce-40f1-b977-4a0c921c0bfb-kube-api-access-hwvp6\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tdmg6\" (UID: \"173039f3-efce-40f1-b977-4a0c921c0bfb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdmg6"
Jul 28 04:26:35.791534 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:35.791493    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/173039f3-efce-40f1-b977-4a0c921c0bfb-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tdmg6\" (UID: \"173039f3-efce-40f1-b977-4a0c921c0bfb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdmg6"
Jul 28 04:26:35.791534 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:35.791539    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/173039f3-efce-40f1-b977-4a0c921c0bfb-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tdmg6\" (UID: \"173039f3-efce-40f1-b977-4a0c921c0bfb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdmg6"
Jul 28 04:26:35.791767 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:35.791667    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/173039f3-efce-40f1-b977-4a0c921c0bfb-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tdmg6\" (UID: \"173039f3-efce-40f1-b977-4a0c921c0bfb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdmg6"
Jul 28 04:26:35.791767 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:35.791739    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/173039f3-efce-40f1-b977-4a0c921c0bfb-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tdmg6\" (UID: \"173039f3-efce-40f1-b977-4a0c921c0bfb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdmg6"
Jul 28 04:26:35.791899 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:35.791816    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/173039f3-efce-40f1-b977-4a0c921c0bfb-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tdmg6\" (UID: \"173039f3-efce-40f1-b977-4a0c921c0bfb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdmg6"
Jul 28 04:26:35.791960 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:35.791906    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwvp6\" (UniqueName: \"kubernetes.io/projected/173039f3-efce-40f1-b977-4a0c921c0bfb-kube-api-access-hwvp6\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tdmg6\" (UID: \"173039f3-efce-40f1-b977-4a0c921c0bfb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdmg6"
Jul 28 04:26:35.792071 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:35.792052    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/173039f3-efce-40f1-b977-4a0c921c0bfb-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tdmg6\" (UID: \"173039f3-efce-40f1-b977-4a0c921c0bfb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdmg6"
Jul 28 04:26:35.792172 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:35.792149    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/173039f3-efce-40f1-b977-4a0c921c0bfb-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tdmg6\" (UID: \"173039f3-efce-40f1-b977-4a0c921c0bfb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdmg6"
Jul 28 04:26:35.792239 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:35.792172    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/173039f3-efce-40f1-b977-4a0c921c0bfb-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tdmg6\" (UID: \"173039f3-efce-40f1-b977-4a0c921c0bfb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdmg6"
Jul 28 04:26:35.793961 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:35.793936    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/173039f3-efce-40f1-b977-4a0c921c0bfb-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tdmg6\" (UID: \"173039f3-efce-40f1-b977-4a0c921c0bfb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdmg6"
Jul 28 04:26:35.794240 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:35.794224    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/173039f3-efce-40f1-b977-4a0c921c0bfb-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tdmg6\" (UID: \"173039f3-efce-40f1-b977-4a0c921c0bfb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdmg6"
Jul 28 04:26:35.801666 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:35.801634    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwvp6\" (UniqueName: \"kubernetes.io/projected/173039f3-efce-40f1-b977-4a0c921c0bfb-kube-api-access-hwvp6\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-tdmg6\" (UID: \"173039f3-efce-40f1-b977-4a0c921c0bfb\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdmg6"
Jul 28 04:26:35.901339 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:35.901250    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdmg6"
Jul 28 04:26:36.039643 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:36.039610    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-tdmg6"]
Jul 28 04:26:36.041404 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:26:36.041374    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod173039f3_efce_40f1_b977_4a0c921c0bfb.slice/crio-c414a76a73f4b053b0de9aaa076a20dc08446d65f9e43fa25b90554353649d0e WatchSource:0}: Error finding container c414a76a73f4b053b0de9aaa076a20dc08446d65f9e43fa25b90554353649d0e: Status 404 returned error can't find the container with id c414a76a73f4b053b0de9aaa076a20dc08446d65f9e43fa25b90554353649d0e
Jul 28 04:26:36.756956 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:36.756918    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdmg6" event={"ID":"173039f3-efce-40f1-b977-4a0c921c0bfb","Type":"ContainerStarted","Data":"2ea4dfffe13d56f575915dda52fa9b62a34ffe1ce10fecfa69b2e18ca1516aab"}
Jul 28 04:26:36.757429 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:36.756971    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdmg6" event={"ID":"173039f3-efce-40f1-b977-4a0c921c0bfb","Type":"ContainerStarted","Data":"c414a76a73f4b053b0de9aaa076a20dc08446d65f9e43fa25b90554353649d0e"}
Jul 28 04:26:36.777184 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:36.777131    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdmg6" podStartSLOduration=1.598489206 podStartE2EDuration="1.7771164s" totalImagesPullingTime="178.627194ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:26:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:26:36.776827752 +0000 UTC m=+897.167003774" watchObservedRunningTime="2026-07-28 04:26:36.7771164 +0000 UTC m=+897.167292421"
Jul 28 04:26:43.574034 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:43.573923    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4glpr_bfd2d3d8-f287-42d8-83c7-f82185c96735/console-operator/2.log"
Jul 28 04:26:43.578053 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:43.575214    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4glpr_bfd2d3d8-f287-42d8-83c7-f82185c96735/console-operator/2.log"
Jul 28 04:26:45.901626 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:45.901581    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdmg6"
Jul 28 04:26:45.902145 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:45.901668    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdmg6"
Jul 28 04:26:45.913946 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:45.913916    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdmg6"
Jul 28 04:26:46.815297 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:46.815272    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-tdmg6"
Jul 28 04:26:49.992465 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:49.992428    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-c4h7c"]
Jul 28 04:26:49.996586 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:49.996565    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-c4h7c"
Jul 28 04:26:49.999136 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:49.999118    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 28 04:26:50.006213 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:50.006191    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-c4h7c"]
Jul 28 04:26:50.123504 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:50.123474    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd436090-b85d-4d4a-af2e-dbacf3e3c3e4-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-c4h7c\" (UID: \"cd436090-b85d-4d4a-af2e-dbacf3e3c3e4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-c4h7c"
Jul 28 04:26:50.123671 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:50.123514    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd436090-b85d-4d4a-af2e-dbacf3e3c3e4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-c4h7c\" (UID: \"cd436090-b85d-4d4a-af2e-dbacf3e3c3e4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-c4h7c"
Jul 28 04:26:50.123671 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:50.123552    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd436090-b85d-4d4a-af2e-dbacf3e3c3e4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-c4h7c\" (UID: \"cd436090-b85d-4d4a-af2e-dbacf3e3c3e4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-c4h7c"
Jul 28 04:26:50.123746 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:50.123670    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6pst\" (UniqueName: \"kubernetes.io/projected/cd436090-b85d-4d4a-af2e-dbacf3e3c3e4-kube-api-access-s6pst\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-c4h7c\" (UID: \"cd436090-b85d-4d4a-af2e-dbacf3e3c3e4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-c4h7c"
Jul 28 04:26:50.123746 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:50.123711    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd436090-b85d-4d4a-af2e-dbacf3e3c3e4-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-c4h7c\" (UID: \"cd436090-b85d-4d4a-af2e-dbacf3e3c3e4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-c4h7c"
Jul 28 04:26:50.123746 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:50.123731    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd436090-b85d-4d4a-af2e-dbacf3e3c3e4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-c4h7c\" (UID: \"cd436090-b85d-4d4a-af2e-dbacf3e3c3e4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-c4h7c"
Jul 28 04:26:50.224852 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:50.224804    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd436090-b85d-4d4a-af2e-dbacf3e3c3e4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-c4h7c\" (UID: \"cd436090-b85d-4d4a-af2e-dbacf3e3c3e4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-c4h7c"
Jul 28 04:26:50.225547 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:50.225132    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6pst\" (UniqueName: \"kubernetes.io/projected/cd436090-b85d-4d4a-af2e-dbacf3e3c3e4-kube-api-access-s6pst\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-c4h7c\" (UID: \"cd436090-b85d-4d4a-af2e-dbacf3e3c3e4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-c4h7c"
Jul 28 04:26:50.225547 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:50.225230    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd436090-b85d-4d4a-af2e-dbacf3e3c3e4-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-c4h7c\" (UID: \"cd436090-b85d-4d4a-af2e-dbacf3e3c3e4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-c4h7c"
Jul 28 04:26:50.225547 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:50.225258    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd436090-b85d-4d4a-af2e-dbacf3e3c3e4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-c4h7c\" (UID: \"cd436090-b85d-4d4a-af2e-dbacf3e3c3e4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-c4h7c"
Jul 28 04:26:50.225547 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:50.225298    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd436090-b85d-4d4a-af2e-dbacf3e3c3e4-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-c4h7c\" (UID: \"cd436090-b85d-4d4a-af2e-dbacf3e3c3e4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-c4h7c"
Jul 28 04:26:50.225547 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:50.225362    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd436090-b85d-4d4a-af2e-dbacf3e3c3e4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-c4h7c\" (UID: \"cd436090-b85d-4d4a-af2e-dbacf3e3c3e4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-c4h7c"
Jul 28 04:26:50.225954 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:50.225657    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd436090-b85d-4d4a-af2e-dbacf3e3c3e4-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-c4h7c\" (UID: \"cd436090-b85d-4d4a-af2e-dbacf3e3c3e4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-c4h7c"
Jul 28 04:26:50.225954 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:50.225685    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd436090-b85d-4d4a-af2e-dbacf3e3c3e4-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-c4h7c\" (UID: \"cd436090-b85d-4d4a-af2e-dbacf3e3c3e4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-c4h7c"
Jul 28 04:26:50.225954 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:50.225794    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd436090-b85d-4d4a-af2e-dbacf3e3c3e4-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-c4h7c\" (UID: \"cd436090-b85d-4d4a-af2e-dbacf3e3c3e4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-c4h7c"
Jul 28 04:26:50.227411 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:50.227391    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd436090-b85d-4d4a-af2e-dbacf3e3c3e4-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-c4h7c\" (UID: \"cd436090-b85d-4d4a-af2e-dbacf3e3c3e4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-c4h7c"
Jul 28 04:26:50.227560 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:50.227541    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd436090-b85d-4d4a-af2e-dbacf3e3c3e4-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-c4h7c\" (UID: \"cd436090-b85d-4d4a-af2e-dbacf3e3c3e4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-c4h7c"
Jul 28 04:26:50.232722 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:50.232700    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6pst\" (UniqueName: \"kubernetes.io/projected/cd436090-b85d-4d4a-af2e-dbacf3e3c3e4-kube-api-access-s6pst\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-c4h7c\" (UID: \"cd436090-b85d-4d4a-af2e-dbacf3e3c3e4\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-c4h7c"
Jul 28 04:26:50.311663 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:50.311632    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-c4h7c"
Jul 28 04:26:50.445947 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:50.445726    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-c4h7c"]
Jul 28 04:26:50.448104 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:26:50.448073    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd436090_b85d_4d4a_af2e_dbacf3e3c3e4.slice/crio-6dc73035937068320845d78065ea26147fc94ae23adea3d0b8244924d5baa100 WatchSource:0}: Error finding container 6dc73035937068320845d78065ea26147fc94ae23adea3d0b8244924d5baa100: Status 404 returned error can't find the container with id 6dc73035937068320845d78065ea26147fc94ae23adea3d0b8244924d5baa100
Jul 28 04:26:50.824082 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:50.824041    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-c4h7c" event={"ID":"cd436090-b85d-4d4a-af2e-dbacf3e3c3e4","Type":"ContainerStarted","Data":"ebbe1ac9da457791c9e3d064ee6cf68c40cb839c165c411b693ae7653d9b097e"}
Jul 28 04:26:50.824082 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:50.824093    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-c4h7c" event={"ID":"cd436090-b85d-4d4a-af2e-dbacf3e3c3e4","Type":"ContainerStarted","Data":"6dc73035937068320845d78065ea26147fc94ae23adea3d0b8244924d5baa100"}
Jul 28 04:26:50.841462 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:26:50.841413    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-c4h7c" podStartSLOduration=1.6540871510000001 podStartE2EDuration="1.841397118s" totalImagesPullingTime="187.309967ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:26:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:26:50.840097411 +0000 UTC m=+911.230273443" watchObservedRunningTime="2026-07-28 04:26:50.841397118 +0000 UTC m=+911.231573138"
Jul 28 04:27:00.312717 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:27:00.312677    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-c4h7c"
Jul 28 04:27:00.312717 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:27:00.312725    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-c4h7c"
Jul 28 04:27:00.325551 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:27:00.325523    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-c4h7c"
Jul 28 04:27:00.886612 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:27:00.886581    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-c4h7c"
Jul 28 04:29:24.485595 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:29:24.485558    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7847888f58-gc6c8"]
Jul 28 04:29:24.486160 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:29:24.485797    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7847888f58-gc6c8" podUID="0208cc05-10d1-4d71-b063-24b462ebfa0b" containerName="manager" containerID="cri-o://be6fd6c0492058a8e675dd9289920994f62a1db1b4483488efc6e46eaec3a123" gracePeriod=30
Jul 28 04:29:24.738244 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:29:24.738185    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7847888f58-gc6c8"
Jul 28 04:29:24.916277 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:29:24.916242    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0208cc05-10d1-4d71-b063-24b462ebfa0b-webhook-cert\") pod \"0208cc05-10d1-4d71-b063-24b462ebfa0b\" (UID: \"0208cc05-10d1-4d71-b063-24b462ebfa0b\") "
Jul 28 04:29:24.916473 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:29:24.916312    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvtmt\" (UniqueName: \"kubernetes.io/projected/0208cc05-10d1-4d71-b063-24b462ebfa0b-kube-api-access-gvtmt\") pod \"0208cc05-10d1-4d71-b063-24b462ebfa0b\" (UID: \"0208cc05-10d1-4d71-b063-24b462ebfa0b\") "
Jul 28 04:29:24.918358 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:29:24.918326    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0208cc05-10d1-4d71-b063-24b462ebfa0b-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "0208cc05-10d1-4d71-b063-24b462ebfa0b" (UID: "0208cc05-10d1-4d71-b063-24b462ebfa0b"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 04:29:24.918499 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:29:24.918474    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0208cc05-10d1-4d71-b063-24b462ebfa0b-kube-api-access-gvtmt" (OuterVolumeSpecName: "kube-api-access-gvtmt") pod "0208cc05-10d1-4d71-b063-24b462ebfa0b" (UID: "0208cc05-10d1-4d71-b063-24b462ebfa0b"). InnerVolumeSpecName "kube-api-access-gvtmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:29:25.017642 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:29:25.017610    2566 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0208cc05-10d1-4d71-b063-24b462ebfa0b-webhook-cert\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:29:25.017642 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:29:25.017637    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvtmt\" (UniqueName: \"kubernetes.io/projected/0208cc05-10d1-4d71-b063-24b462ebfa0b-kube-api-access-gvtmt\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:29:25.554379 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:29:25.554346    2566 generic.go:356] "Generic (PLEG): container finished" podID="0208cc05-10d1-4d71-b063-24b462ebfa0b" containerID="be6fd6c0492058a8e675dd9289920994f62a1db1b4483488efc6e46eaec3a123" exitCode=0
Jul 28 04:29:25.554814 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:29:25.554386    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7847888f58-gc6c8"
Jul 28 04:29:25.554814 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:29:25.554412    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7847888f58-gc6c8" event={"ID":"0208cc05-10d1-4d71-b063-24b462ebfa0b","Type":"ContainerDied","Data":"be6fd6c0492058a8e675dd9289920994f62a1db1b4483488efc6e46eaec3a123"}
Jul 28 04:29:25.554814 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:29:25.554453    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7847888f58-gc6c8" event={"ID":"0208cc05-10d1-4d71-b063-24b462ebfa0b","Type":"ContainerDied","Data":"5b90a34098eaaf3f2d057e26aec75cbf07689ee892e0f4951a77a1071a1c8345"}
Jul 28 04:29:25.554814 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:29:25.554474    2566 scope.go:117] "RemoveContainer" containerID="be6fd6c0492058a8e675dd9289920994f62a1db1b4483488efc6e46eaec3a123"
Jul 28 04:29:25.564052 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:29:25.564030    2566 scope.go:117] "RemoveContainer" containerID="be6fd6c0492058a8e675dd9289920994f62a1db1b4483488efc6e46eaec3a123"
Jul 28 04:29:25.564303 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:29:25.564284    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be6fd6c0492058a8e675dd9289920994f62a1db1b4483488efc6e46eaec3a123\": container with ID starting with be6fd6c0492058a8e675dd9289920994f62a1db1b4483488efc6e46eaec3a123 not found: ID does not exist" containerID="be6fd6c0492058a8e675dd9289920994f62a1db1b4483488efc6e46eaec3a123"
Jul 28 04:29:25.564356 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:29:25.564312    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be6fd6c0492058a8e675dd9289920994f62a1db1b4483488efc6e46eaec3a123"} err="failed to get container status \"be6fd6c0492058a8e675dd9289920994f62a1db1b4483488efc6e46eaec3a123\": rpc error: code = NotFound desc = could not find container \"be6fd6c0492058a8e675dd9289920994f62a1db1b4483488efc6e46eaec3a123\": container with ID starting with be6fd6c0492058a8e675dd9289920994f62a1db1b4483488efc6e46eaec3a123 not found: ID does not exist"
Jul 28 04:29:25.576211 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:29:25.576189    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7847888f58-gc6c8"]
Jul 28 04:29:25.578940 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:29:25.578916    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7847888f58-gc6c8"]
Jul 28 04:29:26.088567 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:29:26.088538    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0208cc05-10d1-4d71-b063-24b462ebfa0b" path="/var/lib/kubelet/pods/0208cc05-10d1-4d71-b063-24b462ebfa0b/volumes"
Jul 28 04:30:26.423020 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:26.422983    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7847888f58-d7hgl"]
Jul 28 04:30:26.423519 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:26.423410    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0208cc05-10d1-4d71-b063-24b462ebfa0b" containerName="manager"
Jul 28 04:30:26.423519 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:26.423423    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0208cc05-10d1-4d71-b063-24b462ebfa0b" containerName="manager"
Jul 28 04:30:26.423519 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:26.423498    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0208cc05-10d1-4d71-b063-24b462ebfa0b" containerName="manager"
Jul 28 04:30:26.426749 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:26.426731    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7847888f58-d7hgl"
Jul 28 04:30:26.429103 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:26.429081    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 28 04:30:26.430386 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:26.430370    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wxk49\""
Jul 28 04:30:26.430460 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:26.430374    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 28 04:30:26.435997 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:26.435970    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7847888f58-d7hgl"]
Jul 28 04:30:26.491121 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:26.491095    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/addcace1-b40a-410f-9e9c-e4f6e477b1bc-webhook-cert\") pod \"maas-controller-7847888f58-d7hgl\" (UID: \"addcace1-b40a-410f-9e9c-e4f6e477b1bc\") " pod="opendatahub/maas-controller-7847888f58-d7hgl"
Jul 28 04:30:26.491253 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:26.491146    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx59d\" (UniqueName: \"kubernetes.io/projected/addcace1-b40a-410f-9e9c-e4f6e477b1bc-kube-api-access-vx59d\") pod \"maas-controller-7847888f58-d7hgl\" (UID: \"addcace1-b40a-410f-9e9c-e4f6e477b1bc\") " pod="opendatahub/maas-controller-7847888f58-d7hgl"
Jul 28 04:30:26.591780 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:26.591747    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/addcace1-b40a-410f-9e9c-e4f6e477b1bc-webhook-cert\") pod \"maas-controller-7847888f58-d7hgl\" (UID: \"addcace1-b40a-410f-9e9c-e4f6e477b1bc\") " pod="opendatahub/maas-controller-7847888f58-d7hgl"
Jul 28 04:30:26.591993 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:26.591801    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx59d\" (UniqueName: \"kubernetes.io/projected/addcace1-b40a-410f-9e9c-e4f6e477b1bc-kube-api-access-vx59d\") pod \"maas-controller-7847888f58-d7hgl\" (UID: \"addcace1-b40a-410f-9e9c-e4f6e477b1bc\") " pod="opendatahub/maas-controller-7847888f58-d7hgl"
Jul 28 04:30:26.594207 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:26.594182    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/addcace1-b40a-410f-9e9c-e4f6e477b1bc-webhook-cert\") pod \"maas-controller-7847888f58-d7hgl\" (UID: \"addcace1-b40a-410f-9e9c-e4f6e477b1bc\") " pod="opendatahub/maas-controller-7847888f58-d7hgl"
Jul 28 04:30:26.601119 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:26.601097    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx59d\" (UniqueName: \"kubernetes.io/projected/addcace1-b40a-410f-9e9c-e4f6e477b1bc-kube-api-access-vx59d\") pod \"maas-controller-7847888f58-d7hgl\" (UID: \"addcace1-b40a-410f-9e9c-e4f6e477b1bc\") " pod="opendatahub/maas-controller-7847888f58-d7hgl"
Jul 28 04:30:26.738136 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:26.738049    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7847888f58-d7hgl"
Jul 28 04:30:26.875342 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:26.875318    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7847888f58-d7hgl"]
Jul 28 04:30:26.877479 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:30:26.877451    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaddcace1_b40a_410f_9e9c_e4f6e477b1bc.slice/crio-864f55b28b851dba851cb4c493d569eac07bb881ec76d95bad3f34842054438c WatchSource:0}: Error finding container 864f55b28b851dba851cb4c493d569eac07bb881ec76d95bad3f34842054438c: Status 404 returned error can't find the container with id 864f55b28b851dba851cb4c493d569eac07bb881ec76d95bad3f34842054438c
Jul 28 04:30:26.878888 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:26.878862    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 04:30:27.862094 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:27.861998    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7847888f58-d7hgl" event={"ID":"addcace1-b40a-410f-9e9c-e4f6e477b1bc","Type":"ContainerStarted","Data":"e52709e4876fa667b924e52d91d01cae2dc7bfea060b71e1934a98cdd1c95882"}
Jul 28 04:30:27.862094 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:27.862053    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7847888f58-d7hgl" event={"ID":"addcace1-b40a-410f-9e9c-e4f6e477b1bc","Type":"ContainerStarted","Data":"864f55b28b851dba851cb4c493d569eac07bb881ec76d95bad3f34842054438c"}
Jul 28 04:30:27.862094 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:27.862081    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7847888f58-d7hgl"
Jul 28 04:30:27.879503 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:27.879459    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7847888f58-d7hgl" podStartSLOduration=1.171159576 podStartE2EDuration="1.879444077s" totalImagesPullingTime="708.284501ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:30:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:30:27.877617937 +0000 UTC m=+1128.267793959" watchObservedRunningTime="2026-07-28 04:30:27.879444077 +0000 UTC m=+1128.269620097"
Jul 28 04:30:38.869622 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:38.869586    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7847888f58-d7hgl"
Jul 28 04:30:43.557555 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:43.557522    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7847888f58-d7hgl"]
Jul 28 04:30:43.558078 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:43.557701    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7847888f58-d7hgl" podUID="addcace1-b40a-410f-9e9c-e4f6e477b1bc" containerName="manager" containerID="cri-o://e52709e4876fa667b924e52d91d01cae2dc7bfea060b71e1934a98cdd1c95882" gracePeriod=30
Jul 28 04:30:43.812338 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:43.812266    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7847888f58-d7hgl"
Jul 28 04:30:43.852205 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:43.852178    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vx59d\" (UniqueName: \"kubernetes.io/projected/addcace1-b40a-410f-9e9c-e4f6e477b1bc-kube-api-access-vx59d\") pod \"addcace1-b40a-410f-9e9c-e4f6e477b1bc\" (UID: \"addcace1-b40a-410f-9e9c-e4f6e477b1bc\") "
Jul 28 04:30:43.852377 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:43.852221    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/addcace1-b40a-410f-9e9c-e4f6e477b1bc-webhook-cert\") pod \"addcace1-b40a-410f-9e9c-e4f6e477b1bc\" (UID: \"addcace1-b40a-410f-9e9c-e4f6e477b1bc\") "
Jul 28 04:30:43.854309 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:43.854270    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/addcace1-b40a-410f-9e9c-e4f6e477b1bc-kube-api-access-vx59d" (OuterVolumeSpecName: "kube-api-access-vx59d") pod "addcace1-b40a-410f-9e9c-e4f6e477b1bc" (UID: "addcace1-b40a-410f-9e9c-e4f6e477b1bc"). InnerVolumeSpecName "kube-api-access-vx59d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:30:43.854463 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:43.854437    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/addcace1-b40a-410f-9e9c-e4f6e477b1bc-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "addcace1-b40a-410f-9e9c-e4f6e477b1bc" (UID: "addcace1-b40a-410f-9e9c-e4f6e477b1bc"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 04:30:43.942044 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:43.942002    2566 generic.go:356] "Generic (PLEG): container finished" podID="addcace1-b40a-410f-9e9c-e4f6e477b1bc" containerID="e52709e4876fa667b924e52d91d01cae2dc7bfea060b71e1934a98cdd1c95882" exitCode=0
Jul 28 04:30:43.942204 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:43.942052    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7847888f58-d7hgl"
Jul 28 04:30:43.942204 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:43.942084    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7847888f58-d7hgl" event={"ID":"addcace1-b40a-410f-9e9c-e4f6e477b1bc","Type":"ContainerDied","Data":"e52709e4876fa667b924e52d91d01cae2dc7bfea060b71e1934a98cdd1c95882"}
Jul 28 04:30:43.942204 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:43.942136    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7847888f58-d7hgl" event={"ID":"addcace1-b40a-410f-9e9c-e4f6e477b1bc","Type":"ContainerDied","Data":"864f55b28b851dba851cb4c493d569eac07bb881ec76d95bad3f34842054438c"}
Jul 28 04:30:43.942204 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:43.942158    2566 scope.go:117] "RemoveContainer" containerID="e52709e4876fa667b924e52d91d01cae2dc7bfea060b71e1934a98cdd1c95882"
Jul 28 04:30:43.950944 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:43.950921    2566 scope.go:117] "RemoveContainer" containerID="e52709e4876fa667b924e52d91d01cae2dc7bfea060b71e1934a98cdd1c95882"
Jul 28 04:30:43.951211 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:30:43.951187    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e52709e4876fa667b924e52d91d01cae2dc7bfea060b71e1934a98cdd1c95882\": container with ID starting with e52709e4876fa667b924e52d91d01cae2dc7bfea060b71e1934a98cdd1c95882 not found: ID does not exist" containerID="e52709e4876fa667b924e52d91d01cae2dc7bfea060b71e1934a98cdd1c95882"
Jul 28 04:30:43.951314 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:43.951216    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e52709e4876fa667b924e52d91d01cae2dc7bfea060b71e1934a98cdd1c95882"} err="failed to get container status \"e52709e4876fa667b924e52d91d01cae2dc7bfea060b71e1934a98cdd1c95882\": rpc error: code = NotFound desc = could not find container \"e52709e4876fa667b924e52d91d01cae2dc7bfea060b71e1934a98cdd1c95882\": container with ID starting with e52709e4876fa667b924e52d91d01cae2dc7bfea060b71e1934a98cdd1c95882 not found: ID does not exist"
Jul 28 04:30:43.953435 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:43.953418    2566 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/addcace1-b40a-410f-9e9c-e4f6e477b1bc-webhook-cert\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:30:43.953494 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:43.953439    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx59d\" (UniqueName: \"kubernetes.io/projected/addcace1-b40a-410f-9e9c-e4f6e477b1bc-kube-api-access-vx59d\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:30:43.962263 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:43.962241    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7847888f58-d7hgl"]
Jul 28 04:30:43.965963 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:43.965945    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7847888f58-d7hgl"]
Jul 28 04:30:44.088355 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:44.088279    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="addcace1-b40a-410f-9e9c-e4f6e477b1bc" path="/var/lib/kubelet/pods/addcace1-b40a-410f-9e9c-e4f6e477b1bc/volumes"
Jul 28 04:30:44.804693 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:44.804656    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7847888f58-4nr6t"]
Jul 28 04:30:44.805206 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:44.805040    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="addcace1-b40a-410f-9e9c-e4f6e477b1bc" containerName="manager"
Jul 28 04:30:44.805206 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:44.805052    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="addcace1-b40a-410f-9e9c-e4f6e477b1bc" containerName="manager"
Jul 28 04:30:44.805206 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:44.805124    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="addcace1-b40a-410f-9e9c-e4f6e477b1bc" containerName="manager"
Jul 28 04:30:44.809208 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:44.809186    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7847888f58-4nr6t"
Jul 28 04:30:44.811407 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:44.811391    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 28 04:30:44.812361 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:44.812341    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-wxk49\""
Jul 28 04:30:44.812477 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:44.812408    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 28 04:30:44.818350 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:44.818329    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7847888f58-4nr6t"]
Jul 28 04:30:44.861514 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:44.861482    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/15ba97a6-0f40-4e47-a51a-2e23a16d5175-webhook-cert\") pod \"maas-controller-7847888f58-4nr6t\" (UID: \"15ba97a6-0f40-4e47-a51a-2e23a16d5175\") " pod="opendatahub/maas-controller-7847888f58-4nr6t"
Jul 28 04:30:44.861686 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:44.861533    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbbvt\" (UniqueName: \"kubernetes.io/projected/15ba97a6-0f40-4e47-a51a-2e23a16d5175-kube-api-access-xbbvt\") pod \"maas-controller-7847888f58-4nr6t\" (UID: \"15ba97a6-0f40-4e47-a51a-2e23a16d5175\") " pod="opendatahub/maas-controller-7847888f58-4nr6t"
Jul 28 04:30:44.962075 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:44.962043    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/15ba97a6-0f40-4e47-a51a-2e23a16d5175-webhook-cert\") pod \"maas-controller-7847888f58-4nr6t\" (UID: \"15ba97a6-0f40-4e47-a51a-2e23a16d5175\") " pod="opendatahub/maas-controller-7847888f58-4nr6t"
Jul 28 04:30:44.962270 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:44.962095    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbbvt\" (UniqueName: \"kubernetes.io/projected/15ba97a6-0f40-4e47-a51a-2e23a16d5175-kube-api-access-xbbvt\") pod \"maas-controller-7847888f58-4nr6t\" (UID: \"15ba97a6-0f40-4e47-a51a-2e23a16d5175\") " pod="opendatahub/maas-controller-7847888f58-4nr6t"
Jul 28 04:30:44.964479 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:44.964455    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/15ba97a6-0f40-4e47-a51a-2e23a16d5175-webhook-cert\") pod \"maas-controller-7847888f58-4nr6t\" (UID: \"15ba97a6-0f40-4e47-a51a-2e23a16d5175\") " pod="opendatahub/maas-controller-7847888f58-4nr6t"
Jul 28 04:30:44.969684 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:44.969660    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbbvt\" (UniqueName: \"kubernetes.io/projected/15ba97a6-0f40-4e47-a51a-2e23a16d5175-kube-api-access-xbbvt\") pod \"maas-controller-7847888f58-4nr6t\" (UID: \"15ba97a6-0f40-4e47-a51a-2e23a16d5175\") " pod="opendatahub/maas-controller-7847888f58-4nr6t"
Jul 28 04:30:45.120583 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:45.120486    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7847888f58-4nr6t"
Jul 28 04:30:45.254982 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:45.254954    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7847888f58-4nr6t"]
Jul 28 04:30:45.258538 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:30:45.258500    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15ba97a6_0f40_4e47_a51a_2e23a16d5175.slice/crio-9f413ecc131017d6662a9a06fd8bfe4b2ee9e743e676a409965f02bb9136ca72 WatchSource:0}: Error finding container 9f413ecc131017d6662a9a06fd8bfe4b2ee9e743e676a409965f02bb9136ca72: Status 404 returned error can't find the container with id 9f413ecc131017d6662a9a06fd8bfe4b2ee9e743e676a409965f02bb9136ca72
Jul 28 04:30:45.954087 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:45.953988    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7847888f58-4nr6t" event={"ID":"15ba97a6-0f40-4e47-a51a-2e23a16d5175","Type":"ContainerStarted","Data":"330604eb7cb437c1dd4e9e132c4d7fb168a466516b18fda592de51242ddf82a4"}
Jul 28 04:30:45.954087 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:45.954049    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7847888f58-4nr6t" event={"ID":"15ba97a6-0f40-4e47-a51a-2e23a16d5175","Type":"ContainerStarted","Data":"9f413ecc131017d6662a9a06fd8bfe4b2ee9e743e676a409965f02bb9136ca72"}
Jul 28 04:30:45.954087 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:45.954082    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7847888f58-4nr6t"
Jul 28 04:30:45.971548 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:45.971500    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7847888f58-4nr6t" podStartSLOduration=1.614618191 podStartE2EDuration="1.971483718s" totalImagesPullingTime="356.865527ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:30:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:30:45.969368535 +0000 UTC m=+1146.359544575" watchObservedRunningTime="2026-07-28 04:30:45.971483718 +0000 UTC m=+1146.361659739"
Jul 28 04:30:56.961674 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:30:56.961646    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7847888f58-4nr6t"
Jul 28 04:31:43.601459 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:31:43.601356    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4glpr_bfd2d3d8-f287-42d8-83c7-f82185c96735/console-operator/2.log"
Jul 28 04:31:43.605793 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:31:43.603766    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4glpr_bfd2d3d8-f287-42d8-83c7-f82185c96735/console-operator/2.log"
Jul 28 04:35:39.787807 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:35:39.787723    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-klr2g"]
Jul 28 04:35:39.788341 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:35:39.787985    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-klr2g" podUID="3631aa8e-ef0c-45eb-aaf7-b9e2d2e344e2" containerName="manager" containerID="cri-o://ceaa31864e77c813bcf5b9c57637ef3f896125d1cd4e1534d83a1d54c986ad68" gracePeriod=10
Jul 28 04:35:42.744854 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:35:42.744818    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-klr2g"
Jul 28 04:35:42.917741 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:35:42.917648    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3631aa8e-ef0c-45eb-aaf7-b9e2d2e344e2-extensions-socket-volume\") pod \"3631aa8e-ef0c-45eb-aaf7-b9e2d2e344e2\" (UID: \"3631aa8e-ef0c-45eb-aaf7-b9e2d2e344e2\") "
Jul 28 04:35:42.917956 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:35:42.917759    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x59l4\" (UniqueName: \"kubernetes.io/projected/3631aa8e-ef0c-45eb-aaf7-b9e2d2e344e2-kube-api-access-x59l4\") pod \"3631aa8e-ef0c-45eb-aaf7-b9e2d2e344e2\" (UID: \"3631aa8e-ef0c-45eb-aaf7-b9e2d2e344e2\") "
Jul 28 04:35:42.918112 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:35:42.918088    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3631aa8e-ef0c-45eb-aaf7-b9e2d2e344e2-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3631aa8e-ef0c-45eb-aaf7-b9e2d2e344e2" (UID: "3631aa8e-ef0c-45eb-aaf7-b9e2d2e344e2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:35:42.919802 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:35:42.919778    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3631aa8e-ef0c-45eb-aaf7-b9e2d2e344e2-kube-api-access-x59l4" (OuterVolumeSpecName: "kube-api-access-x59l4") pod "3631aa8e-ef0c-45eb-aaf7-b9e2d2e344e2" (UID: "3631aa8e-ef0c-45eb-aaf7-b9e2d2e344e2"). InnerVolumeSpecName "kube-api-access-x59l4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:35:43.019464 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:35:43.019426    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x59l4\" (UniqueName: \"kubernetes.io/projected/3631aa8e-ef0c-45eb-aaf7-b9e2d2e344e2-kube-api-access-x59l4\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:35:43.019464 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:35:43.019460    2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3631aa8e-ef0c-45eb-aaf7-b9e2d2e344e2-extensions-socket-volume\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:35:43.366932 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:35:43.366895    2566 generic.go:356] "Generic (PLEG): container finished" podID="3631aa8e-ef0c-45eb-aaf7-b9e2d2e344e2" containerID="ceaa31864e77c813bcf5b9c57637ef3f896125d1cd4e1534d83a1d54c986ad68" exitCode=0
Jul 28 04:35:43.367113 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:35:43.366945    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-klr2g"
Jul 28 04:35:43.367113 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:35:43.366979    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-klr2g" event={"ID":"3631aa8e-ef0c-45eb-aaf7-b9e2d2e344e2","Type":"ContainerDied","Data":"ceaa31864e77c813bcf5b9c57637ef3f896125d1cd4e1534d83a1d54c986ad68"}
Jul 28 04:35:43.367113 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:35:43.367035    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-klr2g" event={"ID":"3631aa8e-ef0c-45eb-aaf7-b9e2d2e344e2","Type":"ContainerDied","Data":"d4c85146b4dd83a883eed402ff32eadec1c98306f718a329e3f43c295c76e42f"}
Jul 28 04:35:43.367113 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:35:43.367058    2566 scope.go:117] "RemoveContainer" containerID="ceaa31864e77c813bcf5b9c57637ef3f896125d1cd4e1534d83a1d54c986ad68"
Jul 28 04:35:43.377025 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:35:43.377006    2566 scope.go:117] "RemoveContainer" containerID="ceaa31864e77c813bcf5b9c57637ef3f896125d1cd4e1534d83a1d54c986ad68"
Jul 28 04:35:43.377280 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:35:43.377260    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceaa31864e77c813bcf5b9c57637ef3f896125d1cd4e1534d83a1d54c986ad68\": container with ID starting with ceaa31864e77c813bcf5b9c57637ef3f896125d1cd4e1534d83a1d54c986ad68 not found: ID does not exist" containerID="ceaa31864e77c813bcf5b9c57637ef3f896125d1cd4e1534d83a1d54c986ad68"
Jul 28 04:35:43.377332 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:35:43.377291    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceaa31864e77c813bcf5b9c57637ef3f896125d1cd4e1534d83a1d54c986ad68"} err="failed to get container status \"ceaa31864e77c813bcf5b9c57637ef3f896125d1cd4e1534d83a1d54c986ad68\": rpc error: code = NotFound desc = could not find container \"ceaa31864e77c813bcf5b9c57637ef3f896125d1cd4e1534d83a1d54c986ad68\": container with ID starting with ceaa31864e77c813bcf5b9c57637ef3f896125d1cd4e1534d83a1d54c986ad68 not found: ID does not exist"
Jul 28 04:35:43.388247 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:35:43.388221    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-klr2g"]
Jul 28 04:35:43.392093 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:35:43.392074    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-klr2g"]
Jul 28 04:35:44.088297 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:35:44.088263    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3631aa8e-ef0c-45eb-aaf7-b9e2d2e344e2" path="/var/lib/kubelet/pods/3631aa8e-ef0c-45eb-aaf7-b9e2d2e344e2/volumes"
Jul 28 04:36:43.630498 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:36:43.630381    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4glpr_bfd2d3d8-f287-42d8-83c7-f82185c96735/console-operator/2.log"
Jul 28 04:36:43.634480 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:36:43.633772    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4glpr_bfd2d3d8-f287-42d8-83c7-f82185c96735/console-operator/2.log"
Jul 28 04:39:31.901541 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:31.901493    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm"]
Jul 28 04:39:31.902157 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:31.902095    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3631aa8e-ef0c-45eb-aaf7-b9e2d2e344e2" containerName="manager"
Jul 28 04:39:31.902157 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:31.902114    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3631aa8e-ef0c-45eb-aaf7-b9e2d2e344e2" containerName="manager"
Jul 28 04:39:31.902274 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:31.902222    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3631aa8e-ef0c-45eb-aaf7-b9e2d2e344e2" containerName="manager"
Jul 28 04:39:31.905730 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:31.905706    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm"
Jul 28 04:39:31.908239 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:31.908216    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-38645996-kserve-self-signed-certs\""
Jul 28 04:39:31.919123 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:31.919101    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm"]
Jul 28 04:39:31.988227 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:31.984649    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af031d18-448c-4eef-812d-24a70c77eea1-home\") pod \"e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm\" (UID: \"af031d18-448c-4eef-812d-24a70c77eea1\") " pod="llm/e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm"
Jul 28 04:39:31.988227 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:31.984795    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af031d18-448c-4eef-812d-24a70c77eea1-dshm\") pod \"e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm\" (UID: \"af031d18-448c-4eef-812d-24a70c77eea1\") " pod="llm/e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm"
Jul 28 04:39:31.988227 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:31.984965    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af031d18-448c-4eef-812d-24a70c77eea1-model-cache\") pod \"e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm\" (UID: \"af031d18-448c-4eef-812d-24a70c77eea1\") " pod="llm/e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm"
Jul 28 04:39:31.988227 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:31.985003    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af031d18-448c-4eef-812d-24a70c77eea1-tls-certs\") pod \"e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm\" (UID: \"af031d18-448c-4eef-812d-24a70c77eea1\") " pod="llm/e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm"
Jul 28 04:39:31.988227 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:31.985038    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kbhb\" (UniqueName: \"kubernetes.io/projected/af031d18-448c-4eef-812d-24a70c77eea1-kube-api-access-6kbhb\") pod \"e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm\" (UID: \"af031d18-448c-4eef-812d-24a70c77eea1\") " pod="llm/e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm"
Jul 28 04:39:31.988227 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:31.985078    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af031d18-448c-4eef-812d-24a70c77eea1-tmp-dir\") pod \"e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm\" (UID: \"af031d18-448c-4eef-812d-24a70c77eea1\") " pod="llm/e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm"
Jul 28 04:39:32.085623 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:32.085586    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af031d18-448c-4eef-812d-24a70c77eea1-tmp-dir\") pod \"e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm\" (UID: \"af031d18-448c-4eef-812d-24a70c77eea1\") " pod="llm/e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm"
Jul 28 04:39:32.085794 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:32.085654    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af031d18-448c-4eef-812d-24a70c77eea1-home\") pod \"e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm\" (UID: \"af031d18-448c-4eef-812d-24a70c77eea1\") " pod="llm/e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm"
Jul 28 04:39:32.085794 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:32.085686    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af031d18-448c-4eef-812d-24a70c77eea1-dshm\") pod \"e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm\" (UID: \"af031d18-448c-4eef-812d-24a70c77eea1\") " pod="llm/e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm"
Jul 28 04:39:32.085927 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:32.085800    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af031d18-448c-4eef-812d-24a70c77eea1-model-cache\") pod \"e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm\" (UID: \"af031d18-448c-4eef-812d-24a70c77eea1\") " pod="llm/e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm"
Jul 28 04:39:32.085927 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:32.085825    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af031d18-448c-4eef-812d-24a70c77eea1-tls-certs\") pod \"e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm\" (UID: \"af031d18-448c-4eef-812d-24a70c77eea1\") " pod="llm/e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm"
Jul 28 04:39:32.085927 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:32.085867    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kbhb\" (UniqueName: \"kubernetes.io/projected/af031d18-448c-4eef-812d-24a70c77eea1-kube-api-access-6kbhb\") pod \"e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm\" (UID: \"af031d18-448c-4eef-812d-24a70c77eea1\") " pod="llm/e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm"
Jul 28 04:39:32.086061 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:32.086043    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af031d18-448c-4eef-812d-24a70c77eea1-tmp-dir\") pod \"e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm\" (UID: \"af031d18-448c-4eef-812d-24a70c77eea1\") " pod="llm/e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm"
Jul 28 04:39:32.086120 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:32.086096    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af031d18-448c-4eef-812d-24a70c77eea1-home\") pod \"e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm\" (UID: \"af031d18-448c-4eef-812d-24a70c77eea1\") " pod="llm/e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm"
Jul 28 04:39:32.086400 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:32.086374    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af031d18-448c-4eef-812d-24a70c77eea1-model-cache\") pod \"e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm\" (UID: \"af031d18-448c-4eef-812d-24a70c77eea1\") " pod="llm/e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm"
Jul 28 04:39:32.088238 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:32.088210    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af031d18-448c-4eef-812d-24a70c77eea1-dshm\") pod \"e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm\" (UID: \"af031d18-448c-4eef-812d-24a70c77eea1\") " pod="llm/e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm"
Jul 28 04:39:32.088561 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:32.088543    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af031d18-448c-4eef-812d-24a70c77eea1-tls-certs\") pod \"e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm\" (UID: \"af031d18-448c-4eef-812d-24a70c77eea1\") " pod="llm/e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm"
Jul 28 04:39:32.093257 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:32.093237    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kbhb\" (UniqueName: \"kubernetes.io/projected/af031d18-448c-4eef-812d-24a70c77eea1-kube-api-access-6kbhb\") pod \"e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm\" (UID: \"af031d18-448c-4eef-812d-24a70c77eea1\") " pod="llm/e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm"
Jul 28 04:39:32.217335 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:32.217243    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm"
Jul 28 04:39:32.349454 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:32.349431    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm"]
Jul 28 04:39:32.351232 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:39:32.351197    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf031d18_448c_4eef_812d_24a70c77eea1.slice/crio-18edebef2ca3d40f6e182f3bf1d4d8eda9717cb14692178caaf2667216322bc6 WatchSource:0}: Error finding container 18edebef2ca3d40f6e182f3bf1d4d8eda9717cb14692178caaf2667216322bc6: Status 404 returned error can't find the container with id 18edebef2ca3d40f6e182f3bf1d4d8eda9717cb14692178caaf2667216322bc6
Jul 28 04:39:32.471588 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:32.471503    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm" event={"ID":"af031d18-448c-4eef-812d-24a70c77eea1","Type":"ContainerStarted","Data":"7d326c25f9efbfce0687877239d1530ab394b451ff06af49d5fca9adde7bd553"}
Jul 28 04:39:32.471588 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:32.471557    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm" event={"ID":"af031d18-448c-4eef-812d-24a70c77eea1","Type":"ContainerStarted","Data":"18edebef2ca3d40f6e182f3bf1d4d8eda9717cb14692178caaf2667216322bc6"}
Jul 28 04:39:32.487511 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:32.487457    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm" podStartSLOduration=1.487441491 podStartE2EDuration="1.487441491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:39:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:39:32.486614911 +0000 UTC m=+1672.876790957" watchObservedRunningTime="2026-07-28 04:39:32.487441491 +0000 UTC m=+1672.877617513"
Jul 28 04:39:41.673629 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:41.673586    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-38645996-kserve-759b54597f-dq27h"]
Jul 28 04:39:41.677064 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:41.677044    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-38645996-kserve-759b54597f-dq27h"
Jul 28 04:39:41.679439 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:41.679421    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-38645996-kserve-self-signed-certs\""
Jul 28 04:39:41.685518 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:41.685498    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-38645996-kserve-759b54597f-dq27h"]
Jul 28 04:39:41.773148 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:41.773107    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5-tmp-dir\") pod \"e2e-same-id-b-38645996-kserve-759b54597f-dq27h\" (UID: \"3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5\") " pod="llm/e2e-same-id-b-38645996-kserve-759b54597f-dq27h"
Jul 28 04:39:41.773337 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:41.773157    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5-model-cache\") pod \"e2e-same-id-b-38645996-kserve-759b54597f-dq27h\" (UID: \"3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5\") " pod="llm/e2e-same-id-b-38645996-kserve-759b54597f-dq27h"
Jul 28 04:39:41.773337 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:41.773190    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5-home\") pod \"e2e-same-id-b-38645996-kserve-759b54597f-dq27h\" (UID: \"3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5\") " pod="llm/e2e-same-id-b-38645996-kserve-759b54597f-dq27h"
Jul 28 04:39:41.773337 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:41.773265    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5-tls-certs\") pod \"e2e-same-id-b-38645996-kserve-759b54597f-dq27h\" (UID: \"3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5\") " pod="llm/e2e-same-id-b-38645996-kserve-759b54597f-dq27h"
Jul 28 04:39:41.773337 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:41.773292    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g9jn\" (UniqueName: \"kubernetes.io/projected/3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5-kube-api-access-6g9jn\") pod \"e2e-same-id-b-38645996-kserve-759b54597f-dq27h\" (UID: \"3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5\") " pod="llm/e2e-same-id-b-38645996-kserve-759b54597f-dq27h"
Jul 28 04:39:41.773337 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:41.773312    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5-dshm\") pod \"e2e-same-id-b-38645996-kserve-759b54597f-dq27h\" (UID: \"3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5\") " pod="llm/e2e-same-id-b-38645996-kserve-759b54597f-dq27h"
Jul 28 04:39:41.874443 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:41.874407    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5-tmp-dir\") pod \"e2e-same-id-b-38645996-kserve-759b54597f-dq27h\" (UID: \"3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5\") " pod="llm/e2e-same-id-b-38645996-kserve-759b54597f-dq27h"
Jul 28 04:39:41.874443 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:41.874444    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5-model-cache\") pod \"e2e-same-id-b-38645996-kserve-759b54597f-dq27h\" (UID: \"3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5\") " pod="llm/e2e-same-id-b-38645996-kserve-759b54597f-dq27h"
Jul 28 04:39:41.874653 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:41.874461    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5-home\") pod \"e2e-same-id-b-38645996-kserve-759b54597f-dq27h\" (UID: \"3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5\") " pod="llm/e2e-same-id-b-38645996-kserve-759b54597f-dq27h"
Jul 28 04:39:41.874653 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:41.874503    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5-tls-certs\") pod \"e2e-same-id-b-38645996-kserve-759b54597f-dq27h\" (UID: \"3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5\") " pod="llm/e2e-same-id-b-38645996-kserve-759b54597f-dq27h"
Jul 28 04:39:41.874653 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:41.874534    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g9jn\" (UniqueName: \"kubernetes.io/projected/3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5-kube-api-access-6g9jn\") pod \"e2e-same-id-b-38645996-kserve-759b54597f-dq27h\" (UID: \"3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5\") " pod="llm/e2e-same-id-b-38645996-kserve-759b54597f-dq27h"
Jul 28 04:39:41.874653 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:41.874564    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5-dshm\") pod \"e2e-same-id-b-38645996-kserve-759b54597f-dq27h\" (UID: \"3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5\") " pod="llm/e2e-same-id-b-38645996-kserve-759b54597f-dq27h"
Jul 28 04:39:41.874913 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:41.874887    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5-tmp-dir\") pod \"e2e-same-id-b-38645996-kserve-759b54597f-dq27h\" (UID: \"3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5\") " pod="llm/e2e-same-id-b-38645996-kserve-759b54597f-dq27h"
Jul 28 04:39:41.874981 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:41.874954    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5-model-cache\") pod \"e2e-same-id-b-38645996-kserve-759b54597f-dq27h\" (UID: \"3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5\") " pod="llm/e2e-same-id-b-38645996-kserve-759b54597f-dq27h"
Jul 28 04:39:41.874981 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:41.874971    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5-home\") pod \"e2e-same-id-b-38645996-kserve-759b54597f-dq27h\" (UID: \"3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5\") " pod="llm/e2e-same-id-b-38645996-kserve-759b54597f-dq27h"
Jul 28 04:39:41.876794 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:41.876770    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5-dshm\") pod \"e2e-same-id-b-38645996-kserve-759b54597f-dq27h\" (UID: \"3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5\") " pod="llm/e2e-same-id-b-38645996-kserve-759b54597f-dq27h"
Jul 28 04:39:41.877013 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:41.876998    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5-tls-certs\") pod \"e2e-same-id-b-38645996-kserve-759b54597f-dq27h\" (UID: \"3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5\") " pod="llm/e2e-same-id-b-38645996-kserve-759b54597f-dq27h"
Jul 28 04:39:41.881919 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:41.881897    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g9jn\" (UniqueName: \"kubernetes.io/projected/3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5-kube-api-access-6g9jn\") pod \"e2e-same-id-b-38645996-kserve-759b54597f-dq27h\" (UID: \"3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5\") " pod="llm/e2e-same-id-b-38645996-kserve-759b54597f-dq27h"
Jul 28 04:39:41.986527 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:41.986442    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-38645996-kserve-759b54597f-dq27h"
Jul 28 04:39:42.122742 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:42.122695    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-38645996-kserve-759b54597f-dq27h"]
Jul 28 04:39:42.124882 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:39:42.124853    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3cfd84a5_7ef6_43cd_b4e8_8f9e92cf0cf5.slice/crio-715c5e9a7dfd073085e870017317db9cdd4bd2cfad78337d4082c918cde15d16 WatchSource:0}: Error finding container 715c5e9a7dfd073085e870017317db9cdd4bd2cfad78337d4082c918cde15d16: Status 404 returned error can't find the container with id 715c5e9a7dfd073085e870017317db9cdd4bd2cfad78337d4082c918cde15d16
Jul 28 04:39:42.217613 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:42.217583    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm"
Jul 28 04:39:42.217727 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:42.217625    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm"
Jul 28 04:39:42.230209 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:42.230188    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm"
Jul 28 04:39:42.521306 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:42.521262    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-38645996-kserve-759b54597f-dq27h" event={"ID":"3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5","Type":"ContainerStarted","Data":"deb9fdbf278898c1785589eb21876280485a6232f0ab7b8c287963be904cb4fc"}
Jul 28 04:39:42.521306 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:42.521314    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-38645996-kserve-759b54597f-dq27h" event={"ID":"3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5","Type":"ContainerStarted","Data":"715c5e9a7dfd073085e870017317db9cdd4bd2cfad78337d4082c918cde15d16"}
Jul 28 04:39:42.532737 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:42.532713    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm"
Jul 28 04:39:42.538876 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:42.538818    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-38645996-kserve-759b54597f-dq27h" podStartSLOduration=1.538802112 podStartE2EDuration="1.538802112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:39:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:39:42.536157999 +0000 UTC m=+1682.926334020" watchObservedRunningTime="2026-07-28 04:39:42.538802112 +0000 UTC m=+1682.928978134"
Jul 28 04:39:51.987386 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:51.987350    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-38645996-kserve-759b54597f-dq27h"
Jul 28 04:39:51.987823 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:51.987402    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-38645996-kserve-759b54597f-dq27h"
Jul 28 04:39:52.000178 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:52.000150    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-38645996-kserve-759b54597f-dq27h"
Jul 28 04:39:52.580006 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:39:52.579979    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-38645996-kserve-759b54597f-dq27h"
Jul 28 04:40:09.904545 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:09.904454    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm"]
Jul 28 04:40:10.495099 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:10.495067    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-38645996-kserve-759b54597f-dq27h"]
Jul 28 04:40:24.925150 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:24.925107    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm" podUID="af031d18-448c-4eef-812d-24a70c77eea1" containerName="main" containerID="cri-o://7d326c25f9efbfce0687877239d1530ab394b451ff06af49d5fca9adde7bd553" gracePeriod=45
Jul 28 04:40:25.184276 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.184210    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm"
Jul 28 04:40:25.300020 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.299987    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af031d18-448c-4eef-812d-24a70c77eea1-tmp-dir\") pod \"af031d18-448c-4eef-812d-24a70c77eea1\" (UID: \"af031d18-448c-4eef-812d-24a70c77eea1\") "
Jul 28 04:40:25.300188 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.300044    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af031d18-448c-4eef-812d-24a70c77eea1-tls-certs\") pod \"af031d18-448c-4eef-812d-24a70c77eea1\" (UID: \"af031d18-448c-4eef-812d-24a70c77eea1\") "
Jul 28 04:40:25.300188 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.300077    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6kbhb\" (UniqueName: \"kubernetes.io/projected/af031d18-448c-4eef-812d-24a70c77eea1-kube-api-access-6kbhb\") pod \"af031d18-448c-4eef-812d-24a70c77eea1\" (UID: \"af031d18-448c-4eef-812d-24a70c77eea1\") "
Jul 28 04:40:25.300188 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.300102    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af031d18-448c-4eef-812d-24a70c77eea1-dshm\") pod \"af031d18-448c-4eef-812d-24a70c77eea1\" (UID: \"af031d18-448c-4eef-812d-24a70c77eea1\") "
Jul 28 04:40:25.300188 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.300125    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af031d18-448c-4eef-812d-24a70c77eea1-model-cache\") pod \"af031d18-448c-4eef-812d-24a70c77eea1\" (UID: \"af031d18-448c-4eef-812d-24a70c77eea1\") "
Jul 28 04:40:25.300188 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.300165    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af031d18-448c-4eef-812d-24a70c77eea1-home\") pod \"af031d18-448c-4eef-812d-24a70c77eea1\" (UID: \"af031d18-448c-4eef-812d-24a70c77eea1\") "
Jul 28 04:40:25.300435 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.300320    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af031d18-448c-4eef-812d-24a70c77eea1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "af031d18-448c-4eef-812d-24a70c77eea1" (UID: "af031d18-448c-4eef-812d-24a70c77eea1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:40:25.300495 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.300417    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af031d18-448c-4eef-812d-24a70c77eea1-model-cache" (OuterVolumeSpecName: "model-cache") pod "af031d18-448c-4eef-812d-24a70c77eea1" (UID: "af031d18-448c-4eef-812d-24a70c77eea1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:40:25.300551 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.300512    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af031d18-448c-4eef-812d-24a70c77eea1-home" (OuterVolumeSpecName: "home") pod "af031d18-448c-4eef-812d-24a70c77eea1" (UID: "af031d18-448c-4eef-812d-24a70c77eea1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:40:25.300599 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.300563    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af031d18-448c-4eef-812d-24a70c77eea1-tmp-dir\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:40:25.300599 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.300587    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af031d18-448c-4eef-812d-24a70c77eea1-model-cache\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:40:25.302383 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.302359    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af031d18-448c-4eef-812d-24a70c77eea1-dshm" (OuterVolumeSpecName: "dshm") pod "af031d18-448c-4eef-812d-24a70c77eea1" (UID: "af031d18-448c-4eef-812d-24a70c77eea1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:40:25.302480 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.302396    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af031d18-448c-4eef-812d-24a70c77eea1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "af031d18-448c-4eef-812d-24a70c77eea1" (UID: "af031d18-448c-4eef-812d-24a70c77eea1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 04:40:25.302518 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.302485    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af031d18-448c-4eef-812d-24a70c77eea1-kube-api-access-6kbhb" (OuterVolumeSpecName: "kube-api-access-6kbhb") pod "af031d18-448c-4eef-812d-24a70c77eea1" (UID: "af031d18-448c-4eef-812d-24a70c77eea1"). InnerVolumeSpecName "kube-api-access-6kbhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:40:25.401531 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.401497    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af031d18-448c-4eef-812d-24a70c77eea1-tls-certs\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:40:25.401531 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.401526    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kbhb\" (UniqueName: \"kubernetes.io/projected/af031d18-448c-4eef-812d-24a70c77eea1-kube-api-access-6kbhb\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:40:25.401531 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.401535    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af031d18-448c-4eef-812d-24a70c77eea1-dshm\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:40:25.401769 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.401544    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af031d18-448c-4eef-812d-24a70c77eea1-home\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:40:25.515347 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.515312    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-38645996-kserve-759b54597f-dq27h" podUID="3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5" containerName="main" containerID="cri-o://deb9fdbf278898c1785589eb21876280485a6232f0ab7b8c287963be904cb4fc" gracePeriod=45
Jul 28 04:40:25.729305 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.729272    2566 generic.go:356] "Generic (PLEG): container finished" podID="3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5" containerID="deb9fdbf278898c1785589eb21876280485a6232f0ab7b8c287963be904cb4fc" exitCode=0
Jul 28 04:40:25.729469 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.729352    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-38645996-kserve-759b54597f-dq27h" event={"ID":"3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5","Type":"ContainerDied","Data":"deb9fdbf278898c1785589eb21876280485a6232f0ab7b8c287963be904cb4fc"}
Jul 28 04:40:25.730631 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.730607    2566 generic.go:356] "Generic (PLEG): container finished" podID="af031d18-448c-4eef-812d-24a70c77eea1" containerID="7d326c25f9efbfce0687877239d1530ab394b451ff06af49d5fca9adde7bd553" exitCode=0
Jul 28 04:40:25.730757 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.730674    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm"
Jul 28 04:40:25.730757 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.730684    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm" event={"ID":"af031d18-448c-4eef-812d-24a70c77eea1","Type":"ContainerDied","Data":"7d326c25f9efbfce0687877239d1530ab394b451ff06af49d5fca9adde7bd553"}
Jul 28 04:40:25.730757 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.730724    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm" event={"ID":"af031d18-448c-4eef-812d-24a70c77eea1","Type":"ContainerDied","Data":"18edebef2ca3d40f6e182f3bf1d4d8eda9717cb14692178caaf2667216322bc6"}
Jul 28 04:40:25.730757 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.730742    2566 scope.go:117] "RemoveContainer" containerID="7d326c25f9efbfce0687877239d1530ab394b451ff06af49d5fca9adde7bd553"
Jul 28 04:40:25.739255 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.739236    2566 scope.go:117] "RemoveContainer" containerID="7d326c25f9efbfce0687877239d1530ab394b451ff06af49d5fca9adde7bd553"
Jul 28 04:40:25.739524 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:40:25.739507    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d326c25f9efbfce0687877239d1530ab394b451ff06af49d5fca9adde7bd553\": container with ID starting with 7d326c25f9efbfce0687877239d1530ab394b451ff06af49d5fca9adde7bd553 not found: ID does not exist" containerID="7d326c25f9efbfce0687877239d1530ab394b451ff06af49d5fca9adde7bd553"
Jul 28 04:40:25.739593 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.739533    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d326c25f9efbfce0687877239d1530ab394b451ff06af49d5fca9adde7bd553"} err="failed to get container status \"7d326c25f9efbfce0687877239d1530ab394b451ff06af49d5fca9adde7bd553\": rpc error: code = NotFound desc = could not find container \"7d326c25f9efbfce0687877239d1530ab394b451ff06af49d5fca9adde7bd553\": container with ID starting with 7d326c25f9efbfce0687877239d1530ab394b451ff06af49d5fca9adde7bd553 not found: ID does not exist"
Jul 28 04:40:25.753648 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.753627    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm"]
Jul 28 04:40:25.757917 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.757895    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-38645996-kserve-dc55b4f9f-zfmdm"]
Jul 28 04:40:25.762130 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.762110    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-38645996-kserve-759b54597f-dq27h"
Jul 28 04:40:25.806222 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.806127    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5-tmp-dir\") pod \"3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5\" (UID: \"3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5\") "
Jul 28 04:40:25.806222 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.806164    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5-model-cache\") pod \"3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5\" (UID: \"3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5\") "
Jul 28 04:40:25.806222 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.806193    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6g9jn\" (UniqueName: \"kubernetes.io/projected/3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5-kube-api-access-6g9jn\") pod \"3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5\" (UID: \"3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5\") "
Jul 28 04:40:25.806516 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.806240    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5-dshm\") pod \"3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5\" (UID: \"3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5\") "
Jul 28 04:40:25.806516 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.806290    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5-home\") pod \"3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5\" (UID: \"3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5\") "
Jul 28 04:40:25.806516 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.806321    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5-tls-certs\") pod \"3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5\" (UID: \"3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5\") "
Jul 28 04:40:25.806516 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.806439    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5-model-cache" (OuterVolumeSpecName: "model-cache") pod "3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5" (UID: "3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:40:25.806516 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.806455    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5" (UID: "3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:40:25.806792 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.806560    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5-home" (OuterVolumeSpecName: "home") pod "3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5" (UID: "3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:40:25.806877 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.806810    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5-home\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:40:25.806877 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.806830    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5-tmp-dir\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:40:25.806877 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.806860    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5-model-cache\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:40:25.808490 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.808459    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5-kube-api-access-6g9jn" (OuterVolumeSpecName: "kube-api-access-6g9jn") pod "3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5" (UID: "3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5"). InnerVolumeSpecName "kube-api-access-6g9jn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:40:25.808812 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.808791    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5" (UID: "3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 04:40:25.808812 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.808799    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5-dshm" (OuterVolumeSpecName: "dshm") pod "3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5" (UID: "3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:40:25.907965 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.907935    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5-tls-certs\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:40:25.907965 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.907962    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g9jn\" (UniqueName: \"kubernetes.io/projected/3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5-kube-api-access-6g9jn\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:40:25.907965 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:25.907972    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5-dshm\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:40:26.088661 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:26.088571    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af031d18-448c-4eef-812d-24a70c77eea1" path="/var/lib/kubelet/pods/af031d18-448c-4eef-812d-24a70c77eea1/volumes"
Jul 28 04:40:26.736624 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:26.736589    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-38645996-kserve-759b54597f-dq27h"
Jul 28 04:40:26.736826 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:26.736593    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-38645996-kserve-759b54597f-dq27h" event={"ID":"3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5","Type":"ContainerDied","Data":"715c5e9a7dfd073085e870017317db9cdd4bd2cfad78337d4082c918cde15d16"}
Jul 28 04:40:26.736826 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:26.736712    2566 scope.go:117] "RemoveContainer" containerID="deb9fdbf278898c1785589eb21876280485a6232f0ab7b8c287963be904cb4fc"
Jul 28 04:40:26.751639 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:26.751612    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-38645996-kserve-759b54597f-dq27h"]
Jul 28 04:40:26.753938 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:26.753918    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-38645996-kserve-759b54597f-dq27h"]
Jul 28 04:40:28.088142 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:40:28.088109    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5" path="/var/lib/kubelet/pods/3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5/volumes"
Jul 28 04:41:43.660362 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:41:43.660239    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4glpr_bfd2d3d8-f287-42d8-83c7-f82185c96735/console-operator/2.log"
Jul 28 04:41:43.665317 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:41:43.665122    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4glpr_bfd2d3d8-f287-42d8-83c7-f82185c96735/console-operator/2.log"
Jul 28 04:43:53.588898 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:53.588861    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ecc0dac1-c6d8f487b-8hxk5"]
Jul 28 04:43:53.589548 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:53.589249    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af031d18-448c-4eef-812d-24a70c77eea1" containerName="main"
Jul 28 04:43:53.589548 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:53.589261    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="af031d18-448c-4eef-812d-24a70c77eea1" containerName="main"
Jul 28 04:43:53.589548 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:53.589274    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5" containerName="main"
Jul 28 04:43:53.589548 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:53.589280    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5" containerName="main"
Jul 28 04:43:53.589548 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:53.589341    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cfd84a5-7ef6-43cd-b4e8-8f9e92cf0cf5" containerName="main"
Jul 28 04:43:53.589548 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:53.589349    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="af031d18-448c-4eef-812d-24a70c77eea1" containerName="main"
Jul 28 04:43:53.592602 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:53.592583    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ecc0dac1-c6d8f487b-8hxk5"
Jul 28 04:43:53.594807 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:53.594787    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ecc0dac1-serving-cert\""
Jul 28 04:43:53.595880 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:53.595833    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 04:43:53.596003 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:53.595903    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gfn77\""
Jul 28 04:43:53.596003 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:53.595909    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 04:43:53.601754 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:53.601726    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ecc0dac1-c6d8f487b-8hxk5"]
Jul 28 04:43:53.761766 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:53.761729    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb4t5\" (UniqueName: \"kubernetes.io/projected/ece14f8a-56e7-4966-915f-74fed04c43a8-kube-api-access-wb4t5\") pod \"maas-api-e2e-ait-ecc0dac1-c6d8f487b-8hxk5\" (UID: \"ece14f8a-56e7-4966-915f-74fed04c43a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ecc0dac1-c6d8f487b-8hxk5"
Jul 28 04:43:53.761766 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:53.761771    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ece14f8a-56e7-4966-915f-74fed04c43a8-maas-api-tls\") pod \"maas-api-e2e-ait-ecc0dac1-c6d8f487b-8hxk5\" (UID: \"ece14f8a-56e7-4966-915f-74fed04c43a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ecc0dac1-c6d8f487b-8hxk5"
Jul 28 04:43:53.762043 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:53.761895    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ece14f8a-56e7-4966-915f-74fed04c43a8-openshift-service-ca\") pod \"maas-api-e2e-ait-ecc0dac1-c6d8f487b-8hxk5\" (UID: \"ece14f8a-56e7-4966-915f-74fed04c43a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ecc0dac1-c6d8f487b-8hxk5"
Jul 28 04:43:53.863253 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:53.863162    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb4t5\" (UniqueName: \"kubernetes.io/projected/ece14f8a-56e7-4966-915f-74fed04c43a8-kube-api-access-wb4t5\") pod \"maas-api-e2e-ait-ecc0dac1-c6d8f487b-8hxk5\" (UID: \"ece14f8a-56e7-4966-915f-74fed04c43a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ecc0dac1-c6d8f487b-8hxk5"
Jul 28 04:43:53.863253 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:53.863203    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ece14f8a-56e7-4966-915f-74fed04c43a8-maas-api-tls\") pod \"maas-api-e2e-ait-ecc0dac1-c6d8f487b-8hxk5\" (UID: \"ece14f8a-56e7-4966-915f-74fed04c43a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ecc0dac1-c6d8f487b-8hxk5"
Jul 28 04:43:53.863476 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:53.863262    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ece14f8a-56e7-4966-915f-74fed04c43a8-openshift-service-ca\") pod \"maas-api-e2e-ait-ecc0dac1-c6d8f487b-8hxk5\" (UID: \"ece14f8a-56e7-4966-915f-74fed04c43a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ecc0dac1-c6d8f487b-8hxk5"
Jul 28 04:43:53.863476 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:43:53.863373    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-ecc0dac1-serving-cert: secret "maas-api-e2e-ait-ecc0dac1-serving-cert" not found
Jul 28 04:43:53.863476 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:43:53.863451    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ece14f8a-56e7-4966-915f-74fed04c43a8-maas-api-tls podName:ece14f8a-56e7-4966-915f-74fed04c43a8 nodeName:}" failed. No retries permitted until 2026-07-28 04:43:54.363433864 +0000 UTC m=+1934.753609864 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ece14f8a-56e7-4966-915f-74fed04c43a8-maas-api-tls") pod "maas-api-e2e-ait-ecc0dac1-c6d8f487b-8hxk5" (UID: "ece14f8a-56e7-4966-915f-74fed04c43a8") : secret "maas-api-e2e-ait-ecc0dac1-serving-cert" not found
Jul 28 04:43:53.863899 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:53.863881    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ece14f8a-56e7-4966-915f-74fed04c43a8-openshift-service-ca\") pod \"maas-api-e2e-ait-ecc0dac1-c6d8f487b-8hxk5\" (UID: \"ece14f8a-56e7-4966-915f-74fed04c43a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ecc0dac1-c6d8f487b-8hxk5"
Jul 28 04:43:53.872035 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:53.872003    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb4t5\" (UniqueName: \"kubernetes.io/projected/ece14f8a-56e7-4966-915f-74fed04c43a8-kube-api-access-wb4t5\") pod \"maas-api-e2e-ait-ecc0dac1-c6d8f487b-8hxk5\" (UID: \"ece14f8a-56e7-4966-915f-74fed04c43a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ecc0dac1-c6d8f487b-8hxk5"
Jul 28 04:43:54.201102 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:54.201010    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ecc0dac1-84564b7b88-m5hjp"]
Jul 28 04:43:54.204624 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:54.204604    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ecc0dac1-84564b7b88-m5hjp"
Jul 28 04:43:54.207518 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:54.207501    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ecc0dac1-dockercfg-r22tk\""
Jul 28 04:43:54.207916 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:54.207898    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ecc0dac1\""
Jul 28 04:43:54.218043 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:54.218024    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ecc0dac1-84564b7b88-m5hjp"]
Jul 28 04:43:54.222972 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:54.222951    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ecc0dac1-655b9cbbd-25b7x"]
Jul 28 04:43:54.226218 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:54.226198    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ecc0dac1-655b9cbbd-25b7x"
Jul 28 04:43:54.240132 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:54.240110    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ecc0dac1-655b9cbbd-25b7x"]
Jul 28 04:43:54.367970 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:54.367930    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rncrv\" (UniqueName: \"kubernetes.io/projected/b16c9f4b-dc95-49bf-95f2-783575f613bf-kube-api-access-rncrv\") pod \"payload-processing-e2e-ait-ecc0dac1-655b9cbbd-25b7x\" (UID: \"b16c9f4b-dc95-49bf-95f2-783575f613bf\") " pod="openshift-ingress/payload-processing-e2e-ait-ecc0dac1-655b9cbbd-25b7x"
Jul 28 04:43:54.367970 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:54.367972    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcktm\" (UniqueName: \"kubernetes.io/projected/670cf778-047b-473b-8151-152f25dc6915-kube-api-access-vcktm\") pod \"payload-pre-processing-e2e-ait-ecc0dac1-84564b7b88-m5hjp\" (UID: \"670cf778-047b-473b-8151-152f25dc6915\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ecc0dac1-84564b7b88-m5hjp"
Jul 28 04:43:54.368224 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:54.368000    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ece14f8a-56e7-4966-915f-74fed04c43a8-maas-api-tls\") pod \"maas-api-e2e-ait-ecc0dac1-c6d8f487b-8hxk5\" (UID: \"ece14f8a-56e7-4966-915f-74fed04c43a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ecc0dac1-c6d8f487b-8hxk5"
Jul 28 04:43:54.368224 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:54.368102    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/670cf778-047b-473b-8151-152f25dc6915-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ecc0dac1-84564b7b88-m5hjp\" (UID: \"670cf778-047b-473b-8151-152f25dc6915\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ecc0dac1-84564b7b88-m5hjp"
Jul 28 04:43:54.368224 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:54.368151    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b16c9f4b-dc95-49bf-95f2-783575f613bf-plugins-config-volume\") pod \"payload-processing-e2e-ait-ecc0dac1-655b9cbbd-25b7x\" (UID: \"b16c9f4b-dc95-49bf-95f2-783575f613bf\") " pod="openshift-ingress/payload-processing-e2e-ait-ecc0dac1-655b9cbbd-25b7x"
Jul 28 04:43:54.370368 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:54.370340    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ece14f8a-56e7-4966-915f-74fed04c43a8-maas-api-tls\") pod \"maas-api-e2e-ait-ecc0dac1-c6d8f487b-8hxk5\" (UID: \"ece14f8a-56e7-4966-915f-74fed04c43a8\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ecc0dac1-c6d8f487b-8hxk5"
Jul 28 04:43:54.468903 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:54.468819    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rncrv\" (UniqueName: \"kubernetes.io/projected/b16c9f4b-dc95-49bf-95f2-783575f613bf-kube-api-access-rncrv\") pod \"payload-processing-e2e-ait-ecc0dac1-655b9cbbd-25b7x\" (UID: \"b16c9f4b-dc95-49bf-95f2-783575f613bf\") " pod="openshift-ingress/payload-processing-e2e-ait-ecc0dac1-655b9cbbd-25b7x"
Jul 28 04:43:54.468903 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:54.468882    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcktm\" (UniqueName: \"kubernetes.io/projected/670cf778-047b-473b-8151-152f25dc6915-kube-api-access-vcktm\") pod \"payload-pre-processing-e2e-ait-ecc0dac1-84564b7b88-m5hjp\" (UID: \"670cf778-047b-473b-8151-152f25dc6915\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ecc0dac1-84564b7b88-m5hjp"
Jul 28 04:43:54.469084 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:54.468928    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/670cf778-047b-473b-8151-152f25dc6915-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ecc0dac1-84564b7b88-m5hjp\" (UID: \"670cf778-047b-473b-8151-152f25dc6915\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ecc0dac1-84564b7b88-m5hjp"
Jul 28 04:43:54.469084 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:54.468953    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b16c9f4b-dc95-49bf-95f2-783575f613bf-plugins-config-volume\") pod \"payload-processing-e2e-ait-ecc0dac1-655b9cbbd-25b7x\" (UID: \"b16c9f4b-dc95-49bf-95f2-783575f613bf\") " pod="openshift-ingress/payload-processing-e2e-ait-ecc0dac1-655b9cbbd-25b7x"
Jul 28 04:43:54.469664 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:54.469645    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b16c9f4b-dc95-49bf-95f2-783575f613bf-plugins-config-volume\") pod \"payload-processing-e2e-ait-ecc0dac1-655b9cbbd-25b7x\" (UID: \"b16c9f4b-dc95-49bf-95f2-783575f613bf\") " pod="openshift-ingress/payload-processing-e2e-ait-ecc0dac1-655b9cbbd-25b7x"
Jul 28 04:43:54.469703 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:54.469657    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/670cf778-047b-473b-8151-152f25dc6915-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ecc0dac1-84564b7b88-m5hjp\" (UID: \"670cf778-047b-473b-8151-152f25dc6915\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ecc0dac1-84564b7b88-m5hjp"
Jul 28 04:43:54.477686 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:54.477660    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcktm\" (UniqueName: \"kubernetes.io/projected/670cf778-047b-473b-8151-152f25dc6915-kube-api-access-vcktm\") pod \"payload-pre-processing-e2e-ait-ecc0dac1-84564b7b88-m5hjp\" (UID: \"670cf778-047b-473b-8151-152f25dc6915\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ecc0dac1-84564b7b88-m5hjp"
Jul 28 04:43:54.477796 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:54.477704    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rncrv\" (UniqueName: \"kubernetes.io/projected/b16c9f4b-dc95-49bf-95f2-783575f613bf-kube-api-access-rncrv\") pod \"payload-processing-e2e-ait-ecc0dac1-655b9cbbd-25b7x\" (UID: \"b16c9f4b-dc95-49bf-95f2-783575f613bf\") " pod="openshift-ingress/payload-processing-e2e-ait-ecc0dac1-655b9cbbd-25b7x"
Jul 28 04:43:54.505441 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:54.505418    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ecc0dac1-c6d8f487b-8hxk5"
Jul 28 04:43:54.515548 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:54.515529    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ecc0dac1-84564b7b88-m5hjp"
Jul 28 04:43:54.535626 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:54.535600    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ecc0dac1-655b9cbbd-25b7x"
Jul 28 04:43:54.665039 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:54.664864    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ecc0dac1-c6d8f487b-8hxk5"]
Jul 28 04:43:54.667147 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:43:54.667116    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podece14f8a_56e7_4966_915f_74fed04c43a8.slice/crio-46767deddec2f8a9f48293265362ee1f779e8736723c7d9812213df5359592fc WatchSource:0}: Error finding container 46767deddec2f8a9f48293265362ee1f779e8736723c7d9812213df5359592fc: Status 404 returned error can't find the container with id 46767deddec2f8a9f48293265362ee1f779e8736723c7d9812213df5359592fc
Jul 28 04:43:54.668680 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:54.668658    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 04:43:54.714169 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:54.714135    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ecc0dac1-c6d8f487b-8hxk5" event={"ID":"ece14f8a-56e7-4966-915f-74fed04c43a8","Type":"ContainerStarted","Data":"46767deddec2f8a9f48293265362ee1f779e8736723c7d9812213df5359592fc"}
Jul 28 04:43:54.720107 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:54.719960    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ecc0dac1-655b9cbbd-25b7x"]
Jul 28 04:43:54.722681 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:43:54.722659    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb16c9f4b_dc95_49bf_95f2_783575f613bf.slice/crio-ec1c484439c4185b618b9fd9d9b0932fa9301e2029c324506e021995a4467ba1 WatchSource:0}: Error finding container ec1c484439c4185b618b9fd9d9b0932fa9301e2029c324506e021995a4467ba1: Status 404 returned error can't find the container with id ec1c484439c4185b618b9fd9d9b0932fa9301e2029c324506e021995a4467ba1
Jul 28 04:43:54.897143 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:54.897111    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ecc0dac1-84564b7b88-m5hjp"]
Jul 28 04:43:54.899096 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:43:54.899069    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod670cf778_047b_473b_8151_152f25dc6915.slice/crio-1468ac2c8aa242ad223ed8de0c7c9e6bf093c9cbc90818646a07a03e15c5bd6f WatchSource:0}: Error finding container 1468ac2c8aa242ad223ed8de0c7c9e6bf093c9cbc90818646a07a03e15c5bd6f: Status 404 returned error can't find the container with id 1468ac2c8aa242ad223ed8de0c7c9e6bf093c9cbc90818646a07a03e15c5bd6f
Jul 28 04:43:55.722945 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:55.722878    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ecc0dac1-655b9cbbd-25b7x" event={"ID":"b16c9f4b-dc95-49bf-95f2-783575f613bf","Type":"ContainerStarted","Data":"0b8982f23352c4602aa9fe97aeee3da060bc6121a76330f154c8696d051ce9e2"}
Jul 28 04:43:55.723415 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:55.722960    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ecc0dac1-655b9cbbd-25b7x"
Jul 28 04:43:55.723415 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:55.722984    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ecc0dac1-655b9cbbd-25b7x" event={"ID":"b16c9f4b-dc95-49bf-95f2-783575f613bf","Type":"ContainerStarted","Data":"ec1c484439c4185b618b9fd9d9b0932fa9301e2029c324506e021995a4467ba1"}
Jul 28 04:43:55.724996 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:55.724958    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ecc0dac1-84564b7b88-m5hjp" event={"ID":"670cf778-047b-473b-8151-152f25dc6915","Type":"ContainerStarted","Data":"cfa1e4e3776c73c5f814441a004ed56c6ad1533bf2dc5f747d6a2760b7cf8459"}
Jul 28 04:43:55.725127 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:55.725015    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ecc0dac1-84564b7b88-m5hjp"
Jul 28 04:43:55.725127 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:55.725040    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ecc0dac1-84564b7b88-m5hjp" event={"ID":"670cf778-047b-473b-8151-152f25dc6915","Type":"ContainerStarted","Data":"1468ac2c8aa242ad223ed8de0c7c9e6bf093c9cbc90818646a07a03e15c5bd6f"}
Jul 28 04:43:55.738285 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:55.738249    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ecc0dac1-655b9cbbd-25b7x" podStartSLOduration=1.738237904 podStartE2EDuration="1.738237904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:43:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:43:55.737467819 +0000 UTC m=+1936.127643840" watchObservedRunningTime="2026-07-28 04:43:55.738237904 +0000 UTC m=+1936.128413947"
Jul 28 04:43:55.751344 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:55.751300    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ecc0dac1-84564b7b88-m5hjp" podStartSLOduration=1.751286732 podStartE2EDuration="1.751286732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:43:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:43:55.750818882 +0000 UTC m=+1936.140994905" watchObservedRunningTime="2026-07-28 04:43:55.751286732 +0000 UTC m=+1936.141462753"
Jul 28 04:43:56.731385 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:56.731283    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ecc0dac1-c6d8f487b-8hxk5" event={"ID":"ece14f8a-56e7-4966-915f-74fed04c43a8","Type":"ContainerStarted","Data":"1d68a1c1cbead4fa741bd9ba23d664d32553e83e92109d499cb3cc9071f4c056"}
Jul 28 04:43:56.731760 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:56.731675    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ecc0dac1-c6d8f487b-8hxk5"
Jul 28 04:43:56.748765 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:43:56.748716    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ecc0dac1-c6d8f487b-8hxk5" podStartSLOduration=2.136867865 podStartE2EDuration="3.748704758s" totalImagesPullingTime="1.611836893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:43:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:43:56.746736825 +0000 UTC m=+1937.136912845" watchObservedRunningTime="2026-07-28 04:43:56.748704758 +0000 UTC m=+1937.138880778"
Jul 28 04:44:03.747603 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:03.747575    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ecc0dac1-c6d8f487b-8hxk5"
Jul 28 04:44:06.733514 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:06.733480    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ecc0dac1-655b9cbbd-25b7x"
Jul 28 04:44:06.733929 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:06.733555    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ecc0dac1-84564b7b88-m5hjp"
Jul 28 04:44:29.636467 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:29.636424    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ecc0dac1-c6d8f487b-8hxk5"]
Jul 28 04:44:29.637156 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:29.636646    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ecc0dac1-c6d8f487b-8hxk5" podUID="ece14f8a-56e7-4966-915f-74fed04c43a8" containerName="maas-api" containerID="cri-o://1d68a1c1cbead4fa741bd9ba23d664d32553e83e92109d499cb3cc9071f4c056" gracePeriod=30
Jul 28 04:44:29.693377 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:29.693346    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ecc0dac1-655b9cbbd-25b7x"]
Jul 28 04:44:29.693618 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:29.693574    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ecc0dac1-655b9cbbd-25b7x" podUID="b16c9f4b-dc95-49bf-95f2-783575f613bf" containerName="payload-processing" containerID="cri-o://0b8982f23352c4602aa9fe97aeee3da060bc6121a76330f154c8696d051ce9e2" gracePeriod=30
Jul 28 04:44:29.699196 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:29.699172    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ecc0dac1-84564b7b88-m5hjp"]
Jul 28 04:44:29.699400 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:29.699375    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ecc0dac1-84564b7b88-m5hjp" podUID="670cf778-047b-473b-8151-152f25dc6915" containerName="payload-pre-processing" containerID="cri-o://cfa1e4e3776c73c5f814441a004ed56c6ad1533bf2dc5f747d6a2760b7cf8459" gracePeriod=30
Jul 28 04:44:29.898379 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:29.898294    2566 generic.go:356] "Generic (PLEG): container finished" podID="b16c9f4b-dc95-49bf-95f2-783575f613bf" containerID="0b8982f23352c4602aa9fe97aeee3da060bc6121a76330f154c8696d051ce9e2" exitCode=0
Jul 28 04:44:29.898379 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:29.898359    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ecc0dac1-655b9cbbd-25b7x" event={"ID":"b16c9f4b-dc95-49bf-95f2-783575f613bf","Type":"ContainerDied","Data":"0b8982f23352c4602aa9fe97aeee3da060bc6121a76330f154c8696d051ce9e2"}
Jul 28 04:44:29.900041 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:29.900016    2566 generic.go:356] "Generic (PLEG): container finished" podID="ece14f8a-56e7-4966-915f-74fed04c43a8" containerID="1d68a1c1cbead4fa741bd9ba23d664d32553e83e92109d499cb3cc9071f4c056" exitCode=0
Jul 28 04:44:29.900171 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:29.900086    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ecc0dac1-c6d8f487b-8hxk5" event={"ID":"ece14f8a-56e7-4966-915f-74fed04c43a8","Type":"ContainerDied","Data":"1d68a1c1cbead4fa741bd9ba23d664d32553e83e92109d499cb3cc9071f4c056"}
Jul 28 04:44:29.901984 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:29.901964    2566 generic.go:356] "Generic (PLEG): container finished" podID="670cf778-047b-473b-8151-152f25dc6915" containerID="cfa1e4e3776c73c5f814441a004ed56c6ad1533bf2dc5f747d6a2760b7cf8459" exitCode=0
Jul 28 04:44:29.902077 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:29.902032    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ecc0dac1-84564b7b88-m5hjp" event={"ID":"670cf778-047b-473b-8151-152f25dc6915","Type":"ContainerDied","Data":"cfa1e4e3776c73c5f814441a004ed56c6ad1533bf2dc5f747d6a2760b7cf8459"}
Jul 28 04:44:29.953971 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:29.953938    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ecc0dac1-c6d8f487b-8hxk5"
Jul 28 04:44:29.993928 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:29.993902    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ecc0dac1-655b9cbbd-25b7x"
Jul 28 04:44:29.997294 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:29.997267    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ecc0dac1-84564b7b88-m5hjp"
Jul 28 04:44:30.097581 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:30.097551    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rncrv\" (UniqueName: \"kubernetes.io/projected/b16c9f4b-dc95-49bf-95f2-783575f613bf-kube-api-access-rncrv\") pod \"b16c9f4b-dc95-49bf-95f2-783575f613bf\" (UID: \"b16c9f4b-dc95-49bf-95f2-783575f613bf\") "
Jul 28 04:44:30.097740 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:30.097596    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b16c9f4b-dc95-49bf-95f2-783575f613bf-plugins-config-volume\") pod \"b16c9f4b-dc95-49bf-95f2-783575f613bf\" (UID: \"b16c9f4b-dc95-49bf-95f2-783575f613bf\") "
Jul 28 04:44:30.097740 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:30.097622    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wb4t5\" (UniqueName: \"kubernetes.io/projected/ece14f8a-56e7-4966-915f-74fed04c43a8-kube-api-access-wb4t5\") pod \"ece14f8a-56e7-4966-915f-74fed04c43a8\" (UID: \"ece14f8a-56e7-4966-915f-74fed04c43a8\") "
Jul 28 04:44:30.097740 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:30.097637    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ece14f8a-56e7-4966-915f-74fed04c43a8-openshift-service-ca\") pod \"ece14f8a-56e7-4966-915f-74fed04c43a8\" (UID: \"ece14f8a-56e7-4966-915f-74fed04c43a8\") "
Jul 28 04:44:30.097740 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:30.097679    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ece14f8a-56e7-4966-915f-74fed04c43a8-maas-api-tls\") pod \"ece14f8a-56e7-4966-915f-74fed04c43a8\" (UID: \"ece14f8a-56e7-4966-915f-74fed04c43a8\") "
Jul 28 04:44:30.097740 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:30.097733    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vcktm\" (UniqueName: \"kubernetes.io/projected/670cf778-047b-473b-8151-152f25dc6915-kube-api-access-vcktm\") pod \"670cf778-047b-473b-8151-152f25dc6915\" (UID: \"670cf778-047b-473b-8151-152f25dc6915\") "
Jul 28 04:44:30.098051 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:30.097782    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/670cf778-047b-473b-8151-152f25dc6915-plugins-config-volume\") pod \"670cf778-047b-473b-8151-152f25dc6915\" (UID: \"670cf778-047b-473b-8151-152f25dc6915\") "
Jul 28 04:44:30.098111 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:30.098054    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b16c9f4b-dc95-49bf-95f2-783575f613bf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b16c9f4b-dc95-49bf-95f2-783575f613bf" (UID: "b16c9f4b-dc95-49bf-95f2-783575f613bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 04:44:30.098164 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:30.098109    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ece14f8a-56e7-4966-915f-74fed04c43a8-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ece14f8a-56e7-4966-915f-74fed04c43a8" (UID: "ece14f8a-56e7-4966-915f-74fed04c43a8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 04:44:30.098221 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:30.098194    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b16c9f4b-dc95-49bf-95f2-783575f613bf-plugins-config-volume\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:44:30.098221 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:30.098213    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ece14f8a-56e7-4966-915f-74fed04c43a8-openshift-service-ca\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:44:30.098489 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:30.098441    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/670cf778-047b-473b-8151-152f25dc6915-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "670cf778-047b-473b-8151-152f25dc6915" (UID: "670cf778-047b-473b-8151-152f25dc6915"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 04:44:30.100086 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:30.100063    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ece14f8a-56e7-4966-915f-74fed04c43a8-kube-api-access-wb4t5" (OuterVolumeSpecName: "kube-api-access-wb4t5") pod "ece14f8a-56e7-4966-915f-74fed04c43a8" (UID: "ece14f8a-56e7-4966-915f-74fed04c43a8"). InnerVolumeSpecName "kube-api-access-wb4t5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:44:30.100191 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:30.100090    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/670cf778-047b-473b-8151-152f25dc6915-kube-api-access-vcktm" (OuterVolumeSpecName: "kube-api-access-vcktm") pod "670cf778-047b-473b-8151-152f25dc6915" (UID: "670cf778-047b-473b-8151-152f25dc6915"). InnerVolumeSpecName "kube-api-access-vcktm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:44:30.100191 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:30.100160    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ece14f8a-56e7-4966-915f-74fed04c43a8-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ece14f8a-56e7-4966-915f-74fed04c43a8" (UID: "ece14f8a-56e7-4966-915f-74fed04c43a8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 04:44:30.100290 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:30.100205    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b16c9f4b-dc95-49bf-95f2-783575f613bf-kube-api-access-rncrv" (OuterVolumeSpecName: "kube-api-access-rncrv") pod "b16c9f4b-dc95-49bf-95f2-783575f613bf" (UID: "b16c9f4b-dc95-49bf-95f2-783575f613bf"). InnerVolumeSpecName "kube-api-access-rncrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:44:30.199382 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:30.199285    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb4t5\" (UniqueName: \"kubernetes.io/projected/ece14f8a-56e7-4966-915f-74fed04c43a8-kube-api-access-wb4t5\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:44:30.199382 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:30.199317    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ece14f8a-56e7-4966-915f-74fed04c43a8-maas-api-tls\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:44:30.199382 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:30.199328    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcktm\" (UniqueName: \"kubernetes.io/projected/670cf778-047b-473b-8151-152f25dc6915-kube-api-access-vcktm\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:44:30.199382 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:30.199336    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/670cf778-047b-473b-8151-152f25dc6915-plugins-config-volume\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:44:30.199382 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:30.199345    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rncrv\" (UniqueName: \"kubernetes.io/projected/b16c9f4b-dc95-49bf-95f2-783575f613bf-kube-api-access-rncrv\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:44:30.907826 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:30.907744    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ecc0dac1-655b9cbbd-25b7x"
Jul 28 04:44:30.907826 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:30.907791    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ecc0dac1-655b9cbbd-25b7x" event={"ID":"b16c9f4b-dc95-49bf-95f2-783575f613bf","Type":"ContainerDied","Data":"ec1c484439c4185b618b9fd9d9b0932fa9301e2029c324506e021995a4467ba1"}
Jul 28 04:44:30.908365 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:30.907888    2566 scope.go:117] "RemoveContainer" containerID="0b8982f23352c4602aa9fe97aeee3da060bc6121a76330f154c8696d051ce9e2"
Jul 28 04:44:30.909321 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:30.909293    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ecc0dac1-c6d8f487b-8hxk5" event={"ID":"ece14f8a-56e7-4966-915f-74fed04c43a8","Type":"ContainerDied","Data":"46767deddec2f8a9f48293265362ee1f779e8736723c7d9812213df5359592fc"}
Jul 28 04:44:30.909321 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:30.909316    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ecc0dac1-c6d8f487b-8hxk5"
Jul 28 04:44:30.910745 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:30.910718    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ecc0dac1-84564b7b88-m5hjp" event={"ID":"670cf778-047b-473b-8151-152f25dc6915","Type":"ContainerDied","Data":"1468ac2c8aa242ad223ed8de0c7c9e6bf093c9cbc90818646a07a03e15c5bd6f"}
Jul 28 04:44:30.910745 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:30.910728    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ecc0dac1-84564b7b88-m5hjp"
Jul 28 04:44:30.917053 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:30.917025    2566 scope.go:117] "RemoveContainer" containerID="1d68a1c1cbead4fa741bd9ba23d664d32553e83e92109d499cb3cc9071f4c056"
Jul 28 04:44:30.926671 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:30.926651    2566 scope.go:117] "RemoveContainer" containerID="cfa1e4e3776c73c5f814441a004ed56c6ad1533bf2dc5f747d6a2760b7cf8459"
Jul 28 04:44:30.936105 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:30.936079    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ecc0dac1-655b9cbbd-25b7x"]
Jul 28 04:44:30.937808 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:30.937788    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ecc0dac1-655b9cbbd-25b7x"]
Jul 28 04:44:30.947374 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:30.947355    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ecc0dac1-c6d8f487b-8hxk5"]
Jul 28 04:44:30.949065 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:30.949047    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ecc0dac1-c6d8f487b-8hxk5"]
Jul 28 04:44:30.958333 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:30.958304    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ecc0dac1-84564b7b88-m5hjp"]
Jul 28 04:44:30.961624 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:30.961606    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ecc0dac1-84564b7b88-m5hjp"]
Jul 28 04:44:32.089082 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:32.089001    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="670cf778-047b-473b-8151-152f25dc6915" path="/var/lib/kubelet/pods/670cf778-047b-473b-8151-152f25dc6915/volumes"
Jul 28 04:44:32.089421 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:32.089403    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b16c9f4b-dc95-49bf-95f2-783575f613bf" path="/var/lib/kubelet/pods/b16c9f4b-dc95-49bf-95f2-783575f613bf/volumes"
Jul 28 04:44:32.089742 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:32.089729    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ece14f8a-56e7-4966-915f-74fed04c43a8" path="/var/lib/kubelet/pods/ece14f8a-56e7-4966-915f-74fed04c43a8/volumes"
Jul 28 04:44:35.384635 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:35.384595    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn"]
Jul 28 04:44:35.385219 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:35.385193    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b16c9f4b-dc95-49bf-95f2-783575f613bf" containerName="payload-processing"
Jul 28 04:44:35.385219 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:35.385217    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b16c9f4b-dc95-49bf-95f2-783575f613bf" containerName="payload-processing"
Jul 28 04:44:35.385341 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:35.385226    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="670cf778-047b-473b-8151-152f25dc6915" containerName="payload-pre-processing"
Jul 28 04:44:35.385341 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:35.385234    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="670cf778-047b-473b-8151-152f25dc6915" containerName="payload-pre-processing"
Jul 28 04:44:35.385341 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:35.385245    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ece14f8a-56e7-4966-915f-74fed04c43a8" containerName="maas-api"
Jul 28 04:44:35.385341 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:35.385254    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ece14f8a-56e7-4966-915f-74fed04c43a8" containerName="maas-api"
Jul 28 04:44:35.385473 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:35.385368    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="670cf778-047b-473b-8151-152f25dc6915" containerName="payload-pre-processing"
Jul 28 04:44:35.385473 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:35.385379    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ece14f8a-56e7-4966-915f-74fed04c43a8" containerName="maas-api"
Jul 28 04:44:35.385473 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:35.385387    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b16c9f4b-dc95-49bf-95f2-783575f613bf" containerName="payload-processing"
Jul 28 04:44:35.391333 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:35.391313    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn"
Jul 28 04:44:35.393619 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:35.393598    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 28 04:44:35.393712 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:35.393600    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-5f6a9856-gw-openshift-default-dockercfg-9r2pk\""
Jul 28 04:44:35.402626 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:35.402602    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn"]
Jul 28 04:44:35.544361 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:35.544330    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-credential-socket\") pod \"e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn\" (UID: \"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c\") " pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn"
Jul 28 04:44:35.544555 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:35.544377    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-istio-data\") pod \"e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn\" (UID: \"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c\") " pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn"
Jul 28 04:44:35.544555 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:35.544397    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dwrt\" (UniqueName: \"kubernetes.io/projected/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-kube-api-access-2dwrt\") pod \"e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn\" (UID: \"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c\") " pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn"
Jul 28 04:44:35.544555 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:35.544424    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-istio-envoy\") pod \"e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn\" (UID: \"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c\") " pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn"
Jul 28 04:44:35.544555 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:35.544485    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-workload-certs\") pod \"e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn\" (UID: \"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c\") " pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn"
Jul 28 04:44:35.544555 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:35.544529    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-istio-token\") pod \"e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn\" (UID: \"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c\") " pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn"
Jul 28 04:44:35.544780 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:35.544571    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-workload-socket\") pod \"e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn\" (UID: \"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c\") " pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn"
Jul 28 04:44:35.544780 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:35.544625    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-istiod-ca-cert\") pod \"e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn\" (UID: \"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c\") " pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn"
Jul 28 04:44:35.544780 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:35.544664    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-istio-podinfo\") pod \"e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn\" (UID: \"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c\") " pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn"
Jul 28 04:44:35.646164 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:35.646073    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-istio-data\") pod \"e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn\" (UID: \"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c\") " pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn"
Jul 28 04:44:35.646164 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:35.646117    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dwrt\" (UniqueName: \"kubernetes.io/projected/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-kube-api-access-2dwrt\") pod \"e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn\" (UID: \"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c\") " pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn"
Jul 28 04:44:35.646380 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:35.646242    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-istio-envoy\") pod \"e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn\" (UID: \"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c\") " pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn"
Jul 28 04:44:35.646380 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:35.646301    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-workload-certs\") pod \"e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn\" (UID: \"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c\") " pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn"
Jul 28 04:44:35.646380 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:35.646340    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-istio-token\") pod \"e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn\" (UID: \"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c\") " pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn"
Jul 28 04:44:35.646542 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:35.646407    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-workload-socket\") pod \"e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn\" (UID: \"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c\") " pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn"
Jul 28 04:44:35.646542 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:35.646476    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-istiod-ca-cert\") pod \"e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn\" (UID: \"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c\") " pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn"
Jul 28 04:44:35.646542 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:35.646489    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-istio-data\") pod \"e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn\" (UID: \"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c\") " pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn"
Jul 28 04:44:35.646699 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:35.646540    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-istio-podinfo\") pod \"e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn\" (UID: \"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c\") " pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn"
Jul 28 04:44:35.646699 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:35.646603    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-credential-socket\") pod \"e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn\" (UID: \"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c\") " pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn"
Jul 28 04:44:35.646812 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:35.646760    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-workload-certs\") pod \"e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn\" (UID: \"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c\") " pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn"
Jul 28 04:44:35.646812 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:35.646772    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-workload-socket\") pod \"e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn\" (UID: \"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c\") " pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn"
Jul 28 04:44:35.647011 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:35.646992    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-credential-socket\") pod \"e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn\" (UID: \"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c\") " pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn"
Jul 28 04:44:35.647114 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:35.647099    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-istiod-ca-cert\") pod \"e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn\" (UID: \"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c\") " pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn"
Jul 28 04:44:35.648469 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:35.648445    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-istio-envoy\") pod \"e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn\" (UID: \"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c\") " pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn"
Jul 28 04:44:35.649244 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:35.649224    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-istio-podinfo\") pod \"e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn\" (UID: \"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c\") " pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn"
Jul 28 04:44:35.653522 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:35.653495    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-istio-token\") pod \"e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn\" (UID: \"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c\") " pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn"
Jul 28 04:44:35.653667 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:35.653651    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dwrt\" (UniqueName: \"kubernetes.io/projected/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-kube-api-access-2dwrt\") pod \"e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn\" (UID: \"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c\") " pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn"
Jul 28 04:44:35.702574 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:35.702549    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn"
Jul 28 04:44:35.844645 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:35.844623    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn"]
Jul 28 04:44:35.938613 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:35.938541    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn" event={"ID":"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c","Type":"ContainerStarted","Data":"a3fb682ae6d65cda989908a0212e17e532011412b9f61ca52f302151c5627bfb"}
Jul 28 04:44:38.539957 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:38.539920    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:44:38.540220 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:38.539990    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:44:38.540220 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:38.540021    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:44:38.956069 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:38.955975    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn" event={"ID":"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c","Type":"ContainerStarted","Data":"ae84c3fde53e0bb864affc8a944a83019bd72755a6950c3a0583775374fda583"}
Jul 28 04:44:38.976403 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:38.976348    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn" podStartSLOduration=1.287661464 podStartE2EDuration="3.976333101s" totalImagesPullingTime="2.688671637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:44:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:44:38.974633578 +0000 UTC m=+1979.364809600" watchObservedRunningTime="2026-07-28 04:44:38.976333101 +0000 UTC m=+1979.366509122"
Jul 28 04:44:39.703090 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:39.703050    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn"
Jul 28 04:44:39.709108 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:39.709085    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn"
Jul 28 04:44:39.960610 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:39.960522    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn"
Jul 28 04:44:39.961453 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:39.961430    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn"
Jul 28 04:44:40.538784 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:40.538743    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5f6a9856-596d9b5c9-qj4fp"]
Jul 28 04:44:40.544411 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:40.544390    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f6a9856-596d9b5c9-qj4fp"
Jul 28 04:44:40.546665 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:40.546641    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-5f6a9856-serving-cert\""
Jul 28 04:44:40.546852 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:40.546812    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 04:44:40.548008 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:40.547863    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gfn77\""
Jul 28 04:44:40.548008 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:40.547921    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 04:44:40.551017 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:40.550984    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5f6a9856-596d9b5c9-qj4fp"]
Jul 28 04:44:40.695405 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:40.695371    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fe68cba-f10d-40a3-839c-7ccd72abbd32-openshift-service-ca\") pod \"maas-api-e2e-migrate-5f6a9856-596d9b5c9-qj4fp\" (UID: \"1fe68cba-f10d-40a3-839c-7ccd72abbd32\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f6a9856-596d9b5c9-qj4fp"
Jul 28 04:44:40.695558 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:40.695417    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fe68cba-f10d-40a3-839c-7ccd72abbd32-maas-api-tls\") pod \"maas-api-e2e-migrate-5f6a9856-596d9b5c9-qj4fp\" (UID: \"1fe68cba-f10d-40a3-839c-7ccd72abbd32\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f6a9856-596d9b5c9-qj4fp"
Jul 28 04:44:40.695558 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:40.695531    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pl4j\" (UniqueName: \"kubernetes.io/projected/1fe68cba-f10d-40a3-839c-7ccd72abbd32-kube-api-access-7pl4j\") pod \"maas-api-e2e-migrate-5f6a9856-596d9b5c9-qj4fp\" (UID: \"1fe68cba-f10d-40a3-839c-7ccd72abbd32\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f6a9856-596d9b5c9-qj4fp"
Jul 28 04:44:40.796635 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:40.796542    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fe68cba-f10d-40a3-839c-7ccd72abbd32-openshift-service-ca\") pod \"maas-api-e2e-migrate-5f6a9856-596d9b5c9-qj4fp\" (UID: \"1fe68cba-f10d-40a3-839c-7ccd72abbd32\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f6a9856-596d9b5c9-qj4fp"
Jul 28 04:44:40.796635 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:40.796592    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fe68cba-f10d-40a3-839c-7ccd72abbd32-maas-api-tls\") pod \"maas-api-e2e-migrate-5f6a9856-596d9b5c9-qj4fp\" (UID: \"1fe68cba-f10d-40a3-839c-7ccd72abbd32\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f6a9856-596d9b5c9-qj4fp"
Jul 28 04:44:40.797097 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:40.796703    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pl4j\" (UniqueName: \"kubernetes.io/projected/1fe68cba-f10d-40a3-839c-7ccd72abbd32-kube-api-access-7pl4j\") pod \"maas-api-e2e-migrate-5f6a9856-596d9b5c9-qj4fp\" (UID: \"1fe68cba-f10d-40a3-839c-7ccd72abbd32\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f6a9856-596d9b5c9-qj4fp"
Jul 28 04:44:40.797288 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:40.797266    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fe68cba-f10d-40a3-839c-7ccd72abbd32-openshift-service-ca\") pod \"maas-api-e2e-migrate-5f6a9856-596d9b5c9-qj4fp\" (UID: \"1fe68cba-f10d-40a3-839c-7ccd72abbd32\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f6a9856-596d9b5c9-qj4fp"
Jul 28 04:44:40.799129 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:40.799109    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fe68cba-f10d-40a3-839c-7ccd72abbd32-maas-api-tls\") pod \"maas-api-e2e-migrate-5f6a9856-596d9b5c9-qj4fp\" (UID: \"1fe68cba-f10d-40a3-839c-7ccd72abbd32\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f6a9856-596d9b5c9-qj4fp"
Jul 28 04:44:40.804108 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:40.804079    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pl4j\" (UniqueName: \"kubernetes.io/projected/1fe68cba-f10d-40a3-839c-7ccd72abbd32-kube-api-access-7pl4j\") pod \"maas-api-e2e-migrate-5f6a9856-596d9b5c9-qj4fp\" (UID: \"1fe68cba-f10d-40a3-839c-7ccd72abbd32\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f6a9856-596d9b5c9-qj4fp"
Jul 28 04:44:40.857288 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:40.857257    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f6a9856-596d9b5c9-qj4fp"
Jul 28 04:44:40.994120 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:40.994093    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5f6a9856-596d9b5c9-qj4fp"]
Jul 28 04:44:40.996266 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:44:40.996232    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fe68cba_f10d_40a3_839c_7ccd72abbd32.slice/crio-31ba599a8703e054f029b9357b7c0f2b5ebc5bcafa92b97160670f16323b489b WatchSource:0}: Error finding container 31ba599a8703e054f029b9357b7c0f2b5ebc5bcafa92b97160670f16323b489b: Status 404 returned error can't find the container with id 31ba599a8703e054f029b9357b7c0f2b5ebc5bcafa92b97160670f16323b489b
Jul 28 04:44:41.267098 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:41.267070    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5f6a9856-5c4d6ddb48-jgvjp"]
Jul 28 04:44:41.271822 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:41.271792    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5f6a9856-5c4d6ddb48-jgvjp"
Jul 28 04:44:41.274146 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:41.274126    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-5f6a9856\""
Jul 28 04:44:41.274267 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:41.274164    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-5f6a9856-dockercfg-g98v7\""
Jul 28 04:44:41.281003 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:41.280982    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5f6a9856-5c4d6ddb48-jgvjp"]
Jul 28 04:44:41.299485 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:41.299457    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5f6a9856-978b56c5c-nsvdd"]
Jul 28 04:44:41.302961 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:41.302942    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-5f6a9856-978b56c5c-nsvdd"
Jul 28 04:44:41.315274 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:41.315248    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5f6a9856-978b56c5c-nsvdd"]
Jul 28 04:44:41.402148 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:41.402112    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6dsc\" (UniqueName: \"kubernetes.io/projected/c0ee5a80-273c-4244-a5d8-9854770d5a7b-kube-api-access-l6dsc\") pod \"payload-pre-processing-e2e-migrate-5f6a9856-5c4d6ddb48-jgvjp\" (UID: \"c0ee5a80-273c-4244-a5d8-9854770d5a7b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5f6a9856-5c4d6ddb48-jgvjp"
Jul 28 04:44:41.402320 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:41.402289    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7pzv\" (UniqueName: \"kubernetes.io/projected/e6f6dbdd-1439-4776-a6ac-e7fbf285ce82-kube-api-access-k7pzv\") pod \"payload-processing-e2e-migrate-5f6a9856-978b56c5c-nsvdd\" (UID: \"e6f6dbdd-1439-4776-a6ac-e7fbf285ce82\") " pod="openshift-ingress/payload-processing-e2e-migrate-5f6a9856-978b56c5c-nsvdd"
Jul 28 04:44:41.402381 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:41.402319    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6f6dbdd-1439-4776-a6ac-e7fbf285ce82-plugins-config-volume\") pod \"payload-processing-e2e-migrate-5f6a9856-978b56c5c-nsvdd\" (UID: \"e6f6dbdd-1439-4776-a6ac-e7fbf285ce82\") " pod="openshift-ingress/payload-processing-e2e-migrate-5f6a9856-978b56c5c-nsvdd"
Jul 28 04:44:41.402381 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:41.402352    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0ee5a80-273c-4244-a5d8-9854770d5a7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-5f6a9856-5c4d6ddb48-jgvjp\" (UID: \"c0ee5a80-273c-4244-a5d8-9854770d5a7b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5f6a9856-5c4d6ddb48-jgvjp"
Jul 28 04:44:41.503635 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:41.503601    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7pzv\" (UniqueName: \"kubernetes.io/projected/e6f6dbdd-1439-4776-a6ac-e7fbf285ce82-kube-api-access-k7pzv\") pod \"payload-processing-e2e-migrate-5f6a9856-978b56c5c-nsvdd\" (UID: \"e6f6dbdd-1439-4776-a6ac-e7fbf285ce82\") " pod="openshift-ingress/payload-processing-e2e-migrate-5f6a9856-978b56c5c-nsvdd"
Jul 28 04:44:41.503635 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:41.503639    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6f6dbdd-1439-4776-a6ac-e7fbf285ce82-plugins-config-volume\") pod \"payload-processing-e2e-migrate-5f6a9856-978b56c5c-nsvdd\" (UID: \"e6f6dbdd-1439-4776-a6ac-e7fbf285ce82\") " pod="openshift-ingress/payload-processing-e2e-migrate-5f6a9856-978b56c5c-nsvdd"
Jul 28 04:44:41.503954 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:41.503666    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0ee5a80-273c-4244-a5d8-9854770d5a7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-5f6a9856-5c4d6ddb48-jgvjp\" (UID: \"c0ee5a80-273c-4244-a5d8-9854770d5a7b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5f6a9856-5c4d6ddb48-jgvjp"
Jul 28 04:44:41.503954 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:41.503715    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6dsc\" (UniqueName: \"kubernetes.io/projected/c0ee5a80-273c-4244-a5d8-9854770d5a7b-kube-api-access-l6dsc\") pod \"payload-pre-processing-e2e-migrate-5f6a9856-5c4d6ddb48-jgvjp\" (UID: \"c0ee5a80-273c-4244-a5d8-9854770d5a7b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5f6a9856-5c4d6ddb48-jgvjp"
Jul 28 04:44:41.504371 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:41.504348    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6f6dbdd-1439-4776-a6ac-e7fbf285ce82-plugins-config-volume\") pod \"payload-processing-e2e-migrate-5f6a9856-978b56c5c-nsvdd\" (UID: \"e6f6dbdd-1439-4776-a6ac-e7fbf285ce82\") " pod="openshift-ingress/payload-processing-e2e-migrate-5f6a9856-978b56c5c-nsvdd"
Jul 28 04:44:41.504688 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:41.504665    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0ee5a80-273c-4244-a5d8-9854770d5a7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-5f6a9856-5c4d6ddb48-jgvjp\" (UID: \"c0ee5a80-273c-4244-a5d8-9854770d5a7b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5f6a9856-5c4d6ddb48-jgvjp"
Jul 28 04:44:41.511302 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:41.511268    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7pzv\" (UniqueName: \"kubernetes.io/projected/e6f6dbdd-1439-4776-a6ac-e7fbf285ce82-kube-api-access-k7pzv\") pod \"payload-processing-e2e-migrate-5f6a9856-978b56c5c-nsvdd\" (UID: \"e6f6dbdd-1439-4776-a6ac-e7fbf285ce82\") " pod="openshift-ingress/payload-processing-e2e-migrate-5f6a9856-978b56c5c-nsvdd"
Jul 28 04:44:41.511523 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:41.511495    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6dsc\" (UniqueName: \"kubernetes.io/projected/c0ee5a80-273c-4244-a5d8-9854770d5a7b-kube-api-access-l6dsc\") pod \"payload-pre-processing-e2e-migrate-5f6a9856-5c4d6ddb48-jgvjp\" (UID: \"c0ee5a80-273c-4244-a5d8-9854770d5a7b\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5f6a9856-5c4d6ddb48-jgvjp"
Jul 28 04:44:41.585548 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:41.585442    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5f6a9856-5c4d6ddb48-jgvjp"
Jul 28 04:44:41.614174 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:41.614143    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-5f6a9856-978b56c5c-nsvdd"
Jul 28 04:44:41.942450 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:41.942423    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5f6a9856-5c4d6ddb48-jgvjp"]
Jul 28 04:44:41.944701 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:44:41.944663    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0ee5a80_273c_4244_a5d8_9854770d5a7b.slice/crio-50f0087e22779d4a47a012a49e2f11508b076fff4958ccf3d35dfd42b64d4bcb WatchSource:0}: Error finding container 50f0087e22779d4a47a012a49e2f11508b076fff4958ccf3d35dfd42b64d4bcb: Status 404 returned error can't find the container with id 50f0087e22779d4a47a012a49e2f11508b076fff4958ccf3d35dfd42b64d4bcb
Jul 28 04:44:41.973570 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:41.973539    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5f6a9856-5c4d6ddb48-jgvjp" event={"ID":"c0ee5a80-273c-4244-a5d8-9854770d5a7b","Type":"ContainerStarted","Data":"50f0087e22779d4a47a012a49e2f11508b076fff4958ccf3d35dfd42b64d4bcb"}
Jul 28 04:44:41.974699 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:41.974677    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5f6a9856-978b56c5c-nsvdd"]
Jul 28 04:44:41.975261 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:41.975229    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f6a9856-596d9b5c9-qj4fp" event={"ID":"1fe68cba-f10d-40a3-839c-7ccd72abbd32","Type":"ContainerStarted","Data":"b93bbe96da1ca99e938680ceb541d362651a37b1ba1ee5c54b76f7b3b3319f7f"}
Jul 28 04:44:41.975340 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:41.975281    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f6a9856-596d9b5c9-qj4fp" event={"ID":"1fe68cba-f10d-40a3-839c-7ccd72abbd32","Type":"ContainerStarted","Data":"31ba599a8703e054f029b9357b7c0f2b5ebc5bcafa92b97160670f16323b489b"}
Jul 28 04:44:41.980957 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:44:41.980920    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6f6dbdd_1439_4776_a6ac_e7fbf285ce82.slice/crio-e3b3e1fc3144bcdc512e50fec73da2a79d33e96d710db9c83ea13c0bd57b7cc9 WatchSource:0}: Error finding container e3b3e1fc3144bcdc512e50fec73da2a79d33e96d710db9c83ea13c0bd57b7cc9: Status 404 returned error can't find the container with id e3b3e1fc3144bcdc512e50fec73da2a79d33e96d710db9c83ea13c0bd57b7cc9
Jul 28 04:44:41.991354 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:41.991304    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f6a9856-596d9b5c9-qj4fp" podStartSLOduration=1.559130683 podStartE2EDuration="1.991289795s" totalImagesPullingTime="432.159112ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:44:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:44:41.990229751 +0000 UTC m=+1982.380405771" watchObservedRunningTime="2026-07-28 04:44:41.991289795 +0000 UTC m=+1982.381465815"
Jul 28 04:44:42.981400 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:42.981360    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5f6a9856-5c4d6ddb48-jgvjp" event={"ID":"c0ee5a80-273c-4244-a5d8-9854770d5a7b","Type":"ContainerStarted","Data":"e4ef10234a8dcb269b57a37bea3fd6e4f4575997a354cf7bd738ba9e2ed75ab0"}
Jul 28 04:44:42.981900 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:42.981554    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5f6a9856-5c4d6ddb48-jgvjp"
Jul 28 04:44:42.982949 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:42.982928    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-5f6a9856-978b56c5c-nsvdd" event={"ID":"e6f6dbdd-1439-4776-a6ac-e7fbf285ce82","Type":"ContainerStarted","Data":"c59151e183aad82bd470e631aae70a23c14386a86951fc0e9b322776abea9ff7"}
Jul 28 04:44:42.983059 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:42.982965    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-5f6a9856-978b56c5c-nsvdd" event={"ID":"e6f6dbdd-1439-4776-a6ac-e7fbf285ce82","Type":"ContainerStarted","Data":"e3b3e1fc3144bcdc512e50fec73da2a79d33e96d710db9c83ea13c0bd57b7cc9"}
Jul 28 04:44:42.983059 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:42.982984    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-5f6a9856-978b56c5c-nsvdd"
Jul 28 04:44:42.983159 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:42.983149    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f6a9856-596d9b5c9-qj4fp"
Jul 28 04:44:42.997649 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:42.997605    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5f6a9856-5c4d6ddb48-jgvjp" podStartSLOduration=1.997589581 podStartE2EDuration="1.997589581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:44:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:44:42.995773895 +0000 UTC m=+1983.385949916" watchObservedRunningTime="2026-07-28 04:44:42.997589581 +0000 UTC m=+1983.387765603"
Jul 28 04:44:43.013207 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:43.013152    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-5f6a9856-978b56c5c-nsvdd" podStartSLOduration=2.013136314 podStartE2EDuration="2.013136314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:44:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:44:43.011806662 +0000 UTC m=+1983.401982684" watchObservedRunningTime="2026-07-28 04:44:43.013136314 +0000 UTC m=+1983.403312336"
Jul 28 04:44:48.992819 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:48.992785    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f6a9856-596d9b5c9-qj4fp"
Jul 28 04:44:52.029595 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:52.029523    2566 generic.go:356] "Generic (PLEG): container finished" podID="bc1dbec2-43ef-43fd-85bb-f95f8b55e64c" containerID="ae84c3fde53e0bb864affc8a944a83019bd72755a6950c3a0583775374fda583" exitCode=0
Jul 28 04:44:52.030023 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:52.029595    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn" event={"ID":"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c","Type":"ContainerDied","Data":"ae84c3fde53e0bb864affc8a944a83019bd72755a6950c3a0583775374fda583"}
Jul 28 04:44:52.030068 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:52.030050    2566 scope.go:117] "RemoveContainer" containerID="ae84c3fde53e0bb864affc8a944a83019bd72755a6950c3a0583775374fda583"
Jul 28 04:44:52.031221 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:52.031192    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:44:52.031290 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:52.031265    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:44:52.031327 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:52.031298    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:44:53.038265 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:53.038229    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn" event={"ID":"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c","Type":"ContainerStarted","Data":"a0cb890f12798c2af0fe2c76e4537a4bc64b22596d08a80d0a58199a4886e2a6"}
Jul 28 04:44:53.703484 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:53.703443    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn"
Jul 28 04:44:53.704801 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:53.704771    2566 patch_prober.go:28] interesting pod/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Jul 28 04:44:53.704915 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:53.704868    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn" podUID="bc1dbec2-43ef-43fd-85bb-f95f8b55e64c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Jul 28 04:44:53.989300 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:53.989218    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5f6a9856-5c4d6ddb48-jgvjp"
Jul 28 04:44:53.989445 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:53.989412    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-5f6a9856-978b56c5c-nsvdd"
Jul 28 04:44:54.703128 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:54.703095    2566 patch_prober.go:28] interesting pod/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Jul 28 04:44:54.703505 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:54.703146    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn" podUID="bc1dbec2-43ef-43fd-85bb-f95f8b55e64c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Jul 28 04:44:54.961017 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:54.960937    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn"
Jul 28 04:44:55.703290 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:55.703254    2566 patch_prober.go:28] interesting pod/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Jul 28 04:44:55.703675 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:55.703307    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn" podUID="bc1dbec2-43ef-43fd-85bb-f95f8b55e64c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Jul 28 04:44:56.703807 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:56.703769    2566 patch_prober.go:28] interesting pod/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Jul 28 04:44:56.704186 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:56.703824    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn" podUID="bc1dbec2-43ef-43fd-85bb-f95f8b55e64c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Jul 28 04:44:57.707427 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:57.707398    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn"
Jul 28 04:44:57.708394 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:44:57.708375    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn"
Jul 28 04:45:16.069051 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:16.069016    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5f6a9856-596d9b5c9-qj4fp"]
Jul 28 04:45:16.069579 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:16.069277    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f6a9856-596d9b5c9-qj4fp" podUID="1fe68cba-f10d-40a3-839c-7ccd72abbd32" containerName="maas-api" containerID="cri-o://b93bbe96da1ca99e938680ceb541d362651a37b1ba1ee5c54b76f7b3b3319f7f" gracePeriod=30
Jul 28 04:45:16.122060 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:16.122024    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5f6a9856-978b56c5c-nsvdd"]
Jul 28 04:45:16.122314 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:16.122255    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-5f6a9856-978b56c5c-nsvdd" podUID="e6f6dbdd-1439-4776-a6ac-e7fbf285ce82" containerName="payload-processing" containerID="cri-o://c59151e183aad82bd470e631aae70a23c14386a86951fc0e9b322776abea9ff7" gracePeriod=30
Jul 28 04:45:16.131386 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:16.131362    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5f6a9856-5c4d6ddb48-jgvjp"]
Jul 28 04:45:16.131607 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:16.131582    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5f6a9856-5c4d6ddb48-jgvjp" podUID="c0ee5a80-273c-4244-a5d8-9854770d5a7b" containerName="payload-pre-processing" containerID="cri-o://e4ef10234a8dcb269b57a37bea3fd6e4f4575997a354cf7bd738ba9e2ed75ab0" gracePeriod=30
Jul 28 04:45:16.392201 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:16.392170    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f6a9856-596d9b5c9-qj4fp"
Jul 28 04:45:16.421769 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:16.421739    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5f6a9856-5c4d6ddb48-jgvjp"
Jul 28 04:45:16.430362 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:16.430340    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-5f6a9856-978b56c5c-nsvdd"
Jul 28 04:45:16.553796 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:16.553765    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7pzv\" (UniqueName: \"kubernetes.io/projected/e6f6dbdd-1439-4776-a6ac-e7fbf285ce82-kube-api-access-k7pzv\") pod \"e6f6dbdd-1439-4776-a6ac-e7fbf285ce82\" (UID: \"e6f6dbdd-1439-4776-a6ac-e7fbf285ce82\") "
Jul 28 04:45:16.553796 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:16.553805    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pl4j\" (UniqueName: \"kubernetes.io/projected/1fe68cba-f10d-40a3-839c-7ccd72abbd32-kube-api-access-7pl4j\") pod \"1fe68cba-f10d-40a3-839c-7ccd72abbd32\" (UID: \"1fe68cba-f10d-40a3-839c-7ccd72abbd32\") "
Jul 28 04:45:16.554032 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:16.553866    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fe68cba-f10d-40a3-839c-7ccd72abbd32-openshift-service-ca\") pod \"1fe68cba-f10d-40a3-839c-7ccd72abbd32\" (UID: \"1fe68cba-f10d-40a3-839c-7ccd72abbd32\") "
Jul 28 04:45:16.554032 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:16.553912    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fe68cba-f10d-40a3-839c-7ccd72abbd32-maas-api-tls\") pod \"1fe68cba-f10d-40a3-839c-7ccd72abbd32\" (UID: \"1fe68cba-f10d-40a3-839c-7ccd72abbd32\") "
Jul 28 04:45:16.554032 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:16.553999    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6f6dbdd-1439-4776-a6ac-e7fbf285ce82-plugins-config-volume\") pod \"e6f6dbdd-1439-4776-a6ac-e7fbf285ce82\" (UID: \"e6f6dbdd-1439-4776-a6ac-e7fbf285ce82\") "
Jul 28 04:45:16.554032 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:16.554029    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l6dsc\" (UniqueName: \"kubernetes.io/projected/c0ee5a80-273c-4244-a5d8-9854770d5a7b-kube-api-access-l6dsc\") pod \"c0ee5a80-273c-4244-a5d8-9854770d5a7b\" (UID: \"c0ee5a80-273c-4244-a5d8-9854770d5a7b\") "
Jul 28 04:45:16.554246 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:16.554056    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0ee5a80-273c-4244-a5d8-9854770d5a7b-plugins-config-volume\") pod \"c0ee5a80-273c-4244-a5d8-9854770d5a7b\" (UID: \"c0ee5a80-273c-4244-a5d8-9854770d5a7b\") "
Jul 28 04:45:16.554305 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:16.554289    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fe68cba-f10d-40a3-839c-7ccd72abbd32-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1fe68cba-f10d-40a3-839c-7ccd72abbd32" (UID: "1fe68cba-f10d-40a3-839c-7ccd72abbd32"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 04:45:16.554419 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:16.554402    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1fe68cba-f10d-40a3-839c-7ccd72abbd32-openshift-service-ca\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:45:16.554525 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:16.554416    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6f6dbdd-1439-4776-a6ac-e7fbf285ce82-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e6f6dbdd-1439-4776-a6ac-e7fbf285ce82" (UID: "e6f6dbdd-1439-4776-a6ac-e7fbf285ce82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 04:45:16.554632 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:16.554606    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0ee5a80-273c-4244-a5d8-9854770d5a7b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c0ee5a80-273c-4244-a5d8-9854770d5a7b" (UID: "c0ee5a80-273c-4244-a5d8-9854770d5a7b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 04:45:16.556454 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:16.556424    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fe68cba-f10d-40a3-839c-7ccd72abbd32-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1fe68cba-f10d-40a3-839c-7ccd72abbd32" (UID: "1fe68cba-f10d-40a3-839c-7ccd72abbd32"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 04:45:16.556551 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:16.556452    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6f6dbdd-1439-4776-a6ac-e7fbf285ce82-kube-api-access-k7pzv" (OuterVolumeSpecName: "kube-api-access-k7pzv") pod "e6f6dbdd-1439-4776-a6ac-e7fbf285ce82" (UID: "e6f6dbdd-1439-4776-a6ac-e7fbf285ce82"). InnerVolumeSpecName "kube-api-access-k7pzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:45:16.556551 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:16.556534    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fe68cba-f10d-40a3-839c-7ccd72abbd32-kube-api-access-7pl4j" (OuterVolumeSpecName: "kube-api-access-7pl4j") pod "1fe68cba-f10d-40a3-839c-7ccd72abbd32" (UID: "1fe68cba-f10d-40a3-839c-7ccd72abbd32"). InnerVolumeSpecName "kube-api-access-7pl4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:45:16.556633 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:16.556562    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0ee5a80-273c-4244-a5d8-9854770d5a7b-kube-api-access-l6dsc" (OuterVolumeSpecName: "kube-api-access-l6dsc") pod "c0ee5a80-273c-4244-a5d8-9854770d5a7b" (UID: "c0ee5a80-273c-4244-a5d8-9854770d5a7b"). InnerVolumeSpecName "kube-api-access-l6dsc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:45:16.655226 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:16.655153    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e6f6dbdd-1439-4776-a6ac-e7fbf285ce82-plugins-config-volume\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:45:16.655226 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:16.655182    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6dsc\" (UniqueName: \"kubernetes.io/projected/c0ee5a80-273c-4244-a5d8-9854770d5a7b-kube-api-access-l6dsc\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:45:16.655226 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:16.655192    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0ee5a80-273c-4244-a5d8-9854770d5a7b-plugins-config-volume\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:45:16.655226 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:16.655201    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7pzv\" (UniqueName: \"kubernetes.io/projected/e6f6dbdd-1439-4776-a6ac-e7fbf285ce82-kube-api-access-k7pzv\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:45:16.655226 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:16.655210    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pl4j\" (UniqueName: \"kubernetes.io/projected/1fe68cba-f10d-40a3-839c-7ccd72abbd32-kube-api-access-7pl4j\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:45:16.655226 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:16.655219    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1fe68cba-f10d-40a3-839c-7ccd72abbd32-maas-api-tls\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:45:17.155062 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:17.155019    2566 generic.go:356] "Generic (PLEG): container finished" podID="1fe68cba-f10d-40a3-839c-7ccd72abbd32" containerID="b93bbe96da1ca99e938680ceb541d362651a37b1ba1ee5c54b76f7b3b3319f7f" exitCode=0
Jul 28 04:45:17.155509 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:17.155086    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f6a9856-596d9b5c9-qj4fp"
Jul 28 04:45:17.155509 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:17.155101    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f6a9856-596d9b5c9-qj4fp" event={"ID":"1fe68cba-f10d-40a3-839c-7ccd72abbd32","Type":"ContainerDied","Data":"b93bbe96da1ca99e938680ceb541d362651a37b1ba1ee5c54b76f7b3b3319f7f"}
Jul 28 04:45:17.155509 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:17.155163    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-5f6a9856-596d9b5c9-qj4fp" event={"ID":"1fe68cba-f10d-40a3-839c-7ccd72abbd32","Type":"ContainerDied","Data":"31ba599a8703e054f029b9357b7c0f2b5ebc5bcafa92b97160670f16323b489b"}
Jul 28 04:45:17.155509 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:17.155196    2566 scope.go:117] "RemoveContainer" containerID="b93bbe96da1ca99e938680ceb541d362651a37b1ba1ee5c54b76f7b3b3319f7f"
Jul 28 04:45:17.156781 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:17.156758    2566 generic.go:356] "Generic (PLEG): container finished" podID="c0ee5a80-273c-4244-a5d8-9854770d5a7b" containerID="e4ef10234a8dcb269b57a37bea3fd6e4f4575997a354cf7bd738ba9e2ed75ab0" exitCode=0
Jul 28 04:45:17.156904 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:17.156787    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5f6a9856-5c4d6ddb48-jgvjp" event={"ID":"c0ee5a80-273c-4244-a5d8-9854770d5a7b","Type":"ContainerDied","Data":"e4ef10234a8dcb269b57a37bea3fd6e4f4575997a354cf7bd738ba9e2ed75ab0"}
Jul 28 04:45:17.156904 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:17.156816    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5f6a9856-5c4d6ddb48-jgvjp"
Jul 28 04:45:17.156904 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:17.156850    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5f6a9856-5c4d6ddb48-jgvjp" event={"ID":"c0ee5a80-273c-4244-a5d8-9854770d5a7b","Type":"ContainerDied","Data":"50f0087e22779d4a47a012a49e2f11508b076fff4958ccf3d35dfd42b64d4bcb"}
Jul 28 04:45:17.158405 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:17.158377    2566 generic.go:356] "Generic (PLEG): container finished" podID="e6f6dbdd-1439-4776-a6ac-e7fbf285ce82" containerID="c59151e183aad82bd470e631aae70a23c14386a86951fc0e9b322776abea9ff7" exitCode=0
Jul 28 04:45:17.158779 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:17.158439    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-5f6a9856-978b56c5c-nsvdd"
Jul 28 04:45:17.158779 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:17.158441    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-5f6a9856-978b56c5c-nsvdd" event={"ID":"e6f6dbdd-1439-4776-a6ac-e7fbf285ce82","Type":"ContainerDied","Data":"c59151e183aad82bd470e631aae70a23c14386a86951fc0e9b322776abea9ff7"}
Jul 28 04:45:17.158779 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:17.158576    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-5f6a9856-978b56c5c-nsvdd" event={"ID":"e6f6dbdd-1439-4776-a6ac-e7fbf285ce82","Type":"ContainerDied","Data":"e3b3e1fc3144bcdc512e50fec73da2a79d33e96d710db9c83ea13c0bd57b7cc9"}
Jul 28 04:45:17.166498 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:17.166469    2566 scope.go:117] "RemoveContainer" containerID="b93bbe96da1ca99e938680ceb541d362651a37b1ba1ee5c54b76f7b3b3319f7f"
Jul 28 04:45:17.166888 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:45:17.166861    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b93bbe96da1ca99e938680ceb541d362651a37b1ba1ee5c54b76f7b3b3319f7f\": container with ID starting with b93bbe96da1ca99e938680ceb541d362651a37b1ba1ee5c54b76f7b3b3319f7f not found: ID does not exist" containerID="b93bbe96da1ca99e938680ceb541d362651a37b1ba1ee5c54b76f7b3b3319f7f"
Jul 28 04:45:17.167052 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:17.167027    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b93bbe96da1ca99e938680ceb541d362651a37b1ba1ee5c54b76f7b3b3319f7f"} err="failed to get container status \"b93bbe96da1ca99e938680ceb541d362651a37b1ba1ee5c54b76f7b3b3319f7f\": rpc error: code = NotFound desc = could not find container \"b93bbe96da1ca99e938680ceb541d362651a37b1ba1ee5c54b76f7b3b3319f7f\": container with ID starting with b93bbe96da1ca99e938680ceb541d362651a37b1ba1ee5c54b76f7b3b3319f7f not found: ID does not exist"
Jul 28 04:45:17.167265 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:17.167245    2566 scope.go:117] "RemoveContainer" containerID="e4ef10234a8dcb269b57a37bea3fd6e4f4575997a354cf7bd738ba9e2ed75ab0"
Jul 28 04:45:17.176429 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:17.176411    2566 scope.go:117] "RemoveContainer" containerID="e4ef10234a8dcb269b57a37bea3fd6e4f4575997a354cf7bd738ba9e2ed75ab0"
Jul 28 04:45:17.176680 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:45:17.176659    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4ef10234a8dcb269b57a37bea3fd6e4f4575997a354cf7bd738ba9e2ed75ab0\": container with ID starting with e4ef10234a8dcb269b57a37bea3fd6e4f4575997a354cf7bd738ba9e2ed75ab0 not found: ID does not exist" containerID="e4ef10234a8dcb269b57a37bea3fd6e4f4575997a354cf7bd738ba9e2ed75ab0"
Jul 28 04:45:17.176769 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:17.176690    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4ef10234a8dcb269b57a37bea3fd6e4f4575997a354cf7bd738ba9e2ed75ab0"} err="failed to get container status \"e4ef10234a8dcb269b57a37bea3fd6e4f4575997a354cf7bd738ba9e2ed75ab0\": rpc error: code = NotFound desc = could not find container \"e4ef10234a8dcb269b57a37bea3fd6e4f4575997a354cf7bd738ba9e2ed75ab0\": container with ID starting with e4ef10234a8dcb269b57a37bea3fd6e4f4575997a354cf7bd738ba9e2ed75ab0 not found: ID does not exist"
Jul 28 04:45:17.176769 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:17.176713    2566 scope.go:117] "RemoveContainer" containerID="c59151e183aad82bd470e631aae70a23c14386a86951fc0e9b322776abea9ff7"
Jul 28 04:45:17.183453 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:17.183430    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5f6a9856-596d9b5c9-qj4fp"]
Jul 28 04:45:17.184667 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:17.184632    2566 scope.go:117] "RemoveContainer" containerID="c59151e183aad82bd470e631aae70a23c14386a86951fc0e9b322776abea9ff7"
Jul 28 04:45:17.185207 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:45:17.185178    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c59151e183aad82bd470e631aae70a23c14386a86951fc0e9b322776abea9ff7\": container with ID starting with c59151e183aad82bd470e631aae70a23c14386a86951fc0e9b322776abea9ff7 not found: ID does not exist" containerID="c59151e183aad82bd470e631aae70a23c14386a86951fc0e9b322776abea9ff7"
Jul 28 04:45:17.185301 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:17.185213    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c59151e183aad82bd470e631aae70a23c14386a86951fc0e9b322776abea9ff7"} err="failed to get container status \"c59151e183aad82bd470e631aae70a23c14386a86951fc0e9b322776abea9ff7\": rpc error: code = NotFound desc = could not find container \"c59151e183aad82bd470e631aae70a23c14386a86951fc0e9b322776abea9ff7\": container with ID starting with c59151e183aad82bd470e631aae70a23c14386a86951fc0e9b322776abea9ff7 not found: ID does not exist"
Jul 28 04:45:17.186956 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:17.186938    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-5f6a9856-596d9b5c9-qj4fp"]
Jul 28 04:45:17.197088 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:17.197069    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5f6a9856-978b56c5c-nsvdd"]
Jul 28 04:45:17.199748 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:17.199726    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-5f6a9856-978b56c5c-nsvdd"]
Jul 28 04:45:17.208586 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:17.208564    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5f6a9856-5c4d6ddb48-jgvjp"]
Jul 28 04:45:17.212072 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:17.212051    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5f6a9856-5c4d6ddb48-jgvjp"]
Jul 28 04:45:18.088963 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:18.088926    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1fe68cba-f10d-40a3-839c-7ccd72abbd32" path="/var/lib/kubelet/pods/1fe68cba-f10d-40a3-839c-7ccd72abbd32/volumes"
Jul 28 04:45:18.089338 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:18.089323    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0ee5a80-273c-4244-a5d8-9854770d5a7b" path="/var/lib/kubelet/pods/c0ee5a80-273c-4244-a5d8-9854770d5a7b/volumes"
Jul 28 04:45:18.089676 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:18.089664    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6f6dbdd-1439-4776-a6ac-e7fbf285ce82" path="/var/lib/kubelet/pods/e6f6dbdd-1439-4776-a6ac-e7fbf285ce82/volumes"
Jul 28 04:45:20.892525 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:20.892487    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn"]
Jul 28 04:45:20.892957 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:20.892753    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn" podUID="bc1dbec2-43ef-43fd-85bb-f95f8b55e64c" containerName="istio-proxy" containerID="cri-o://a0cb890f12798c2af0fe2c76e4537a4bc64b22596d08a80d0a58199a4886e2a6" gracePeriod=30
Jul 28 04:45:21.727943 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:21.727902    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp"]
Jul 28 04:45:21.728481 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:21.728465    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1fe68cba-f10d-40a3-839c-7ccd72abbd32" containerName="maas-api"
Jul 28 04:45:21.728592 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:21.728484    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1fe68cba-f10d-40a3-839c-7ccd72abbd32" containerName="maas-api"
Jul 28 04:45:21.728592 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:21.728500    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6f6dbdd-1439-4776-a6ac-e7fbf285ce82" containerName="payload-processing"
Jul 28 04:45:21.728592 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:21.728509    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6f6dbdd-1439-4776-a6ac-e7fbf285ce82" containerName="payload-processing"
Jul 28 04:45:21.728592 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:21.728527    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0ee5a80-273c-4244-a5d8-9854770d5a7b" containerName="payload-pre-processing"
Jul 28 04:45:21.728592 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:21.728536    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0ee5a80-273c-4244-a5d8-9854770d5a7b" containerName="payload-pre-processing"
Jul 28 04:45:21.728878 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:21.728617    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1fe68cba-f10d-40a3-839c-7ccd72abbd32" containerName="maas-api"
Jul 28 04:45:21.728878 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:21.728628    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e6f6dbdd-1439-4776-a6ac-e7fbf285ce82" containerName="payload-processing"
Jul 28 04:45:21.728878 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:21.728640    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0ee5a80-273c-4244-a5d8-9854770d5a7b" containerName="payload-pre-processing"
Jul 28 04:45:21.733590 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:21.733570    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp"
Jul 28 04:45:21.735947 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:21.735928    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b42b78c0-openshift-default-dockercfg-nhj4k\""
Jul 28 04:45:21.742138 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:21.741961    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp"]
Jul 28 04:45:21.907572 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:21.907536    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a62608d-9359-43c9-9cfc-b13b26bb00b8-credential-socket\") pod \"e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp\" (UID: \"8a62608d-9359-43c9-9cfc-b13b26bb00b8\") " pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp"
Jul 28 04:45:21.907963 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:21.907591    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a62608d-9359-43c9-9cfc-b13b26bb00b8-istiod-ca-cert\") pod \"e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp\" (UID: \"8a62608d-9359-43c9-9cfc-b13b26bb00b8\") " pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp"
Jul 28 04:45:21.907963 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:21.907641    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a62608d-9359-43c9-9cfc-b13b26bb00b8-istio-envoy\") pod \"e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp\" (UID: \"8a62608d-9359-43c9-9cfc-b13b26bb00b8\") " pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp"
Jul 28 04:45:21.907963 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:21.907710    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a62608d-9359-43c9-9cfc-b13b26bb00b8-workload-certs\") pod \"e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp\" (UID: \"8a62608d-9359-43c9-9cfc-b13b26bb00b8\") " pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp"
Jul 28 04:45:21.907963 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:21.907754    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a62608d-9359-43c9-9cfc-b13b26bb00b8-istio-data\") pod \"e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp\" (UID: \"8a62608d-9359-43c9-9cfc-b13b26bb00b8\") " pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp"
Jul 28 04:45:21.907963 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:21.907780    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a62608d-9359-43c9-9cfc-b13b26bb00b8-istio-token\") pod \"e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp\" (UID: \"8a62608d-9359-43c9-9cfc-b13b26bb00b8\") " pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp"
Jul 28 04:45:21.907963 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:21.907817    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r77jd\" (UniqueName: \"kubernetes.io/projected/8a62608d-9359-43c9-9cfc-b13b26bb00b8-kube-api-access-r77jd\") pod \"e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp\" (UID: \"8a62608d-9359-43c9-9cfc-b13b26bb00b8\") " pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp"
Jul 28 04:45:21.907963 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:21.907885    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a62608d-9359-43c9-9cfc-b13b26bb00b8-workload-socket\") pod \"e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp\" (UID: \"8a62608d-9359-43c9-9cfc-b13b26bb00b8\") " pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp"
Jul 28 04:45:21.907963 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:21.907956    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a62608d-9359-43c9-9cfc-b13b26bb00b8-istio-podinfo\") pod \"e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp\" (UID: \"8a62608d-9359-43c9-9cfc-b13b26bb00b8\") " pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp"
Jul 28 04:45:22.009159 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:22.009059    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a62608d-9359-43c9-9cfc-b13b26bb00b8-istiod-ca-cert\") pod \"e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp\" (UID: \"8a62608d-9359-43c9-9cfc-b13b26bb00b8\") " pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp"
Jul 28 04:45:22.009159 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:22.009114    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a62608d-9359-43c9-9cfc-b13b26bb00b8-istio-envoy\") pod \"e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp\" (UID: \"8a62608d-9359-43c9-9cfc-b13b26bb00b8\") " pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp"
Jul 28 04:45:22.009159 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:22.009148    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a62608d-9359-43c9-9cfc-b13b26bb00b8-workload-certs\") pod \"e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp\" (UID: \"8a62608d-9359-43c9-9cfc-b13b26bb00b8\") " pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp"
Jul 28 04:45:22.009432 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:22.009183    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a62608d-9359-43c9-9cfc-b13b26bb00b8-istio-data\") pod \"e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp\" (UID: \"8a62608d-9359-43c9-9cfc-b13b26bb00b8\") " pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp"
Jul 28 04:45:22.009432 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:22.009210    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a62608d-9359-43c9-9cfc-b13b26bb00b8-istio-token\") pod \"e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp\" (UID: \"8a62608d-9359-43c9-9cfc-b13b26bb00b8\") " pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp"
Jul 28 04:45:22.009432 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:22.009245    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r77jd\" (UniqueName: \"kubernetes.io/projected/8a62608d-9359-43c9-9cfc-b13b26bb00b8-kube-api-access-r77jd\") pod \"e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp\" (UID: \"8a62608d-9359-43c9-9cfc-b13b26bb00b8\") " pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp"
Jul 28 04:45:22.009432 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:22.009288    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a62608d-9359-43c9-9cfc-b13b26bb00b8-workload-socket\") pod \"e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp\" (UID: \"8a62608d-9359-43c9-9cfc-b13b26bb00b8\") " pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp"
Jul 28 04:45:22.009432 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:22.009328    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a62608d-9359-43c9-9cfc-b13b26bb00b8-istio-podinfo\") pod \"e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp\" (UID: \"8a62608d-9359-43c9-9cfc-b13b26bb00b8\") " pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp"
Jul 28 04:45:22.009432 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:22.009416    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a62608d-9359-43c9-9cfc-b13b26bb00b8-credential-socket\") pod \"e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp\" (UID: \"8a62608d-9359-43c9-9cfc-b13b26bb00b8\") " pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp"
Jul 28 04:45:22.009739 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:22.009591    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a62608d-9359-43c9-9cfc-b13b26bb00b8-workload-certs\") pod \"e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp\" (UID: \"8a62608d-9359-43c9-9cfc-b13b26bb00b8\") " pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp"
Jul 28 04:45:22.009739 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:22.009673    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a62608d-9359-43c9-9cfc-b13b26bb00b8-istio-data\") pod \"e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp\" (UID: \"8a62608d-9359-43c9-9cfc-b13b26bb00b8\") " pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp"
Jul 28 04:45:22.009828 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:22.009745    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a62608d-9359-43c9-9cfc-b13b26bb00b8-credential-socket\") pod \"e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp\" (UID: \"8a62608d-9359-43c9-9cfc-b13b26bb00b8\") " pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp"
Jul 28 04:45:22.009944 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:22.009922    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a62608d-9359-43c9-9cfc-b13b26bb00b8-istiod-ca-cert\") pod \"e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp\" (UID: \"8a62608d-9359-43c9-9cfc-b13b26bb00b8\") " pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp"
Jul 28 04:45:22.010180 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:22.010151    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a62608d-9359-43c9-9cfc-b13b26bb00b8-workload-socket\") pod \"e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp\" (UID: \"8a62608d-9359-43c9-9cfc-b13b26bb00b8\") " pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp"
Jul 28 04:45:22.011559 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:22.011534    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a62608d-9359-43c9-9cfc-b13b26bb00b8-istio-envoy\") pod \"e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp\" (UID: \"8a62608d-9359-43c9-9cfc-b13b26bb00b8\") " pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp"
Jul 28 04:45:22.012145 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:22.012114    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a62608d-9359-43c9-9cfc-b13b26bb00b8-istio-podinfo\") pod \"e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp\" (UID: \"8a62608d-9359-43c9-9cfc-b13b26bb00b8\") " pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp"
Jul 28 04:45:22.017090 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:22.017070    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r77jd\" (UniqueName: \"kubernetes.io/projected/8a62608d-9359-43c9-9cfc-b13b26bb00b8-kube-api-access-r77jd\") pod \"e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp\" (UID: \"8a62608d-9359-43c9-9cfc-b13b26bb00b8\") " pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp"
Jul 28 04:45:22.017206 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:22.017134    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a62608d-9359-43c9-9cfc-b13b26bb00b8-istio-token\") pod \"e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp\" (UID: \"8a62608d-9359-43c9-9cfc-b13b26bb00b8\") " pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp"
Jul 28 04:45:22.046873 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:22.046824    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp"
Jul 28 04:45:22.204863 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:45:22.204813    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a62608d_9359_43c9_9cfc_b13b26bb00b8.slice/crio-25f087b94ca6701f4c4da033584681c8f1d36f4bfde6916905a521e164b799ca WatchSource:0}: Error finding container 25f087b94ca6701f4c4da033584681c8f1d36f4bfde6916905a521e164b799ca: Status 404 returned error can't find the container with id 25f087b94ca6701f4c4da033584681c8f1d36f4bfde6916905a521e164b799ca
Jul 28 04:45:22.205876 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:22.205831    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp"]
Jul 28 04:45:22.207037 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:22.207007    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:45:22.207117 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:22.207065    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:45:22.207117 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:22.207093    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:45:23.193783 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:23.193747    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp" event={"ID":"8a62608d-9359-43c9-9cfc-b13b26bb00b8","Type":"ContainerStarted","Data":"5055391c05f868a1363b150596a4753fe4d7432c09680862c67e5147ae6c18aa"}
Jul 28 04:45:23.194275 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:23.193817    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp" event={"ID":"8a62608d-9359-43c9-9cfc-b13b26bb00b8","Type":"ContainerStarted","Data":"25f087b94ca6701f4c4da033584681c8f1d36f4bfde6916905a521e164b799ca"}
Jul 28 04:45:23.213881 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:23.213821    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp" podStartSLOduration=2.213804673 podStartE2EDuration="2.213804673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:45:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:45:23.213243517 +0000 UTC m=+2023.603419540" watchObservedRunningTime="2026-07-28 04:45:23.213804673 +0000 UTC m=+2023.603980697"
Jul 28 04:45:24.047798 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:24.047763    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp"
Jul 28 04:45:24.052905 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:24.052881    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp"
Jul 28 04:45:24.198917 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:24.198892    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp"
Jul 28 04:45:24.199916 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:24.199894    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp"
Jul 28 04:45:24.651072 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:24.651038    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b42b78c0-5b8875694-2kjwc"]
Jul 28 04:45:24.654852 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:24.654825    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b42b78c0-5b8875694-2kjwc"
Jul 28 04:45:24.657341 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:24.657317    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b42b78c0-serving-cert\""
Jul 28 04:45:24.657533 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:24.657391    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gfn77\""
Jul 28 04:45:24.657533 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:24.657451    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 04:45:24.657676 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:24.657618    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 04:45:24.665281 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:24.665257    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b42b78c0-5b8875694-2kjwc"]
Jul 28 04:45:24.837695 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:24.837656    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/610b351b-f97f-4a99-a9ab-408f9750e5d7-maas-api-tls\") pod \"maas-api-e2e-ait-b42b78c0-5b8875694-2kjwc\" (UID: \"610b351b-f97f-4a99-a9ab-408f9750e5d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b42b78c0-5b8875694-2kjwc"
Jul 28 04:45:24.837912 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:24.837710    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/610b351b-f97f-4a99-a9ab-408f9750e5d7-openshift-service-ca\") pod \"maas-api-e2e-ait-b42b78c0-5b8875694-2kjwc\" (UID: \"610b351b-f97f-4a99-a9ab-408f9750e5d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b42b78c0-5b8875694-2kjwc"
Jul 28 04:45:24.837912 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:24.837734    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xsbr\" (UniqueName: \"kubernetes.io/projected/610b351b-f97f-4a99-a9ab-408f9750e5d7-kube-api-access-6xsbr\") pod \"maas-api-e2e-ait-b42b78c0-5b8875694-2kjwc\" (UID: \"610b351b-f97f-4a99-a9ab-408f9750e5d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b42b78c0-5b8875694-2kjwc"
Jul 28 04:45:24.938975 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:24.938878    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/610b351b-f97f-4a99-a9ab-408f9750e5d7-maas-api-tls\") pod \"maas-api-e2e-ait-b42b78c0-5b8875694-2kjwc\" (UID: \"610b351b-f97f-4a99-a9ab-408f9750e5d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b42b78c0-5b8875694-2kjwc"
Jul 28 04:45:24.938975 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:24.938934    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/610b351b-f97f-4a99-a9ab-408f9750e5d7-openshift-service-ca\") pod \"maas-api-e2e-ait-b42b78c0-5b8875694-2kjwc\" (UID: \"610b351b-f97f-4a99-a9ab-408f9750e5d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b42b78c0-5b8875694-2kjwc"
Jul 28 04:45:24.938975 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:24.938956    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xsbr\" (UniqueName: \"kubernetes.io/projected/610b351b-f97f-4a99-a9ab-408f9750e5d7-kube-api-access-6xsbr\") pod \"maas-api-e2e-ait-b42b78c0-5b8875694-2kjwc\" (UID: \"610b351b-f97f-4a99-a9ab-408f9750e5d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b42b78c0-5b8875694-2kjwc"
Jul 28 04:45:24.939254 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:45:24.939067    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-b42b78c0-serving-cert: secret "maas-api-e2e-ait-b42b78c0-serving-cert" not found
Jul 28 04:45:24.939254 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:45:24.939161    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/610b351b-f97f-4a99-a9ab-408f9750e5d7-maas-api-tls podName:610b351b-f97f-4a99-a9ab-408f9750e5d7 nodeName:}" failed. No retries permitted until 2026-07-28 04:45:25.439137944 +0000 UTC m=+2025.829313957 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/610b351b-f97f-4a99-a9ab-408f9750e5d7-maas-api-tls") pod "maas-api-e2e-ait-b42b78c0-5b8875694-2kjwc" (UID: "610b351b-f97f-4a99-a9ab-408f9750e5d7") : secret "maas-api-e2e-ait-b42b78c0-serving-cert" not found
Jul 28 04:45:24.939565 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:24.939548    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/610b351b-f97f-4a99-a9ab-408f9750e5d7-openshift-service-ca\") pod \"maas-api-e2e-ait-b42b78c0-5b8875694-2kjwc\" (UID: \"610b351b-f97f-4a99-a9ab-408f9750e5d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b42b78c0-5b8875694-2kjwc"
Jul 28 04:45:24.949964 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:24.949939    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xsbr\" (UniqueName: \"kubernetes.io/projected/610b351b-f97f-4a99-a9ab-408f9750e5d7-kube-api-access-6xsbr\") pod \"maas-api-e2e-ait-b42b78c0-5b8875694-2kjwc\" (UID: \"610b351b-f97f-4a99-a9ab-408f9750e5d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b42b78c0-5b8875694-2kjwc"
Jul 28 04:45:25.444000 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:25.443963    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/610b351b-f97f-4a99-a9ab-408f9750e5d7-maas-api-tls\") pod \"maas-api-e2e-ait-b42b78c0-5b8875694-2kjwc\" (UID: \"610b351b-f97f-4a99-a9ab-408f9750e5d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b42b78c0-5b8875694-2kjwc"
Jul 28 04:45:25.446492 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:25.446465    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/610b351b-f97f-4a99-a9ab-408f9750e5d7-maas-api-tls\") pod \"maas-api-e2e-ait-b42b78c0-5b8875694-2kjwc\" (UID: \"610b351b-f97f-4a99-a9ab-408f9750e5d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b42b78c0-5b8875694-2kjwc"
Jul 28 04:45:25.467534 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:25.467508    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b42b78c0-c5ff7dd4-r8p9k"]
Jul 28 04:45:25.471408 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:25.471390    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b42b78c0-c5ff7dd4-r8p9k"
Jul 28 04:45:25.473667 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:25.473650    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b42b78c0-dockercfg-896h8\""
Jul 28 04:45:25.473743 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:25.473656    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b42b78c0\""
Jul 28 04:45:25.480535 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:25.480509    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b42b78c0-67f6969f48-k8l6w"]
Jul 28 04:45:25.483944 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:25.483924    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b42b78c0-c5ff7dd4-r8p9k"]
Jul 28 04:45:25.484026 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:25.484017    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b42b78c0-67f6969f48-k8l6w"
Jul 28 04:45:25.492290 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:25.492134    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b42b78c0-67f6969f48-k8l6w"]
Jul 28 04:45:25.568099 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:25.568061    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b42b78c0-5b8875694-2kjwc"
Jul 28 04:45:25.646257 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:25.646217    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/295c0359-e240-47ca-b8f1-1829a3d86699-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b42b78c0-c5ff7dd4-r8p9k\" (UID: \"295c0359-e240-47ca-b8f1-1829a3d86699\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b42b78c0-c5ff7dd4-r8p9k"
Jul 28 04:45:25.646432 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:25.646274    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tmf8\" (UniqueName: \"kubernetes.io/projected/600028df-487a-4254-98db-9a2fb5b894a4-kube-api-access-4tmf8\") pod \"payload-processing-e2e-ait-b42b78c0-67f6969f48-k8l6w\" (UID: \"600028df-487a-4254-98db-9a2fb5b894a4\") " pod="openshift-ingress/payload-processing-e2e-ait-b42b78c0-67f6969f48-k8l6w"
Jul 28 04:45:25.646432 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:25.646314    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2kqr\" (UniqueName: \"kubernetes.io/projected/295c0359-e240-47ca-b8f1-1829a3d86699-kube-api-access-z2kqr\") pod \"payload-pre-processing-e2e-ait-b42b78c0-c5ff7dd4-r8p9k\" (UID: \"295c0359-e240-47ca-b8f1-1829a3d86699\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b42b78c0-c5ff7dd4-r8p9k"
Jul 28 04:45:25.646432 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:25.646355    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/600028df-487a-4254-98db-9a2fb5b894a4-plugins-config-volume\") pod \"payload-processing-e2e-ait-b42b78c0-67f6969f48-k8l6w\" (UID: \"600028df-487a-4254-98db-9a2fb5b894a4\") " pod="openshift-ingress/payload-processing-e2e-ait-b42b78c0-67f6969f48-k8l6w"
Jul 28 04:45:25.711791 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:25.711733    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b42b78c0-5b8875694-2kjwc"]
Jul 28 04:45:25.714331 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:45:25.714300    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod610b351b_f97f_4a99_a9ab_408f9750e5d7.slice/crio-805b93a94c10853196b6b0abe05c5ad82cc97a48fc5c7fe5f555fed272b25331 WatchSource:0}: Error finding container 805b93a94c10853196b6b0abe05c5ad82cc97a48fc5c7fe5f555fed272b25331: Status 404 returned error can't find the container with id 805b93a94c10853196b6b0abe05c5ad82cc97a48fc5c7fe5f555fed272b25331
Jul 28 04:45:25.747738 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:25.747710    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/600028df-487a-4254-98db-9a2fb5b894a4-plugins-config-volume\") pod \"payload-processing-e2e-ait-b42b78c0-67f6969f48-k8l6w\" (UID: \"600028df-487a-4254-98db-9a2fb5b894a4\") " pod="openshift-ingress/payload-processing-e2e-ait-b42b78c0-67f6969f48-k8l6w"
Jul 28 04:45:25.747906 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:25.747800    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/295c0359-e240-47ca-b8f1-1829a3d86699-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b42b78c0-c5ff7dd4-r8p9k\" (UID: \"295c0359-e240-47ca-b8f1-1829a3d86699\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b42b78c0-c5ff7dd4-r8p9k"
Jul 28 04:45:25.747906 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:25.747884    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tmf8\" (UniqueName: \"kubernetes.io/projected/600028df-487a-4254-98db-9a2fb5b894a4-kube-api-access-4tmf8\") pod \"payload-processing-e2e-ait-b42b78c0-67f6969f48-k8l6w\" (UID: \"600028df-487a-4254-98db-9a2fb5b894a4\") " pod="openshift-ingress/payload-processing-e2e-ait-b42b78c0-67f6969f48-k8l6w"
Jul 28 04:45:25.748029 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:25.747946    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2kqr\" (UniqueName: \"kubernetes.io/projected/295c0359-e240-47ca-b8f1-1829a3d86699-kube-api-access-z2kqr\") pod \"payload-pre-processing-e2e-ait-b42b78c0-c5ff7dd4-r8p9k\" (UID: \"295c0359-e240-47ca-b8f1-1829a3d86699\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b42b78c0-c5ff7dd4-r8p9k"
Jul 28 04:45:25.748480 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:25.748460    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/295c0359-e240-47ca-b8f1-1829a3d86699-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b42b78c0-c5ff7dd4-r8p9k\" (UID: \"295c0359-e240-47ca-b8f1-1829a3d86699\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b42b78c0-c5ff7dd4-r8p9k"
Jul 28 04:45:25.748540 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:25.748467    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/600028df-487a-4254-98db-9a2fb5b894a4-plugins-config-volume\") pod \"payload-processing-e2e-ait-b42b78c0-67f6969f48-k8l6w\" (UID: \"600028df-487a-4254-98db-9a2fb5b894a4\") " pod="openshift-ingress/payload-processing-e2e-ait-b42b78c0-67f6969f48-k8l6w"
Jul 28 04:45:25.755467 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:25.755445    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2kqr\" (UniqueName: \"kubernetes.io/projected/295c0359-e240-47ca-b8f1-1829a3d86699-kube-api-access-z2kqr\") pod \"payload-pre-processing-e2e-ait-b42b78c0-c5ff7dd4-r8p9k\" (UID: \"295c0359-e240-47ca-b8f1-1829a3d86699\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b42b78c0-c5ff7dd4-r8p9k"
Jul 28 04:45:25.755547 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:25.755467    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tmf8\" (UniqueName: \"kubernetes.io/projected/600028df-487a-4254-98db-9a2fb5b894a4-kube-api-access-4tmf8\") pod \"payload-processing-e2e-ait-b42b78c0-67f6969f48-k8l6w\" (UID: \"600028df-487a-4254-98db-9a2fb5b894a4\") " pod="openshift-ingress/payload-processing-e2e-ait-b42b78c0-67f6969f48-k8l6w"
Jul 28 04:45:25.783353 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:25.783331    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b42b78c0-c5ff7dd4-r8p9k"
Jul 28 04:45:25.796809 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:25.796790    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b42b78c0-67f6969f48-k8l6w"
Jul 28 04:45:25.939044 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:25.939016    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b42b78c0-c5ff7dd4-r8p9k"]
Jul 28 04:45:25.964700 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:45:25.964622    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod295c0359_e240_47ca_b8f1_1829a3d86699.slice/crio-2b794987dff5367b093b9c9532ad45e2d67d210c9981cdb75c46e198062ef3f9 WatchSource:0}: Error finding container 2b794987dff5367b093b9c9532ad45e2d67d210c9981cdb75c46e198062ef3f9: Status 404 returned error can't find the container with id 2b794987dff5367b093b9c9532ad45e2d67d210c9981cdb75c46e198062ef3f9
Jul 28 04:45:26.168578 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.168436    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b42b78c0-67f6969f48-k8l6w"]
Jul 28 04:45:26.171504 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:45:26.171457    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod600028df_487a_4254_98db_9a2fb5b894a4.slice/crio-07c32c97594b53f72b42cac3bf4576c892cd55abcce3b97e3d4b43fea10e3f73 WatchSource:0}: Error finding container 07c32c97594b53f72b42cac3bf4576c892cd55abcce3b97e3d4b43fea10e3f73: Status 404 returned error can't find the container with id 07c32c97594b53f72b42cac3bf4576c892cd55abcce3b97e3d4b43fea10e3f73
Jul 28 04:45:26.177021 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.176999    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn"
Jul 28 04:45:26.217110 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.216968    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b42b78c0-67f6969f48-k8l6w" event={"ID":"600028df-487a-4254-98db-9a2fb5b894a4","Type":"ContainerStarted","Data":"07c32c97594b53f72b42cac3bf4576c892cd55abcce3b97e3d4b43fea10e3f73"}
Jul 28 04:45:26.219719 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.219684    2566 generic.go:356] "Generic (PLEG): container finished" podID="bc1dbec2-43ef-43fd-85bb-f95f8b55e64c" containerID="a0cb890f12798c2af0fe2c76e4537a4bc64b22596d08a80d0a58199a4886e2a6" exitCode=0
Jul 28 04:45:26.219949 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.219920    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn"
Jul 28 04:45:26.220313 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.219816    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn" event={"ID":"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c","Type":"ContainerDied","Data":"a0cb890f12798c2af0fe2c76e4537a4bc64b22596d08a80d0a58199a4886e2a6"}
Jul 28 04:45:26.220313 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.220150    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn" event={"ID":"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c","Type":"ContainerDied","Data":"a3fb682ae6d65cda989908a0212e17e532011412b9f61ca52f302151c5627bfb"}
Jul 28 04:45:26.220313 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.220190    2566 scope.go:117] "RemoveContainer" containerID="a0cb890f12798c2af0fe2c76e4537a4bc64b22596d08a80d0a58199a4886e2a6"
Jul 28 04:45:26.223082 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.223056    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b42b78c0-c5ff7dd4-r8p9k" event={"ID":"295c0359-e240-47ca-b8f1-1829a3d86699","Type":"ContainerStarted","Data":"74b9307bcbbdfe423e66f3ea6657fdbed5ba5cc0ee8acde916639e476a4fec3e"}
Jul 28 04:45:26.223188 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.223109    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b42b78c0-c5ff7dd4-r8p9k" event={"ID":"295c0359-e240-47ca-b8f1-1829a3d86699","Type":"ContainerStarted","Data":"2b794987dff5367b093b9c9532ad45e2d67d210c9981cdb75c46e198062ef3f9"}
Jul 28 04:45:26.223741 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.223709    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b42b78c0-c5ff7dd4-r8p9k"
Jul 28 04:45:26.226242 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.226173    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b42b78c0-5b8875694-2kjwc" event={"ID":"610b351b-f97f-4a99-a9ab-408f9750e5d7","Type":"ContainerStarted","Data":"e4baa7e56c4782936aecf6b7ccf881cc61f82b7fbac3859bb4046b9630966273"}
Jul 28 04:45:26.226242 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.226220    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b42b78c0-5b8875694-2kjwc" event={"ID":"610b351b-f97f-4a99-a9ab-408f9750e5d7","Type":"ContainerStarted","Data":"805b93a94c10853196b6b0abe05c5ad82cc97a48fc5c7fe5f555fed272b25331"}
Jul 28 04:45:26.226242 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.226263    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b42b78c0-5b8875694-2kjwc"
Jul 28 04:45:26.231560 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.231538    2566 scope.go:117] "RemoveContainer" containerID="ae84c3fde53e0bb864affc8a944a83019bd72755a6950c3a0583775374fda583"
Jul 28 04:45:26.242166 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.242113    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-b42b78c0-c5ff7dd4-r8p9k" podStartSLOduration=1.242096792 podStartE2EDuration="1.242096792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:45:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:45:26.240946066 +0000 UTC m=+2026.631122088" watchObservedRunningTime="2026-07-28 04:45:26.242096792 +0000 UTC m=+2026.632272815"
Jul 28 04:45:26.243489 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.243433    2566 scope.go:117] "RemoveContainer" containerID="a0cb890f12798c2af0fe2c76e4537a4bc64b22596d08a80d0a58199a4886e2a6"
Jul 28 04:45:26.243829 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:45:26.243810    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0cb890f12798c2af0fe2c76e4537a4bc64b22596d08a80d0a58199a4886e2a6\": container with ID starting with a0cb890f12798c2af0fe2c76e4537a4bc64b22596d08a80d0a58199a4886e2a6 not found: ID does not exist" containerID="a0cb890f12798c2af0fe2c76e4537a4bc64b22596d08a80d0a58199a4886e2a6"
Jul 28 04:45:26.243920 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.243853    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0cb890f12798c2af0fe2c76e4537a4bc64b22596d08a80d0a58199a4886e2a6"} err="failed to get container status \"a0cb890f12798c2af0fe2c76e4537a4bc64b22596d08a80d0a58199a4886e2a6\": rpc error: code = NotFound desc = could not find container \"a0cb890f12798c2af0fe2c76e4537a4bc64b22596d08a80d0a58199a4886e2a6\": container with ID starting with a0cb890f12798c2af0fe2c76e4537a4bc64b22596d08a80d0a58199a4886e2a6 not found: ID does not exist"
Jul 28 04:45:26.243920 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.243880    2566 scope.go:117] "RemoveContainer" containerID="ae84c3fde53e0bb864affc8a944a83019bd72755a6950c3a0583775374fda583"
Jul 28 04:45:26.244169 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:45:26.244152    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae84c3fde53e0bb864affc8a944a83019bd72755a6950c3a0583775374fda583\": container with ID starting with ae84c3fde53e0bb864affc8a944a83019bd72755a6950c3a0583775374fda583 not found: ID does not exist" containerID="ae84c3fde53e0bb864affc8a944a83019bd72755a6950c3a0583775374fda583"
Jul 28 04:45:26.244214 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.244181    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae84c3fde53e0bb864affc8a944a83019bd72755a6950c3a0583775374fda583"} err="failed to get container status \"ae84c3fde53e0bb864affc8a944a83019bd72755a6950c3a0583775374fda583\": rpc error: code = NotFound desc = could not find container \"ae84c3fde53e0bb864affc8a944a83019bd72755a6950c3a0583775374fda583\": container with ID starting with ae84c3fde53e0bb864affc8a944a83019bd72755a6950c3a0583775374fda583 not found: ID does not exist"
Jul 28 04:45:26.252403 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.252383    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-istiod-ca-cert\") pod \"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c\" (UID: \"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c\") "
Jul 28 04:45:26.252499 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.252413    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-workload-certs\") pod \"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c\" (UID: \"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c\") "
Jul 28 04:45:26.252499 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.252433    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-credential-socket\") pod \"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c\" (UID: \"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c\") "
Jul 28 04:45:26.252499 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.252454    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-istio-token\") pod \"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c\" (UID: \"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c\") "
Jul 28 04:45:26.252499 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.252491    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dwrt\" (UniqueName: \"kubernetes.io/projected/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-kube-api-access-2dwrt\") pod \"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c\" (UID: \"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c\") "
Jul 28 04:45:26.252629 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.252534    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-istio-envoy\") pod \"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c\" (UID: \"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c\") "
Jul 28 04:45:26.252629 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.252559    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-istio-data\") pod \"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c\" (UID: \"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c\") "
Jul 28 04:45:26.252629 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.252574    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-workload-socket\") pod \"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c\" (UID: \"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c\") "
Jul 28 04:45:26.252629 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.252594    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-istio-podinfo\") pod \"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c\" (UID: \"bc1dbec2-43ef-43fd-85bb-f95f8b55e64c\") "
Jul 28 04:45:26.252749 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.252734    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bc1dbec2-43ef-43fd-85bb-f95f8b55e64c" (UID: "bc1dbec2-43ef-43fd-85bb-f95f8b55e64c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 04:45:26.252994 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.252971    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bc1dbec2-43ef-43fd-85bb-f95f8b55e64c" (UID: "bc1dbec2-43ef-43fd-85bb-f95f8b55e64c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:45:26.253037 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.252979    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-istiod-ca-cert\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:45:26.253192 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.253171    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bc1dbec2-43ef-43fd-85bb-f95f8b55e64c" (UID: "bc1dbec2-43ef-43fd-85bb-f95f8b55e64c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:45:26.253753 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.253681    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bc1dbec2-43ef-43fd-85bb-f95f8b55e64c" (UID: "bc1dbec2-43ef-43fd-85bb-f95f8b55e64c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:45:26.254365 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.254342    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-istio-data" (OuterVolumeSpecName: "istio-data") pod "bc1dbec2-43ef-43fd-85bb-f95f8b55e64c" (UID: "bc1dbec2-43ef-43fd-85bb-f95f8b55e64c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:45:26.255632 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.255613    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-istio-token" (OuterVolumeSpecName: "istio-token") pod "bc1dbec2-43ef-43fd-85bb-f95f8b55e64c" (UID: "bc1dbec2-43ef-43fd-85bb-f95f8b55e64c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:45:26.255705 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.255632    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-kube-api-access-2dwrt" (OuterVolumeSpecName: "kube-api-access-2dwrt") pod "bc1dbec2-43ef-43fd-85bb-f95f8b55e64c" (UID: "bc1dbec2-43ef-43fd-85bb-f95f8b55e64c"). InnerVolumeSpecName "kube-api-access-2dwrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:45:26.255742 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.255721    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bc1dbec2-43ef-43fd-85bb-f95f8b55e64c" (UID: "bc1dbec2-43ef-43fd-85bb-f95f8b55e64c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:45:26.256211 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.256177    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bc1dbec2-43ef-43fd-85bb-f95f8b55e64c" (UID: "bc1dbec2-43ef-43fd-85bb-f95f8b55e64c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 04:45:26.259683 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.259643    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b42b78c0-5b8875694-2kjwc" podStartSLOduration=1.872409041 podStartE2EDuration="2.259630354s" totalImagesPullingTime="387.221313ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:45:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:45:26.257323267 +0000 UTC m=+2026.647499306" watchObservedRunningTime="2026-07-28 04:45:26.259630354 +0000 UTC m=+2026.649806374"
Jul 28 04:45:26.353548 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.353514    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-istio-envoy\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:45:26.353548 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.353543    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-istio-data\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:45:26.353548 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.353553    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-workload-socket\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:45:26.353760 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.353561    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-istio-podinfo\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:45:26.353760 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.353570    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-workload-certs\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:45:26.353760 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.353578    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-credential-socket\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:45:26.353760 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.353585    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-istio-token\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:45:26.353760 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.353594    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dwrt\" (UniqueName: \"kubernetes.io/projected/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c-kube-api-access-2dwrt\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:45:26.542814 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.542772    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn"]
Jul 28 04:45:26.545700 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:26.545678    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-5f6a9856-gw-openshift-default-7f4486f956-5f5qn"]
Jul 28 04:45:27.233464 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:27.233364    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b42b78c0-67f6969f48-k8l6w" event={"ID":"600028df-487a-4254-98db-9a2fb5b894a4","Type":"ContainerStarted","Data":"9963085041b395df773da53e4bb07f923d76cc12d4f2186455db9050d5f7d620"}
Jul 28 04:45:27.233464 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:27.233437    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-b42b78c0-67f6969f48-k8l6w"
Jul 28 04:45:27.249655 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:27.249609    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-b42b78c0-67f6969f48-k8l6w" podStartSLOduration=2.249594602 podStartE2EDuration="2.249594602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:45:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:45:27.247667846 +0000 UTC m=+2027.637843879" watchObservedRunningTime="2026-07-28 04:45:27.249594602 +0000 UTC m=+2027.639770622"
Jul 28 04:45:28.088922 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:28.088889    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc1dbec2-43ef-43fd-85bb-f95f8b55e64c" path="/var/lib/kubelet/pods/bc1dbec2-43ef-43fd-85bb-f95f8b55e64c/volumes"
Jul 28 04:45:32.240619 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:32.240586    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b42b78c0-5b8875694-2kjwc"
Jul 28 04:45:38.241522 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:38.241422    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b42b78c0-c5ff7dd4-r8p9k"
Jul 28 04:45:38.241967 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:38.241747    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-b42b78c0-67f6969f48-k8l6w"
Jul 28 04:45:38.292308 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:38.292270    2566 generic.go:356] "Generic (PLEG): container finished" podID="8a62608d-9359-43c9-9cfc-b13b26bb00b8" containerID="5055391c05f868a1363b150596a4753fe4d7432c09680862c67e5147ae6c18aa" exitCode=0
Jul 28 04:45:38.292809 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:38.292349    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp" event={"ID":"8a62608d-9359-43c9-9cfc-b13b26bb00b8","Type":"ContainerDied","Data":"5055391c05f868a1363b150596a4753fe4d7432c09680862c67e5147ae6c18aa"}
Jul 28 04:45:38.292957 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:38.292937    2566 scope.go:117] "RemoveContainer" containerID="5055391c05f868a1363b150596a4753fe4d7432c09680862c67e5147ae6c18aa"
Jul 28 04:45:38.294084 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:38.294056    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:45:38.294184 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:38.294126    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:45:38.294184 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:38.294164    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:45:39.199111 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:39.199080    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp"
Jul 28 04:45:39.302715 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:39.302666    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp" event={"ID":"8a62608d-9359-43c9-9cfc-b13b26bb00b8","Type":"ContainerStarted","Data":"04e07a66b2e080f1aed920028a64b8553327399f7215b94b191b8c33940b5425"}
Jul 28 04:45:40.047271 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:40.047233    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp"
Jul 28 04:45:40.048583 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:40.048557    2566 patch_prober.go:28] interesting pod/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Jul 28 04:45:40.048702 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:40.048602    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp" podUID="8a62608d-9359-43c9-9cfc-b13b26bb00b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Jul 28 04:45:41.047512 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:41.047474    2566 patch_prober.go:28] interesting pod/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Jul 28 04:45:41.047954 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:41.047535    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp" podUID="8a62608d-9359-43c9-9cfc-b13b26bb00b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Jul 28 04:45:42.047809 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:42.047769    2566 patch_prober.go:28] interesting pod/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Jul 28 04:45:42.048232 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:42.047832    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp" podUID="8a62608d-9359-43c9-9cfc-b13b26bb00b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Jul 28 04:45:43.048190 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:43.048157    2566 patch_prober.go:28] interesting pod/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Jul 28 04:45:43.048619 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:43.048221    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp" podUID="8a62608d-9359-43c9-9cfc-b13b26bb00b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Jul 28 04:45:44.051975 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:44.051941    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp"
Jul 28 04:45:44.052347 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:44.052309    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp"
Jul 28 04:45:44.053097 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:45:44.053080    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp"
Jul 28 04:46:00.964934 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:00.964900    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b42b78c0-5b8875694-2kjwc"]
Jul 28 04:46:00.965438 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:00.965253    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b42b78c0-5b8875694-2kjwc" podUID="610b351b-f97f-4a99-a9ab-408f9750e5d7" containerName="maas-api" containerID="cri-o://e4baa7e56c4782936aecf6b7ccf881cc61f82b7fbac3859bb4046b9630966273" gracePeriod=30
Jul 28 04:46:01.019497 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.019461    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b42b78c0-67f6969f48-k8l6w"]
Jul 28 04:46:01.019787 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.019728    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-b42b78c0-67f6969f48-k8l6w" podUID="600028df-487a-4254-98db-9a2fb5b894a4" containerName="payload-processing" containerID="cri-o://9963085041b395df773da53e4bb07f923d76cc12d4f2186455db9050d5f7d620" gracePeriod=30
Jul 28 04:46:01.027575 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.027540    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b42b78c0-c5ff7dd4-r8p9k"]
Jul 28 04:46:01.027737 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.027715    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-b42b78c0-c5ff7dd4-r8p9k" podUID="295c0359-e240-47ca-b8f1-1829a3d86699" containerName="payload-pre-processing" containerID="cri-o://74b9307bcbbdfe423e66f3ea6657fdbed5ba5cc0ee8acde916639e476a4fec3e" gracePeriod=30
Jul 28 04:46:01.295661 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.295638    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b42b78c0-67f6969f48-k8l6w"
Jul 28 04:46:01.312901 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.312615    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/600028df-487a-4254-98db-9a2fb5b894a4-plugins-config-volume\") pod \"600028df-487a-4254-98db-9a2fb5b894a4\" (UID: \"600028df-487a-4254-98db-9a2fb5b894a4\") "
Jul 28 04:46:01.312901 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.312691    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4tmf8\" (UniqueName: \"kubernetes.io/projected/600028df-487a-4254-98db-9a2fb5b894a4-kube-api-access-4tmf8\") pod \"600028df-487a-4254-98db-9a2fb5b894a4\" (UID: \"600028df-487a-4254-98db-9a2fb5b894a4\") "
Jul 28 04:46:01.313141 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.313110    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/600028df-487a-4254-98db-9a2fb5b894a4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "600028df-487a-4254-98db-9a2fb5b894a4" (UID: "600028df-487a-4254-98db-9a2fb5b894a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 04:46:01.315246 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.315214    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/600028df-487a-4254-98db-9a2fb5b894a4-kube-api-access-4tmf8" (OuterVolumeSpecName: "kube-api-access-4tmf8") pod "600028df-487a-4254-98db-9a2fb5b894a4" (UID: "600028df-487a-4254-98db-9a2fb5b894a4"). InnerVolumeSpecName "kube-api-access-4tmf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:46:01.320875 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.320828    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b42b78c0-5b8875694-2kjwc"
Jul 28 04:46:01.329107 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.329087    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b42b78c0-c5ff7dd4-r8p9k"
Jul 28 04:46:01.413984 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.413951    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/610b351b-f97f-4a99-a9ab-408f9750e5d7-maas-api-tls\") pod \"610b351b-f97f-4a99-a9ab-408f9750e5d7\" (UID: \"610b351b-f97f-4a99-a9ab-408f9750e5d7\") "
Jul 28 04:46:01.414181 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.414052    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/295c0359-e240-47ca-b8f1-1829a3d86699-plugins-config-volume\") pod \"295c0359-e240-47ca-b8f1-1829a3d86699\" (UID: \"295c0359-e240-47ca-b8f1-1829a3d86699\") "
Jul 28 04:46:01.414181 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.414080    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/610b351b-f97f-4a99-a9ab-408f9750e5d7-openshift-service-ca\") pod \"610b351b-f97f-4a99-a9ab-408f9750e5d7\" (UID: \"610b351b-f97f-4a99-a9ab-408f9750e5d7\") "
Jul 28 04:46:01.414181 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.414112    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xsbr\" (UniqueName: \"kubernetes.io/projected/610b351b-f97f-4a99-a9ab-408f9750e5d7-kube-api-access-6xsbr\") pod \"610b351b-f97f-4a99-a9ab-408f9750e5d7\" (UID: \"610b351b-f97f-4a99-a9ab-408f9750e5d7\") "
Jul 28 04:46:01.414181 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.414145    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2kqr\" (UniqueName: \"kubernetes.io/projected/295c0359-e240-47ca-b8f1-1829a3d86699-kube-api-access-z2kqr\") pod \"295c0359-e240-47ca-b8f1-1829a3d86699\" (UID: \"295c0359-e240-47ca-b8f1-1829a3d86699\") "
Jul 28 04:46:01.414401 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.414361    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/600028df-487a-4254-98db-9a2fb5b894a4-plugins-config-volume\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:46:01.414401 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.414378    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tmf8\" (UniqueName: \"kubernetes.io/projected/600028df-487a-4254-98db-9a2fb5b894a4-kube-api-access-4tmf8\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:46:01.414511 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.414468    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/610b351b-f97f-4a99-a9ab-408f9750e5d7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "610b351b-f97f-4a99-a9ab-408f9750e5d7" (UID: "610b351b-f97f-4a99-a9ab-408f9750e5d7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 04:46:01.414568 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.414504    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/295c0359-e240-47ca-b8f1-1829a3d86699-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "295c0359-e240-47ca-b8f1-1829a3d86699" (UID: "295c0359-e240-47ca-b8f1-1829a3d86699"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 04:46:01.416332 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.416299    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/610b351b-f97f-4a99-a9ab-408f9750e5d7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "610b351b-f97f-4a99-a9ab-408f9750e5d7" (UID: "610b351b-f97f-4a99-a9ab-408f9750e5d7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 04:46:01.416332 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.416298    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/295c0359-e240-47ca-b8f1-1829a3d86699-kube-api-access-z2kqr" (OuterVolumeSpecName: "kube-api-access-z2kqr") pod "295c0359-e240-47ca-b8f1-1829a3d86699" (UID: "295c0359-e240-47ca-b8f1-1829a3d86699"). InnerVolumeSpecName "kube-api-access-z2kqr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:46:01.416480 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.416340    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/610b351b-f97f-4a99-a9ab-408f9750e5d7-kube-api-access-6xsbr" (OuterVolumeSpecName: "kube-api-access-6xsbr") pod "610b351b-f97f-4a99-a9ab-408f9750e5d7" (UID: "610b351b-f97f-4a99-a9ab-408f9750e5d7"). InnerVolumeSpecName "kube-api-access-6xsbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:46:01.431900 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.431877    2566 generic.go:356] "Generic (PLEG): container finished" podID="600028df-487a-4254-98db-9a2fb5b894a4" containerID="9963085041b395df773da53e4bb07f923d76cc12d4f2186455db9050d5f7d620" exitCode=0
Jul 28 04:46:01.432002 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.431918    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b42b78c0-67f6969f48-k8l6w"
Jul 28 04:46:01.432002 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.431955    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b42b78c0-67f6969f48-k8l6w" event={"ID":"600028df-487a-4254-98db-9a2fb5b894a4","Type":"ContainerDied","Data":"9963085041b395df773da53e4bb07f923d76cc12d4f2186455db9050d5f7d620"}
Jul 28 04:46:01.432118 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.432007    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b42b78c0-67f6969f48-k8l6w" event={"ID":"600028df-487a-4254-98db-9a2fb5b894a4","Type":"ContainerDied","Data":"07c32c97594b53f72b42cac3bf4576c892cd55abcce3b97e3d4b43fea10e3f73"}
Jul 28 04:46:01.432118 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.432034    2566 scope.go:117] "RemoveContainer" containerID="9963085041b395df773da53e4bb07f923d76cc12d4f2186455db9050d5f7d620"
Jul 28 04:46:01.433314 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.433262    2566 generic.go:356] "Generic (PLEG): container finished" podID="295c0359-e240-47ca-b8f1-1829a3d86699" containerID="74b9307bcbbdfe423e66f3ea6657fdbed5ba5cc0ee8acde916639e476a4fec3e" exitCode=0
Jul 28 04:46:01.433314 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.433296    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b42b78c0-c5ff7dd4-r8p9k"
Jul 28 04:46:01.433463 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.433346    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b42b78c0-c5ff7dd4-r8p9k" event={"ID":"295c0359-e240-47ca-b8f1-1829a3d86699","Type":"ContainerDied","Data":"74b9307bcbbdfe423e66f3ea6657fdbed5ba5cc0ee8acde916639e476a4fec3e"}
Jul 28 04:46:01.433463 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.433388    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b42b78c0-c5ff7dd4-r8p9k" event={"ID":"295c0359-e240-47ca-b8f1-1829a3d86699","Type":"ContainerDied","Data":"2b794987dff5367b093b9c9532ad45e2d67d210c9981cdb75c46e198062ef3f9"}
Jul 28 04:46:01.434706 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.434683    2566 generic.go:356] "Generic (PLEG): container finished" podID="610b351b-f97f-4a99-a9ab-408f9750e5d7" containerID="e4baa7e56c4782936aecf6b7ccf881cc61f82b7fbac3859bb4046b9630966273" exitCode=0
Jul 28 04:46:01.434809 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.434726    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b42b78c0-5b8875694-2kjwc"
Jul 28 04:46:01.434809 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.434735    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b42b78c0-5b8875694-2kjwc" event={"ID":"610b351b-f97f-4a99-a9ab-408f9750e5d7","Type":"ContainerDied","Data":"e4baa7e56c4782936aecf6b7ccf881cc61f82b7fbac3859bb4046b9630966273"}
Jul 28 04:46:01.434809 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.434772    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b42b78c0-5b8875694-2kjwc" event={"ID":"610b351b-f97f-4a99-a9ab-408f9750e5d7","Type":"ContainerDied","Data":"805b93a94c10853196b6b0abe05c5ad82cc97a48fc5c7fe5f555fed272b25331"}
Jul 28 04:46:01.442738 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.442716    2566 scope.go:117] "RemoveContainer" containerID="9963085041b395df773da53e4bb07f923d76cc12d4f2186455db9050d5f7d620"
Jul 28 04:46:01.443035 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:46:01.443008    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9963085041b395df773da53e4bb07f923d76cc12d4f2186455db9050d5f7d620\": container with ID starting with 9963085041b395df773da53e4bb07f923d76cc12d4f2186455db9050d5f7d620 not found: ID does not exist" containerID="9963085041b395df773da53e4bb07f923d76cc12d4f2186455db9050d5f7d620"
Jul 28 04:46:01.443124 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.443035    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9963085041b395df773da53e4bb07f923d76cc12d4f2186455db9050d5f7d620"} err="failed to get container status \"9963085041b395df773da53e4bb07f923d76cc12d4f2186455db9050d5f7d620\": rpc error: code = NotFound desc = could not find container \"9963085041b395df773da53e4bb07f923d76cc12d4f2186455db9050d5f7d620\": container with ID starting with 9963085041b395df773da53e4bb07f923d76cc12d4f2186455db9050d5f7d620 not found: ID does not exist"
Jul 28 04:46:01.443124 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.443058    2566 scope.go:117] "RemoveContainer" containerID="74b9307bcbbdfe423e66f3ea6657fdbed5ba5cc0ee8acde916639e476a4fec3e"
Jul 28 04:46:01.451011 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.450998    2566 scope.go:117] "RemoveContainer" containerID="74b9307bcbbdfe423e66f3ea6657fdbed5ba5cc0ee8acde916639e476a4fec3e"
Jul 28 04:46:01.451260 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:46:01.451240    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74b9307bcbbdfe423e66f3ea6657fdbed5ba5cc0ee8acde916639e476a4fec3e\": container with ID starting with 74b9307bcbbdfe423e66f3ea6657fdbed5ba5cc0ee8acde916639e476a4fec3e not found: ID does not exist" containerID="74b9307bcbbdfe423e66f3ea6657fdbed5ba5cc0ee8acde916639e476a4fec3e"
Jul 28 04:46:01.451350 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.451264    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74b9307bcbbdfe423e66f3ea6657fdbed5ba5cc0ee8acde916639e476a4fec3e"} err="failed to get container status \"74b9307bcbbdfe423e66f3ea6657fdbed5ba5cc0ee8acde916639e476a4fec3e\": rpc error: code = NotFound desc = could not find container \"74b9307bcbbdfe423e66f3ea6657fdbed5ba5cc0ee8acde916639e476a4fec3e\": container with ID starting with 74b9307bcbbdfe423e66f3ea6657fdbed5ba5cc0ee8acde916639e476a4fec3e not found: ID does not exist"
Jul 28 04:46:01.451350 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.451279    2566 scope.go:117] "RemoveContainer" containerID="e4baa7e56c4782936aecf6b7ccf881cc61f82b7fbac3859bb4046b9630966273"
Jul 28 04:46:01.459712 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.459693    2566 scope.go:117] "RemoveContainer" containerID="e4baa7e56c4782936aecf6b7ccf881cc61f82b7fbac3859bb4046b9630966273"
Jul 28 04:46:01.459994 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.459950    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b42b78c0-c5ff7dd4-r8p9k"]
Jul 28 04:46:01.460130 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:46:01.460104    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4baa7e56c4782936aecf6b7ccf881cc61f82b7fbac3859bb4046b9630966273\": container with ID starting with e4baa7e56c4782936aecf6b7ccf881cc61f82b7fbac3859bb4046b9630966273 not found: ID does not exist" containerID="e4baa7e56c4782936aecf6b7ccf881cc61f82b7fbac3859bb4046b9630966273"
Jul 28 04:46:01.460188 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.460142    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4baa7e56c4782936aecf6b7ccf881cc61f82b7fbac3859bb4046b9630966273"} err="failed to get container status \"e4baa7e56c4782936aecf6b7ccf881cc61f82b7fbac3859bb4046b9630966273\": rpc error: code = NotFound desc = could not find container \"e4baa7e56c4782936aecf6b7ccf881cc61f82b7fbac3859bb4046b9630966273\": container with ID starting with e4baa7e56c4782936aecf6b7ccf881cc61f82b7fbac3859bb4046b9630966273 not found: ID does not exist"
Jul 28 04:46:01.461992 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.461974    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b42b78c0-c5ff7dd4-r8p9k"]
Jul 28 04:46:01.470629 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.470609    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b42b78c0-67f6969f48-k8l6w"]
Jul 28 04:46:01.474003 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.473984    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b42b78c0-67f6969f48-k8l6w"]
Jul 28 04:46:01.483638 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.483617    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b42b78c0-5b8875694-2kjwc"]
Jul 28 04:46:01.489479 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.489460    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b42b78c0-5b8875694-2kjwc"]
Jul 28 04:46:01.515361 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.515342    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/295c0359-e240-47ca-b8f1-1829a3d86699-plugins-config-volume\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:46:01.515436 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.515363    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/610b351b-f97f-4a99-a9ab-408f9750e5d7-openshift-service-ca\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:46:01.515436 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.515373    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xsbr\" (UniqueName: \"kubernetes.io/projected/610b351b-f97f-4a99-a9ab-408f9750e5d7-kube-api-access-6xsbr\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:46:01.515436 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.515382    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2kqr\" (UniqueName: \"kubernetes.io/projected/295c0359-e240-47ca-b8f1-1829a3d86699-kube-api-access-z2kqr\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:46:01.515436 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:01.515393    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/610b351b-f97f-4a99-a9ab-408f9750e5d7-maas-api-tls\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:46:02.088717 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:02.088687    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="295c0359-e240-47ca-b8f1-1829a3d86699" path="/var/lib/kubelet/pods/295c0359-e240-47ca-b8f1-1829a3d86699/volumes"
Jul 28 04:46:02.089068 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:02.089055    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="600028df-487a-4254-98db-9a2fb5b894a4" path="/var/lib/kubelet/pods/600028df-487a-4254-98db-9a2fb5b894a4/volumes"
Jul 28 04:46:02.089411 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:02.089398    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="610b351b-f97f-4a99-a9ab-408f9750e5d7" path="/var/lib/kubelet/pods/610b351b-f97f-4a99-a9ab-408f9750e5d7/volumes"
Jul 28 04:46:06.902545 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:06.902505    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp"]
Jul 28 04:46:06.903076 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:06.902774    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp" podUID="8a62608d-9359-43c9-9cfc-b13b26bb00b8" containerName="istio-proxy" containerID="cri-o://04e07a66b2e080f1aed920028a64b8553327399f7215b94b191b8c33940b5425" gracePeriod=30
Jul 28 04:46:07.730510 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:07.730479    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw"]
Jul 28 04:46:07.730867 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:07.730830    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc1dbec2-43ef-43fd-85bb-f95f8b55e64c" containerName="istio-proxy"
Jul 28 04:46:07.730867 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:07.730865    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc1dbec2-43ef-43fd-85bb-f95f8b55e64c" containerName="istio-proxy"
Jul 28 04:46:07.731032 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:07.730875    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="295c0359-e240-47ca-b8f1-1829a3d86699" containerName="payload-pre-processing"
Jul 28 04:46:07.731032 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:07.730880    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="295c0359-e240-47ca-b8f1-1829a3d86699" containerName="payload-pre-processing"
Jul 28 04:46:07.731032 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:07.730903    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="610b351b-f97f-4a99-a9ab-408f9750e5d7" containerName="maas-api"
Jul 28 04:46:07.731032 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:07.730911    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="610b351b-f97f-4a99-a9ab-408f9750e5d7" containerName="maas-api"
Jul 28 04:46:07.731032 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:07.730924    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="600028df-487a-4254-98db-9a2fb5b894a4" containerName="payload-processing"
Jul 28 04:46:07.731032 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:07.730929    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="600028df-487a-4254-98db-9a2fb5b894a4" containerName="payload-processing"
Jul 28 04:46:07.731032 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:07.730984    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="610b351b-f97f-4a99-a9ab-408f9750e5d7" containerName="maas-api"
Jul 28 04:46:07.731032 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:07.730994    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc1dbec2-43ef-43fd-85bb-f95f8b55e64c" containerName="istio-proxy"
Jul 28 04:46:07.731032 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:07.731000    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc1dbec2-43ef-43fd-85bb-f95f8b55e64c" containerName="istio-proxy"
Jul 28 04:46:07.731032 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:07.731006    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="295c0359-e240-47ca-b8f1-1829a3d86699" containerName="payload-pre-processing"
Jul 28 04:46:07.731032 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:07.731011    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="600028df-487a-4254-98db-9a2fb5b894a4" containerName="payload-processing"
Jul 28 04:46:07.735715 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:07.735694    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw"
Jul 28 04:46:07.738034 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:07.738016    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-ac8148f2-openshift-default-dockercfg-4d4rb\""
Jul 28 04:46:07.746418 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:07.746395    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw"]
Jul 28 04:46:07.769945 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:07.769920    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da666aad-410e-4a82-bdf9-aa79cd238520-workload-socket\") pod \"e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw\" (UID: \"da666aad-410e-4a82-bdf9-aa79cd238520\") " pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw"
Jul 28 04:46:07.770050 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:07.769976    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da666aad-410e-4a82-bdf9-aa79cd238520-credential-socket\") pod \"e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw\" (UID: \"da666aad-410e-4a82-bdf9-aa79cd238520\") " pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw"
Jul 28 04:46:07.770050 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:07.770000    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da666aad-410e-4a82-bdf9-aa79cd238520-istiod-ca-cert\") pod \"e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw\" (UID: \"da666aad-410e-4a82-bdf9-aa79cd238520\") " pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw"
Jul 28 04:46:07.770128 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:07.770074    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjgrm\" (UniqueName: \"kubernetes.io/projected/da666aad-410e-4a82-bdf9-aa79cd238520-kube-api-access-gjgrm\") pod \"e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw\" (UID: \"da666aad-410e-4a82-bdf9-aa79cd238520\") " pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw"
Jul 28 04:46:07.770128 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:07.770121    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da666aad-410e-4a82-bdf9-aa79cd238520-istio-token\") pod \"e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw\" (UID: \"da666aad-410e-4a82-bdf9-aa79cd238520\") " pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw"
Jul 28 04:46:07.770204 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:07.770144    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da666aad-410e-4a82-bdf9-aa79cd238520-istio-podinfo\") pod \"e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw\" (UID: \"da666aad-410e-4a82-bdf9-aa79cd238520\") " pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw"
Jul 28 04:46:07.770245 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:07.770194    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da666aad-410e-4a82-bdf9-aa79cd238520-workload-certs\") pod \"e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw\" (UID: \"da666aad-410e-4a82-bdf9-aa79cd238520\") " pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw"
Jul 28 04:46:07.770290 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:07.770256    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da666aad-410e-4a82-bdf9-aa79cd238520-istio-data\") pod \"e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw\" (UID: \"da666aad-410e-4a82-bdf9-aa79cd238520\") " pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw"
Jul 28 04:46:07.770329 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:07.770291    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da666aad-410e-4a82-bdf9-aa79cd238520-istio-envoy\") pod \"e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw\" (UID: \"da666aad-410e-4a82-bdf9-aa79cd238520\") " pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw"
Jul 28 04:46:07.871410 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:07.871371    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da666aad-410e-4a82-bdf9-aa79cd238520-credential-socket\") pod \"e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw\" (UID: \"da666aad-410e-4a82-bdf9-aa79cd238520\") " pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw"
Jul 28 04:46:07.871607 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:07.871418    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da666aad-410e-4a82-bdf9-aa79cd238520-istiod-ca-cert\") pod \"e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw\" (UID: \"da666aad-410e-4a82-bdf9-aa79cd238520\") " pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw"
Jul 28 04:46:07.871607 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:07.871464    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjgrm\" (UniqueName: \"kubernetes.io/projected/da666aad-410e-4a82-bdf9-aa79cd238520-kube-api-access-gjgrm\") pod \"e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw\" (UID: \"da666aad-410e-4a82-bdf9-aa79cd238520\") " pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw"
Jul 28 04:46:07.871607 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:07.871499    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da666aad-410e-4a82-bdf9-aa79cd238520-istio-token\") pod \"e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw\" (UID: \"da666aad-410e-4a82-bdf9-aa79cd238520\") " pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw"
Jul 28 04:46:07.871797 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:07.871635    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da666aad-410e-4a82-bdf9-aa79cd238520-istio-podinfo\") pod \"e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw\" (UID: \"da666aad-410e-4a82-bdf9-aa79cd238520\") " pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw"
Jul 28 04:46:07.871797 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:07.871697    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da666aad-410e-4a82-bdf9-aa79cd238520-workload-certs\") pod \"e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw\" (UID: \"da666aad-410e-4a82-bdf9-aa79cd238520\") " pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw"
Jul 28 04:46:07.871797 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:07.871766    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da666aad-410e-4a82-bdf9-aa79cd238520-istio-data\") pod \"e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw\" (UID: \"da666aad-410e-4a82-bdf9-aa79cd238520\") " pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw"
Jul 28 04:46:07.872003 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:07.871803    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da666aad-410e-4a82-bdf9-aa79cd238520-istio-envoy\") pod \"e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw\" (UID: \"da666aad-410e-4a82-bdf9-aa79cd238520\") " pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw"
Jul 28 04:46:07.872003 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:07.871896    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da666aad-410e-4a82-bdf9-aa79cd238520-workload-socket\") pod \"e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw\" (UID: \"da666aad-410e-4a82-bdf9-aa79cd238520\") " pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw"
Jul 28 04:46:07.872003 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:07.871920    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da666aad-410e-4a82-bdf9-aa79cd238520-credential-socket\") pod \"e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw\" (UID: \"da666aad-410e-4a82-bdf9-aa79cd238520\") " pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw"
Jul 28 04:46:07.872171 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:07.872067    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da666aad-410e-4a82-bdf9-aa79cd238520-workload-certs\") pod \"e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw\" (UID: \"da666aad-410e-4a82-bdf9-aa79cd238520\") " pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw"
Jul 28 04:46:07.872171 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:07.872135    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da666aad-410e-4a82-bdf9-aa79cd238520-istio-data\") pod \"e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw\" (UID: \"da666aad-410e-4a82-bdf9-aa79cd238520\") " pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw"
Jul 28 04:46:07.872267 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:07.872235    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da666aad-410e-4a82-bdf9-aa79cd238520-workload-socket\") pod \"e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw\" (UID: \"da666aad-410e-4a82-bdf9-aa79cd238520\") " pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw"
Jul 28 04:46:07.872267 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:07.872250    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da666aad-410e-4a82-bdf9-aa79cd238520-istiod-ca-cert\") pod \"e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw\" (UID: \"da666aad-410e-4a82-bdf9-aa79cd238520\") " pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw"
Jul 28 04:46:07.874048 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:07.874022    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da666aad-410e-4a82-bdf9-aa79cd238520-istio-envoy\") pod \"e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw\" (UID: \"da666aad-410e-4a82-bdf9-aa79cd238520\") " pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw"
Jul 28 04:46:07.874158 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:07.874138    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da666aad-410e-4a82-bdf9-aa79cd238520-istio-podinfo\") pod \"e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw\" (UID: \"da666aad-410e-4a82-bdf9-aa79cd238520\") " pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw"
Jul 28 04:46:07.878951 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:07.878928    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjgrm\" (UniqueName: \"kubernetes.io/projected/da666aad-410e-4a82-bdf9-aa79cd238520-kube-api-access-gjgrm\") pod \"e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw\" (UID: \"da666aad-410e-4a82-bdf9-aa79cd238520\") " pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw"
Jul 28 04:46:07.879043 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:07.878950    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da666aad-410e-4a82-bdf9-aa79cd238520-istio-token\") pod \"e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw\" (UID: \"da666aad-410e-4a82-bdf9-aa79cd238520\") " pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw"
Jul 28 04:46:08.047698 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:08.047669    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw"
Jul 28 04:46:08.204506 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:08.204476    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw"]
Jul 28 04:46:08.207695 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:46:08.207663    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda666aad_410e_4a82_bdf9_aa79cd238520.slice/crio-392f19a5583d98aeed927faf678696e9b973e9eac5a1bff552d5a95ee48f9027 WatchSource:0}: Error finding container 392f19a5583d98aeed927faf678696e9b973e9eac5a1bff552d5a95ee48f9027: Status 404 returned error can't find the container with id 392f19a5583d98aeed927faf678696e9b973e9eac5a1bff552d5a95ee48f9027
Jul 28 04:46:08.210014 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:08.209975    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:46:08.210108 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:08.210067    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:46:08.210176 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:08.210123    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:46:08.479245 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:08.479154    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw" event={"ID":"da666aad-410e-4a82-bdf9-aa79cd238520","Type":"ContainerStarted","Data":"73f08a25aa75aeaceece5c96c11b859461049bf85013b08b8a8edfd29bb073b3"}
Jul 28 04:46:08.479245 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:08.479215    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw" event={"ID":"da666aad-410e-4a82-bdf9-aa79cd238520","Type":"ContainerStarted","Data":"392f19a5583d98aeed927faf678696e9b973e9eac5a1bff552d5a95ee48f9027"}
Jul 28 04:46:08.500533 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:08.500343    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw" podStartSLOduration=1.500326237 podStartE2EDuration="1.500326237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:46:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:46:08.500239614 +0000 UTC m=+2068.890415638" watchObservedRunningTime="2026-07-28 04:46:08.500326237 +0000 UTC m=+2068.890502257"
Jul 28 04:46:09.048248 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:09.048216    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw"
Jul 28 04:46:09.053161 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:09.053139    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw"
Jul 28 04:46:09.484529 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:09.484455    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw"
Jul 28 04:46:09.485603 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:09.485580    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw"
Jul 28 04:46:10.166282 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:10.166242    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ac8148f2-766fb89d6-jw79t"]
Jul 28 04:46:10.166826 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:10.166805    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc1dbec2-43ef-43fd-85bb-f95f8b55e64c" containerName="istio-proxy"
Jul 28 04:46:10.166943 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:10.166829    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc1dbec2-43ef-43fd-85bb-f95f8b55e64c" containerName="istio-proxy"
Jul 28 04:46:10.170121 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:10.170099    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ac8148f2-766fb89d6-jw79t"
Jul 28 04:46:10.173480 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:10.173450    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 04:46:10.173756 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:10.173734    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gfn77\""
Jul 28 04:46:10.174283 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:10.174267    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-ac8148f2-serving-cert\""
Jul 28 04:46:10.174364 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:10.174309    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 04:46:10.180579 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:10.180558    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ac8148f2-766fb89d6-jw79t"]
Jul 28 04:46:10.295893 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:10.295852    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcda0637-d837-4a0d-819a-0dcc264c3ddb-maas-api-tls\") pod \"maas-api-e2e-derive-ac8148f2-766fb89d6-jw79t\" (UID: \"bcda0637-d837-4a0d-819a-0dcc264c3ddb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ac8148f2-766fb89d6-jw79t"
Jul 28 04:46:10.296079 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:10.295908    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-872wl\" (UniqueName: \"kubernetes.io/projected/bcda0637-d837-4a0d-819a-0dcc264c3ddb-kube-api-access-872wl\") pod \"maas-api-e2e-derive-ac8148f2-766fb89d6-jw79t\" (UID: \"bcda0637-d837-4a0d-819a-0dcc264c3ddb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ac8148f2-766fb89d6-jw79t"
Jul 28 04:46:10.296079 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:10.295948    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcda0637-d837-4a0d-819a-0dcc264c3ddb-openshift-service-ca\") pod \"maas-api-e2e-derive-ac8148f2-766fb89d6-jw79t\" (UID: \"bcda0637-d837-4a0d-819a-0dcc264c3ddb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ac8148f2-766fb89d6-jw79t"
Jul 28 04:46:10.397182 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:10.397145    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcda0637-d837-4a0d-819a-0dcc264c3ddb-maas-api-tls\") pod \"maas-api-e2e-derive-ac8148f2-766fb89d6-jw79t\" (UID: \"bcda0637-d837-4a0d-819a-0dcc264c3ddb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ac8148f2-766fb89d6-jw79t"
Jul 28 04:46:10.397182 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:10.397187    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-872wl\" (UniqueName: \"kubernetes.io/projected/bcda0637-d837-4a0d-819a-0dcc264c3ddb-kube-api-access-872wl\") pod \"maas-api-e2e-derive-ac8148f2-766fb89d6-jw79t\" (UID: \"bcda0637-d837-4a0d-819a-0dcc264c3ddb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ac8148f2-766fb89d6-jw79t"
Jul 28 04:46:10.397448 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:10.397214    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcda0637-d837-4a0d-819a-0dcc264c3ddb-openshift-service-ca\") pod \"maas-api-e2e-derive-ac8148f2-766fb89d6-jw79t\" (UID: \"bcda0637-d837-4a0d-819a-0dcc264c3ddb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ac8148f2-766fb89d6-jw79t"
Jul 28 04:46:10.397448 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:46:10.397308    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-ac8148f2-serving-cert: secret "maas-api-e2e-derive-ac8148f2-serving-cert" not found
Jul 28 04:46:10.397448 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:46:10.397407    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcda0637-d837-4a0d-819a-0dcc264c3ddb-maas-api-tls podName:bcda0637-d837-4a0d-819a-0dcc264c3ddb nodeName:}" failed. No retries permitted until 2026-07-28 04:46:10.897384445 +0000 UTC m=+2071.287560444 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bcda0637-d837-4a0d-819a-0dcc264c3ddb-maas-api-tls") pod "maas-api-e2e-derive-ac8148f2-766fb89d6-jw79t" (UID: "bcda0637-d837-4a0d-819a-0dcc264c3ddb") : secret "maas-api-e2e-derive-ac8148f2-serving-cert" not found
Jul 28 04:46:10.397883 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:10.397834    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcda0637-d837-4a0d-819a-0dcc264c3ddb-openshift-service-ca\") pod \"maas-api-e2e-derive-ac8148f2-766fb89d6-jw79t\" (UID: \"bcda0637-d837-4a0d-819a-0dcc264c3ddb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ac8148f2-766fb89d6-jw79t"
Jul 28 04:46:10.405798 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:10.405775    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-872wl\" (UniqueName: \"kubernetes.io/projected/bcda0637-d837-4a0d-819a-0dcc264c3ddb-kube-api-access-872wl\") pod \"maas-api-e2e-derive-ac8148f2-766fb89d6-jw79t\" (UID: \"bcda0637-d837-4a0d-819a-0dcc264c3ddb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ac8148f2-766fb89d6-jw79t"
Jul 28 04:46:10.844155 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:10.844118    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ac8148f2-77879b4d68-qwpdl"]
Jul 28 04:46:10.847737 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:10.847720    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ac8148f2-77879b4d68-qwpdl"
Jul 28 04:46:10.850046 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:10.850016    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-ac8148f2\""
Jul 28 04:46:10.850198 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:10.850163    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-ac8148f2-dockercfg-7l64z\""
Jul 28 04:46:10.857892 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:10.857866    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ac8148f2-77879b4d68-qwpdl"]
Jul 28 04:46:10.901405 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:10.901373    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab32ed5-3b6e-489d-b5a4-74f14daa836c-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ac8148f2-77879b4d68-qwpdl\" (UID: \"6ab32ed5-3b6e-489d-b5a4-74f14daa836c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ac8148f2-77879b4d68-qwpdl"
Jul 28 04:46:10.901574 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:10.901453    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4cf9\" (UniqueName: \"kubernetes.io/projected/6ab32ed5-3b6e-489d-b5a4-74f14daa836c-kube-api-access-h4cf9\") pod \"payload-pre-processing-e2e-derive-ac8148f2-77879b4d68-qwpdl\" (UID: \"6ab32ed5-3b6e-489d-b5a4-74f14daa836c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ac8148f2-77879b4d68-qwpdl"
Jul 28 04:46:10.901574 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:10.901482    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcda0637-d837-4a0d-819a-0dcc264c3ddb-maas-api-tls\") pod \"maas-api-e2e-derive-ac8148f2-766fb89d6-jw79t\" (UID: \"bcda0637-d837-4a0d-819a-0dcc264c3ddb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ac8148f2-766fb89d6-jw79t"
Jul 28 04:46:10.903811 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:10.903791    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcda0637-d837-4a0d-819a-0dcc264c3ddb-maas-api-tls\") pod \"maas-api-e2e-derive-ac8148f2-766fb89d6-jw79t\" (UID: \"bcda0637-d837-4a0d-819a-0dcc264c3ddb\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ac8148f2-766fb89d6-jw79t"
Jul 28 04:46:11.002980 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:11.002930    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab32ed5-3b6e-489d-b5a4-74f14daa836c-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ac8148f2-77879b4d68-qwpdl\" (UID: \"6ab32ed5-3b6e-489d-b5a4-74f14daa836c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ac8148f2-77879b4d68-qwpdl"
Jul 28 04:46:11.003155 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:11.003093    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4cf9\" (UniqueName: \"kubernetes.io/projected/6ab32ed5-3b6e-489d-b5a4-74f14daa836c-kube-api-access-h4cf9\") pod \"payload-pre-processing-e2e-derive-ac8148f2-77879b4d68-qwpdl\" (UID: \"6ab32ed5-3b6e-489d-b5a4-74f14daa836c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ac8148f2-77879b4d68-qwpdl"
Jul 28 04:46:11.003766 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:11.003746    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab32ed5-3b6e-489d-b5a4-74f14daa836c-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ac8148f2-77879b4d68-qwpdl\" (UID: \"6ab32ed5-3b6e-489d-b5a4-74f14daa836c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ac8148f2-77879b4d68-qwpdl"
Jul 28 04:46:11.011253 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:11.011228    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4cf9\" (UniqueName: \"kubernetes.io/projected/6ab32ed5-3b6e-489d-b5a4-74f14daa836c-kube-api-access-h4cf9\") pod \"payload-pre-processing-e2e-derive-ac8148f2-77879b4d68-qwpdl\" (UID: \"6ab32ed5-3b6e-489d-b5a4-74f14daa836c\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ac8148f2-77879b4d68-qwpdl"
Jul 28 04:46:11.081968 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:11.081933    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ac8148f2-766fb89d6-jw79t"
Jul 28 04:46:11.159401 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:11.159359    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ac8148f2-77879b4d68-qwpdl"
Jul 28 04:46:11.222233 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:11.222185    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ac8148f2-766fb89d6-jw79t"]
Jul 28 04:46:11.303068 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:11.302938    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ac8148f2-77879b4d68-qwpdl"]
Jul 28 04:46:11.305248 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:46:11.305220    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ab32ed5_3b6e_489d_b5a4_74f14daa836c.slice/crio-6b8ca6f7c18dbe9b4c2f75829ad04bc32c83e48aeda5e49c12017a8eadd29771 WatchSource:0}: Error finding container 6b8ca6f7c18dbe9b4c2f75829ad04bc32c83e48aeda5e49c12017a8eadd29771: Status 404 returned error can't find the container with id 6b8ca6f7c18dbe9b4c2f75829ad04bc32c83e48aeda5e49c12017a8eadd29771
Jul 28 04:46:11.499033 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:11.498946    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ac8148f2-766fb89d6-jw79t" event={"ID":"bcda0637-d837-4a0d-819a-0dcc264c3ddb","Type":"ContainerStarted","Data":"77256b48c100db245b2db3906a1516ee2d3f02f83a7ff8197ec39b5d98e2eed7"}
Jul 28 04:46:11.500399 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:11.500368    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ac8148f2-77879b4d68-qwpdl" event={"ID":"6ab32ed5-3b6e-489d-b5a4-74f14daa836c","Type":"ContainerStarted","Data":"72816a32ee0e1cf949331a847f5a998bf3203998c4f8ac57c2aabed630732832"}
Jul 28 04:46:11.500538 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:11.500420    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ac8148f2-77879b4d68-qwpdl" event={"ID":"6ab32ed5-3b6e-489d-b5a4-74f14daa836c","Type":"ContainerStarted","Data":"6b8ca6f7c18dbe9b4c2f75829ad04bc32c83e48aeda5e49c12017a8eadd29771"}
Jul 28 04:46:11.500538 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:11.500450    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-ac8148f2-77879b4d68-qwpdl"
Jul 28 04:46:11.517537 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:11.517494    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-ac8148f2-77879b4d68-qwpdl" podStartSLOduration=1.517479502 podStartE2EDuration="1.517479502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:46:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:46:11.516184371 +0000 UTC m=+2071.906360395" watchObservedRunningTime="2026-07-28 04:46:11.517479502 +0000 UTC m=+2071.907655523"
Jul 28 04:46:12.158236 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:12.158212    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp"
Jul 28 04:46:12.214963 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:12.214876    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a62608d-9359-43c9-9cfc-b13b26bb00b8-istio-token\") pod \"8a62608d-9359-43c9-9cfc-b13b26bb00b8\" (UID: \"8a62608d-9359-43c9-9cfc-b13b26bb00b8\") "
Jul 28 04:46:12.214963 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:12.214947    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a62608d-9359-43c9-9cfc-b13b26bb00b8-istio-data\") pod \"8a62608d-9359-43c9-9cfc-b13b26bb00b8\" (UID: \"8a62608d-9359-43c9-9cfc-b13b26bb00b8\") "
Jul 28 04:46:12.215177 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:12.214977    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a62608d-9359-43c9-9cfc-b13b26bb00b8-workload-socket\") pod \"8a62608d-9359-43c9-9cfc-b13b26bb00b8\" (UID: \"8a62608d-9359-43c9-9cfc-b13b26bb00b8\") "
Jul 28 04:46:12.215177 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:12.214999    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a62608d-9359-43c9-9cfc-b13b26bb00b8-credential-socket\") pod \"8a62608d-9359-43c9-9cfc-b13b26bb00b8\" (UID: \"8a62608d-9359-43c9-9cfc-b13b26bb00b8\") "
Jul 28 04:46:12.215177 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:12.215068    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r77jd\" (UniqueName: \"kubernetes.io/projected/8a62608d-9359-43c9-9cfc-b13b26bb00b8-kube-api-access-r77jd\") pod \"8a62608d-9359-43c9-9cfc-b13b26bb00b8\" (UID: \"8a62608d-9359-43c9-9cfc-b13b26bb00b8\") "
Jul 28 04:46:12.215177 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:12.215122    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a62608d-9359-43c9-9cfc-b13b26bb00b8-workload-certs\") pod \"8a62608d-9359-43c9-9cfc-b13b26bb00b8\" (UID: \"8a62608d-9359-43c9-9cfc-b13b26bb00b8\") "
Jul 28 04:46:12.215177 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:12.215158    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a62608d-9359-43c9-9cfc-b13b26bb00b8-istio-podinfo\") pod \"8a62608d-9359-43c9-9cfc-b13b26bb00b8\" (UID: \"8a62608d-9359-43c9-9cfc-b13b26bb00b8\") "
Jul 28 04:46:12.215495 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:12.215189    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a62608d-9359-43c9-9cfc-b13b26bb00b8-istiod-ca-cert\") pod \"8a62608d-9359-43c9-9cfc-b13b26bb00b8\" (UID: \"8a62608d-9359-43c9-9cfc-b13b26bb00b8\") "
Jul 28 04:46:12.215495 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:12.215211    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a62608d-9359-43c9-9cfc-b13b26bb00b8-istio-envoy\") pod \"8a62608d-9359-43c9-9cfc-b13b26bb00b8\" (UID: \"8a62608d-9359-43c9-9cfc-b13b26bb00b8\") "
Jul 28 04:46:12.215495 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:12.215279    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a62608d-9359-43c9-9cfc-b13b26bb00b8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8a62608d-9359-43c9-9cfc-b13b26bb00b8" (UID: "8a62608d-9359-43c9-9cfc-b13b26bb00b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:46:12.215495 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:12.215330    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a62608d-9359-43c9-9cfc-b13b26bb00b8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8a62608d-9359-43c9-9cfc-b13b26bb00b8" (UID: "8a62608d-9359-43c9-9cfc-b13b26bb00b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:46:12.215711 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:12.215674    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a62608d-9359-43c9-9cfc-b13b26bb00b8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8a62608d-9359-43c9-9cfc-b13b26bb00b8" (UID: "8a62608d-9359-43c9-9cfc-b13b26bb00b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 04:46:12.215711 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:12.215695    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a62608d-9359-43c9-9cfc-b13b26bb00b8-workload-socket\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:46:12.215822 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:12.215721    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a62608d-9359-43c9-9cfc-b13b26bb00b8-credential-socket\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:46:12.215996 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:12.215952    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a62608d-9359-43c9-9cfc-b13b26bb00b8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8a62608d-9359-43c9-9cfc-b13b26bb00b8" (UID: "8a62608d-9359-43c9-9cfc-b13b26bb00b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:46:12.216101 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:12.216055    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a62608d-9359-43c9-9cfc-b13b26bb00b8-istio-data" (OuterVolumeSpecName: "istio-data") pod "8a62608d-9359-43c9-9cfc-b13b26bb00b8" (UID: "8a62608d-9359-43c9-9cfc-b13b26bb00b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:46:12.217777 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:12.217749    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a62608d-9359-43c9-9cfc-b13b26bb00b8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8a62608d-9359-43c9-9cfc-b13b26bb00b8" (UID: "8a62608d-9359-43c9-9cfc-b13b26bb00b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 04:46:12.217971 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:12.217914    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a62608d-9359-43c9-9cfc-b13b26bb00b8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8a62608d-9359-43c9-9cfc-b13b26bb00b8" (UID: "8a62608d-9359-43c9-9cfc-b13b26bb00b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:46:12.218317 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:12.218286    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a62608d-9359-43c9-9cfc-b13b26bb00b8-istio-token" (OuterVolumeSpecName: "istio-token") pod "8a62608d-9359-43c9-9cfc-b13b26bb00b8" (UID: "8a62608d-9359-43c9-9cfc-b13b26bb00b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:46:12.218317 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:12.218286    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a62608d-9359-43c9-9cfc-b13b26bb00b8-kube-api-access-r77jd" (OuterVolumeSpecName: "kube-api-access-r77jd") pod "8a62608d-9359-43c9-9cfc-b13b26bb00b8" (UID: "8a62608d-9359-43c9-9cfc-b13b26bb00b8"). InnerVolumeSpecName "kube-api-access-r77jd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:46:12.316902 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:12.316870    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r77jd\" (UniqueName: \"kubernetes.io/projected/8a62608d-9359-43c9-9cfc-b13b26bb00b8-kube-api-access-r77jd\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:46:12.316902 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:12.316894    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a62608d-9359-43c9-9cfc-b13b26bb00b8-workload-certs\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:46:12.316902 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:12.316904    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a62608d-9359-43c9-9cfc-b13b26bb00b8-istio-podinfo\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:46:12.317334 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:12.316912    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a62608d-9359-43c9-9cfc-b13b26bb00b8-istiod-ca-cert\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:46:12.317334 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:12.316921    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a62608d-9359-43c9-9cfc-b13b26bb00b8-istio-envoy\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:46:12.317334 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:12.316928    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a62608d-9359-43c9-9cfc-b13b26bb00b8-istio-token\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:46:12.317334 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:12.316935    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a62608d-9359-43c9-9cfc-b13b26bb00b8-istio-data\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:46:12.508547 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:12.508374    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ac8148f2-766fb89d6-jw79t" event={"ID":"bcda0637-d837-4a0d-819a-0dcc264c3ddb","Type":"ContainerStarted","Data":"3de5a9821110b009e8f8958bd337db508cf916fcd8b731e2ba36070f50c494a5"}
Jul 28 04:46:12.508547 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:12.508455    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ac8148f2-766fb89d6-jw79t"
Jul 28 04:46:12.510770 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:12.510739    2566 generic.go:356] "Generic (PLEG): container finished" podID="8a62608d-9359-43c9-9cfc-b13b26bb00b8" containerID="04e07a66b2e080f1aed920028a64b8553327399f7215b94b191b8c33940b5425" exitCode=0
Jul 28 04:46:12.511009 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:12.510784    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp"
Jul 28 04:46:12.511009 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:12.510825    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp" event={"ID":"8a62608d-9359-43c9-9cfc-b13b26bb00b8","Type":"ContainerDied","Data":"04e07a66b2e080f1aed920028a64b8553327399f7215b94b191b8c33940b5425"}
Jul 28 04:46:12.511009 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:12.510910    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp" event={"ID":"8a62608d-9359-43c9-9cfc-b13b26bb00b8","Type":"ContainerDied","Data":"25f087b94ca6701f4c4da033584681c8f1d36f4bfde6916905a521e164b799ca"}
Jul 28 04:46:12.511009 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:12.510943    2566 scope.go:117] "RemoveContainer" containerID="04e07a66b2e080f1aed920028a64b8553327399f7215b94b191b8c33940b5425"
Jul 28 04:46:12.523883 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:12.523865    2566 scope.go:117] "RemoveContainer" containerID="5055391c05f868a1363b150596a4753fe4d7432c09680862c67e5147ae6c18aa"
Jul 28 04:46:12.526252 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:12.526213    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ac8148f2-766fb89d6-jw79t" podStartSLOduration=2.092815734 podStartE2EDuration="2.52619654s" totalImagesPullingTime="433.380806ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:46:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:46:12.52537838 +0000 UTC m=+2072.915554401" watchObservedRunningTime="2026-07-28 04:46:12.52619654 +0000 UTC m=+2072.916372562"
Jul 28 04:46:12.532314 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:12.532295    2566 scope.go:117] "RemoveContainer" containerID="04e07a66b2e080f1aed920028a64b8553327399f7215b94b191b8c33940b5425"
Jul 28 04:46:12.532706 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:46:12.532680    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04e07a66b2e080f1aed920028a64b8553327399f7215b94b191b8c33940b5425\": container with ID starting with 04e07a66b2e080f1aed920028a64b8553327399f7215b94b191b8c33940b5425 not found: ID does not exist" containerID="04e07a66b2e080f1aed920028a64b8553327399f7215b94b191b8c33940b5425"
Jul 28 04:46:12.532806 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:12.532714    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04e07a66b2e080f1aed920028a64b8553327399f7215b94b191b8c33940b5425"} err="failed to get container status \"04e07a66b2e080f1aed920028a64b8553327399f7215b94b191b8c33940b5425\": rpc error: code = NotFound desc = could not find container \"04e07a66b2e080f1aed920028a64b8553327399f7215b94b191b8c33940b5425\": container with ID starting with 04e07a66b2e080f1aed920028a64b8553327399f7215b94b191b8c33940b5425 not found: ID does not exist"
Jul 28 04:46:12.532806 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:12.532736    2566 scope.go:117] "RemoveContainer" containerID="5055391c05f868a1363b150596a4753fe4d7432c09680862c67e5147ae6c18aa"
Jul 28 04:46:12.533061 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:46:12.533040    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5055391c05f868a1363b150596a4753fe4d7432c09680862c67e5147ae6c18aa\": container with ID starting with 5055391c05f868a1363b150596a4753fe4d7432c09680862c67e5147ae6c18aa not found: ID does not exist" containerID="5055391c05f868a1363b150596a4753fe4d7432c09680862c67e5147ae6c18aa"
Jul 28 04:46:12.533110 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:12.533069    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5055391c05f868a1363b150596a4753fe4d7432c09680862c67e5147ae6c18aa"} err="failed to get container status \"5055391c05f868a1363b150596a4753fe4d7432c09680862c67e5147ae6c18aa\": rpc error: code = NotFound desc = could not find container \"5055391c05f868a1363b150596a4753fe4d7432c09680862c67e5147ae6c18aa\": container with ID starting with 5055391c05f868a1363b150596a4753fe4d7432c09680862c67e5147ae6c18aa not found: ID does not exist"
Jul 28 04:46:12.539080 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:12.539057    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp"]
Jul 28 04:46:12.542508 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:12.542489    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b42b78c0-openshift-default-554ddd5844-d22bp"]
Jul 28 04:46:14.088179 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:14.088145    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a62608d-9359-43c9-9cfc-b13b26bb00b8" path="/var/lib/kubelet/pods/8a62608d-9359-43c9-9cfc-b13b26bb00b8/volumes"
Jul 28 04:46:18.529584 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:18.529544    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ac8148f2-766fb89d6-jw79t"
Jul 28 04:46:22.513821 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:22.513789    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-ac8148f2-77879b4d68-qwpdl"
Jul 28 04:46:23.574929 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:23.574819    2566 generic.go:356] "Generic (PLEG): container finished" podID="da666aad-410e-4a82-bdf9-aa79cd238520" containerID="73f08a25aa75aeaceece5c96c11b859461049bf85013b08b8a8edfd29bb073b3" exitCode=0
Jul 28 04:46:23.574929 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:23.574903    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw" event={"ID":"da666aad-410e-4a82-bdf9-aa79cd238520","Type":"ContainerDied","Data":"73f08a25aa75aeaceece5c96c11b859461049bf85013b08b8a8edfd29bb073b3"}
Jul 28 04:46:23.575683 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:23.575451    2566 scope.go:117] "RemoveContainer" containerID="73f08a25aa75aeaceece5c96c11b859461049bf85013b08b8a8edfd29bb073b3"
Jul 28 04:46:23.576534 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:23.576507    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:46:23.576603 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:23.576572    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:46:23.576645 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:23.576602    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:46:24.484939 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:24.484907    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw"
Jul 28 04:46:24.581937 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:24.581905    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw" event={"ID":"da666aad-410e-4a82-bdf9-aa79cd238520","Type":"ContainerStarted","Data":"f57ecba6e7d3cba305f00c01acd54c32d7d9ba7605b5b133a5173d03a86911eb"}
Jul 28 04:46:25.047951 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:25.047915    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw"
Jul 28 04:46:25.049362 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:25.049337    2566 patch_prober.go:28] interesting pod/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Jul 28 04:46:25.049497 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:25.049387    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw" podUID="da666aad-410e-4a82-bdf9-aa79cd238520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Jul 28 04:46:26.049049 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:26.049008    2566 patch_prober.go:28] interesting pod/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Jul 28 04:46:26.049473 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:26.049068    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw" podUID="da666aad-410e-4a82-bdf9-aa79cd238520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Jul 28 04:46:27.048673 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:27.048635    2566 patch_prober.go:28] interesting pod/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Jul 28 04:46:27.048894 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:27.048689    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw" podUID="da666aad-410e-4a82-bdf9-aa79cd238520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Jul 28 04:46:28.048959 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:28.048922    2566 patch_prober.go:28] interesting pod/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Jul 28 04:46:28.049383 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:28.048976    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw" podUID="da666aad-410e-4a82-bdf9-aa79cd238520" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Jul 28 04:46:29.052204 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:29.052162    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw"
Jul 28 04:46:29.052670 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:29.052572    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw"
Jul 28 04:46:29.053328 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:29.053307    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw"
Jul 28 04:46:29.748595 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:29.748558    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ac8148f2-766fb89d6-jw79t"]
Jul 28 04:46:29.748909 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:29.748877    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ac8148f2-766fb89d6-jw79t" podUID="bcda0637-d837-4a0d-819a-0dcc264c3ddb" containerName="maas-api" containerID="cri-o://3de5a9821110b009e8f8958bd337db508cf916fcd8b731e2ba36070f50c494a5" gracePeriod=30
Jul 28 04:46:29.814803 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:29.814766    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ac8148f2-77879b4d68-qwpdl"]
Jul 28 04:46:29.815015 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:29.814994    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-ac8148f2-77879b4d68-qwpdl" podUID="6ab32ed5-3b6e-489d-b5a4-74f14daa836c" containerName="payload-pre-processing" containerID="cri-o://72816a32ee0e1cf949331a847f5a998bf3203998c4f8ac57c2aabed630732832" gracePeriod=30
Jul 28 04:46:30.108451 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:30.108423    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ac8148f2-766fb89d6-jw79t"
Jul 28 04:46:30.111931 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:30.111908    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ac8148f2-77879b4d68-qwpdl"
Jul 28 04:46:30.306324 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:30.306235    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4cf9\" (UniqueName: \"kubernetes.io/projected/6ab32ed5-3b6e-489d-b5a4-74f14daa836c-kube-api-access-h4cf9\") pod \"6ab32ed5-3b6e-489d-b5a4-74f14daa836c\" (UID: \"6ab32ed5-3b6e-489d-b5a4-74f14daa836c\") "
Jul 28 04:46:30.306324 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:30.306310    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-872wl\" (UniqueName: \"kubernetes.io/projected/bcda0637-d837-4a0d-819a-0dcc264c3ddb-kube-api-access-872wl\") pod \"bcda0637-d837-4a0d-819a-0dcc264c3ddb\" (UID: \"bcda0637-d837-4a0d-819a-0dcc264c3ddb\") "
Jul 28 04:46:30.306510 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:30.306369    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab32ed5-3b6e-489d-b5a4-74f14daa836c-plugins-config-volume\") pod \"6ab32ed5-3b6e-489d-b5a4-74f14daa836c\" (UID: \"6ab32ed5-3b6e-489d-b5a4-74f14daa836c\") "
Jul 28 04:46:30.306510 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:30.306416    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcda0637-d837-4a0d-819a-0dcc264c3ddb-maas-api-tls\") pod \"bcda0637-d837-4a0d-819a-0dcc264c3ddb\" (UID: \"bcda0637-d837-4a0d-819a-0dcc264c3ddb\") "
Jul 28 04:46:30.306510 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:30.306479    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcda0637-d837-4a0d-819a-0dcc264c3ddb-openshift-service-ca\") pod \"bcda0637-d837-4a0d-819a-0dcc264c3ddb\" (UID: \"bcda0637-d837-4a0d-819a-0dcc264c3ddb\") "
Jul 28 04:46:30.306886 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:30.306834    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ab32ed5-3b6e-489d-b5a4-74f14daa836c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6ab32ed5-3b6e-489d-b5a4-74f14daa836c" (UID: "6ab32ed5-3b6e-489d-b5a4-74f14daa836c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 04:46:30.306997 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:30.306966    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcda0637-d837-4a0d-819a-0dcc264c3ddb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bcda0637-d837-4a0d-819a-0dcc264c3ddb" (UID: "bcda0637-d837-4a0d-819a-0dcc264c3ddb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 04:46:30.308507 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:30.308481    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ab32ed5-3b6e-489d-b5a4-74f14daa836c-kube-api-access-h4cf9" (OuterVolumeSpecName: "kube-api-access-h4cf9") pod "6ab32ed5-3b6e-489d-b5a4-74f14daa836c" (UID: "6ab32ed5-3b6e-489d-b5a4-74f14daa836c"). InnerVolumeSpecName "kube-api-access-h4cf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:46:30.308507 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:30.308492    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcda0637-d837-4a0d-819a-0dcc264c3ddb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bcda0637-d837-4a0d-819a-0dcc264c3ddb" (UID: "bcda0637-d837-4a0d-819a-0dcc264c3ddb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 04:46:30.308646 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:30.308518    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcda0637-d837-4a0d-819a-0dcc264c3ddb-kube-api-access-872wl" (OuterVolumeSpecName: "kube-api-access-872wl") pod "bcda0637-d837-4a0d-819a-0dcc264c3ddb" (UID: "bcda0637-d837-4a0d-819a-0dcc264c3ddb"). InnerVolumeSpecName "kube-api-access-872wl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:46:30.408097 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:30.408048    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-872wl\" (UniqueName: \"kubernetes.io/projected/bcda0637-d837-4a0d-819a-0dcc264c3ddb-kube-api-access-872wl\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:46:30.408097 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:30.408082    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ab32ed5-3b6e-489d-b5a4-74f14daa836c-plugins-config-volume\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:46:30.408097 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:30.408094    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcda0637-d837-4a0d-819a-0dcc264c3ddb-maas-api-tls\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:46:30.408097 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:30.408103    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcda0637-d837-4a0d-819a-0dcc264c3ddb-openshift-service-ca\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:46:30.408097 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:30.408111    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4cf9\" (UniqueName: \"kubernetes.io/projected/6ab32ed5-3b6e-489d-b5a4-74f14daa836c-kube-api-access-h4cf9\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:46:30.613180 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:30.613073    2566 generic.go:356] "Generic (PLEG): container finished" podID="bcda0637-d837-4a0d-819a-0dcc264c3ddb" containerID="3de5a9821110b009e8f8958bd337db508cf916fcd8b731e2ba36070f50c494a5" exitCode=0
Jul 28 04:46:30.613180 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:30.613114    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ac8148f2-766fb89d6-jw79t"
Jul 28 04:46:30.613385 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:30.613165    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ac8148f2-766fb89d6-jw79t" event={"ID":"bcda0637-d837-4a0d-819a-0dcc264c3ddb","Type":"ContainerDied","Data":"3de5a9821110b009e8f8958bd337db508cf916fcd8b731e2ba36070f50c494a5"}
Jul 28 04:46:30.613385 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:30.613245    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ac8148f2-766fb89d6-jw79t" event={"ID":"bcda0637-d837-4a0d-819a-0dcc264c3ddb","Type":"ContainerDied","Data":"77256b48c100db245b2db3906a1516ee2d3f02f83a7ff8197ec39b5d98e2eed7"}
Jul 28 04:46:30.613385 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:30.613306    2566 scope.go:117] "RemoveContainer" containerID="3de5a9821110b009e8f8958bd337db508cf916fcd8b731e2ba36070f50c494a5"
Jul 28 04:46:30.614641 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:30.614613    2566 generic.go:356] "Generic (PLEG): container finished" podID="6ab32ed5-3b6e-489d-b5a4-74f14daa836c" containerID="72816a32ee0e1cf949331a847f5a998bf3203998c4f8ac57c2aabed630732832" exitCode=0
Jul 28 04:46:30.614768 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:30.614671    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ac8148f2-77879b4d68-qwpdl"
Jul 28 04:46:30.614768 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:30.614700    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ac8148f2-77879b4d68-qwpdl" event={"ID":"6ab32ed5-3b6e-489d-b5a4-74f14daa836c","Type":"ContainerDied","Data":"72816a32ee0e1cf949331a847f5a998bf3203998c4f8ac57c2aabed630732832"}
Jul 28 04:46:30.614768 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:30.614758    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ac8148f2-77879b4d68-qwpdl" event={"ID":"6ab32ed5-3b6e-489d-b5a4-74f14daa836c","Type":"ContainerDied","Data":"6b8ca6f7c18dbe9b4c2f75829ad04bc32c83e48aeda5e49c12017a8eadd29771"}
Jul 28 04:46:30.623187 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:30.623171    2566 scope.go:117] "RemoveContainer" containerID="3de5a9821110b009e8f8958bd337db508cf916fcd8b731e2ba36070f50c494a5"
Jul 28 04:46:30.623497 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:46:30.623474    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3de5a9821110b009e8f8958bd337db508cf916fcd8b731e2ba36070f50c494a5\": container with ID starting with 3de5a9821110b009e8f8958bd337db508cf916fcd8b731e2ba36070f50c494a5 not found: ID does not exist" containerID="3de5a9821110b009e8f8958bd337db508cf916fcd8b731e2ba36070f50c494a5"
Jul 28 04:46:30.623638 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:30.623503    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3de5a9821110b009e8f8958bd337db508cf916fcd8b731e2ba36070f50c494a5"} err="failed to get container status \"3de5a9821110b009e8f8958bd337db508cf916fcd8b731e2ba36070f50c494a5\": rpc error: code = NotFound desc = could not find container \"3de5a9821110b009e8f8958bd337db508cf916fcd8b731e2ba36070f50c494a5\": container with ID starting with 3de5a9821110b009e8f8958bd337db508cf916fcd8b731e2ba36070f50c494a5 not found: ID does not exist"
Jul 28 04:46:30.623638 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:30.623520    2566 scope.go:117] "RemoveContainer" containerID="72816a32ee0e1cf949331a847f5a998bf3203998c4f8ac57c2aabed630732832"
Jul 28 04:46:30.633683 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:30.633640    2566 scope.go:117] "RemoveContainer" containerID="72816a32ee0e1cf949331a847f5a998bf3203998c4f8ac57c2aabed630732832"
Jul 28 04:46:30.634182 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:46:30.634162    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72816a32ee0e1cf949331a847f5a998bf3203998c4f8ac57c2aabed630732832\": container with ID starting with 72816a32ee0e1cf949331a847f5a998bf3203998c4f8ac57c2aabed630732832 not found: ID does not exist" containerID="72816a32ee0e1cf949331a847f5a998bf3203998c4f8ac57c2aabed630732832"
Jul 28 04:46:30.634244 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:30.634189    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72816a32ee0e1cf949331a847f5a998bf3203998c4f8ac57c2aabed630732832"} err="failed to get container status \"72816a32ee0e1cf949331a847f5a998bf3203998c4f8ac57c2aabed630732832\": rpc error: code = NotFound desc = could not find container \"72816a32ee0e1cf949331a847f5a998bf3203998c4f8ac57c2aabed630732832\": container with ID starting with 72816a32ee0e1cf949331a847f5a998bf3203998c4f8ac57c2aabed630732832 not found: ID does not exist"
Jul 28 04:46:30.638802 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:30.638780    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ac8148f2-77879b4d68-qwpdl"]
Jul 28 04:46:30.642372 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:30.642323    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ac8148f2-77879b4d68-qwpdl"]
Jul 28 04:46:30.651746 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:30.651724    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ac8148f2-766fb89d6-jw79t"]
Jul 28 04:46:30.656756 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:30.656734    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ac8148f2-766fb89d6-jw79t"]
Jul 28 04:46:32.089111 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:32.089077    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ab32ed5-3b6e-489d-b5a4-74f14daa836c" path="/var/lib/kubelet/pods/6ab32ed5-3b6e-489d-b5a4-74f14daa836c/volumes"
Jul 28 04:46:32.089495 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:32.089483    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcda0637-d837-4a0d-819a-0dcc264c3ddb" path="/var/lib/kubelet/pods/bcda0637-d837-4a0d-819a-0dcc264c3ddb/volumes"
Jul 28 04:46:34.782171 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:34.782135    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw"]
Jul 28 04:46:34.782612 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:34.782357    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw" podUID="da666aad-410e-4a82-bdf9-aa79cd238520" containerName="istio-proxy" containerID="cri-o://f57ecba6e7d3cba305f00c01acd54c32d7d9ba7605b5b133a5173d03a86911eb" gracePeriod=30
Jul 28 04:46:40.038442 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:40.038417    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw"
Jul 28 04:46:40.107031 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:40.106948    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da666aad-410e-4a82-bdf9-aa79cd238520-workload-socket\") pod \"da666aad-410e-4a82-bdf9-aa79cd238520\" (UID: \"da666aad-410e-4a82-bdf9-aa79cd238520\") "
Jul 28 04:46:40.107031 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:40.107031    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da666aad-410e-4a82-bdf9-aa79cd238520-istio-token\") pod \"da666aad-410e-4a82-bdf9-aa79cd238520\" (UID: \"da666aad-410e-4a82-bdf9-aa79cd238520\") "
Jul 28 04:46:40.107236 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:40.107071    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjgrm\" (UniqueName: \"kubernetes.io/projected/da666aad-410e-4a82-bdf9-aa79cd238520-kube-api-access-gjgrm\") pod \"da666aad-410e-4a82-bdf9-aa79cd238520\" (UID: \"da666aad-410e-4a82-bdf9-aa79cd238520\") "
Jul 28 04:46:40.107236 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:40.107098    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da666aad-410e-4a82-bdf9-aa79cd238520-credential-socket\") pod \"da666aad-410e-4a82-bdf9-aa79cd238520\" (UID: \"da666aad-410e-4a82-bdf9-aa79cd238520\") "
Jul 28 04:46:40.107236 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:40.107123    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da666aad-410e-4a82-bdf9-aa79cd238520-istio-podinfo\") pod \"da666aad-410e-4a82-bdf9-aa79cd238520\" (UID: \"da666aad-410e-4a82-bdf9-aa79cd238520\") "
Jul 28 04:46:40.107236 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:40.107152    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da666aad-410e-4a82-bdf9-aa79cd238520-istio-envoy\") pod \"da666aad-410e-4a82-bdf9-aa79cd238520\" (UID: \"da666aad-410e-4a82-bdf9-aa79cd238520\") "
Jul 28 04:46:40.107236 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:40.107189    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da666aad-410e-4a82-bdf9-aa79cd238520-istiod-ca-cert\") pod \"da666aad-410e-4a82-bdf9-aa79cd238520\" (UID: \"da666aad-410e-4a82-bdf9-aa79cd238520\") "
Jul 28 04:46:40.107236 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:40.107203    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da666aad-410e-4a82-bdf9-aa79cd238520-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "da666aad-410e-4a82-bdf9-aa79cd238520" (UID: "da666aad-410e-4a82-bdf9-aa79cd238520"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:46:40.107236 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:40.107216    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da666aad-410e-4a82-bdf9-aa79cd238520-istio-data\") pod \"da666aad-410e-4a82-bdf9-aa79cd238520\" (UID: \"da666aad-410e-4a82-bdf9-aa79cd238520\") "
Jul 28 04:46:40.107581 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:40.107288    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da666aad-410e-4a82-bdf9-aa79cd238520-workload-certs\") pod \"da666aad-410e-4a82-bdf9-aa79cd238520\" (UID: \"da666aad-410e-4a82-bdf9-aa79cd238520\") "
Jul 28 04:46:40.107581 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:40.107511    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da666aad-410e-4a82-bdf9-aa79cd238520-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "da666aad-410e-4a82-bdf9-aa79cd238520" (UID: "da666aad-410e-4a82-bdf9-aa79cd238520"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:46:40.107733 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:40.107695    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da666aad-410e-4a82-bdf9-aa79cd238520-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "da666aad-410e-4a82-bdf9-aa79cd238520" (UID: "da666aad-410e-4a82-bdf9-aa79cd238520"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:46:40.107796 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:40.107722    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da666aad-410e-4a82-bdf9-aa79cd238520-credential-socket\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:46:40.107796 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:40.107758    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da666aad-410e-4a82-bdf9-aa79cd238520-workload-socket\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:46:40.107908 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:40.107833    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da666aad-410e-4a82-bdf9-aa79cd238520-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "da666aad-410e-4a82-bdf9-aa79cd238520" (UID: "da666aad-410e-4a82-bdf9-aa79cd238520"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 04:46:40.108142 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:40.108118    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da666aad-410e-4a82-bdf9-aa79cd238520-istio-data" (OuterVolumeSpecName: "istio-data") pod "da666aad-410e-4a82-bdf9-aa79cd238520" (UID: "da666aad-410e-4a82-bdf9-aa79cd238520"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:46:40.109551 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:40.109526    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da666aad-410e-4a82-bdf9-aa79cd238520-kube-api-access-gjgrm" (OuterVolumeSpecName: "kube-api-access-gjgrm") pod "da666aad-410e-4a82-bdf9-aa79cd238520" (UID: "da666aad-410e-4a82-bdf9-aa79cd238520"). InnerVolumeSpecName "kube-api-access-gjgrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:46:40.109704 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:40.109676    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da666aad-410e-4a82-bdf9-aa79cd238520-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "da666aad-410e-4a82-bdf9-aa79cd238520" (UID: "da666aad-410e-4a82-bdf9-aa79cd238520"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:46:40.109773 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:40.109709    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da666aad-410e-4a82-bdf9-aa79cd238520-istio-token" (OuterVolumeSpecName: "istio-token") pod "da666aad-410e-4a82-bdf9-aa79cd238520" (UID: "da666aad-410e-4a82-bdf9-aa79cd238520"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:46:40.109830 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:40.109765    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da666aad-410e-4a82-bdf9-aa79cd238520-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "da666aad-410e-4a82-bdf9-aa79cd238520" (UID: "da666aad-410e-4a82-bdf9-aa79cd238520"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 04:46:40.208809 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:40.208777    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da666aad-410e-4a82-bdf9-aa79cd238520-workload-certs\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:46:40.208809 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:40.208807    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da666aad-410e-4a82-bdf9-aa79cd238520-istio-token\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:46:40.208975 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:40.208820    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjgrm\" (UniqueName: \"kubernetes.io/projected/da666aad-410e-4a82-bdf9-aa79cd238520-kube-api-access-gjgrm\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:46:40.208975 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:40.208833    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da666aad-410e-4a82-bdf9-aa79cd238520-istio-podinfo\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:46:40.208975 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:40.208862    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da666aad-410e-4a82-bdf9-aa79cd238520-istio-envoy\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:46:40.208975 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:40.208873    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da666aad-410e-4a82-bdf9-aa79cd238520-istiod-ca-cert\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:46:40.208975 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:40.208885    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da666aad-410e-4a82-bdf9-aa79cd238520-istio-data\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:46:40.667030 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:40.666993    2566 generic.go:356] "Generic (PLEG): container finished" podID="da666aad-410e-4a82-bdf9-aa79cd238520" containerID="f57ecba6e7d3cba305f00c01acd54c32d7d9ba7605b5b133a5173d03a86911eb" exitCode=0
Jul 28 04:46:40.667200 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:40.667042    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw" event={"ID":"da666aad-410e-4a82-bdf9-aa79cd238520","Type":"ContainerDied","Data":"f57ecba6e7d3cba305f00c01acd54c32d7d9ba7605b5b133a5173d03a86911eb"}
Jul 28 04:46:40.667200 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:40.667048    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw"
Jul 28 04:46:40.667200 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:40.667095    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw" event={"ID":"da666aad-410e-4a82-bdf9-aa79cd238520","Type":"ContainerDied","Data":"392f19a5583d98aeed927faf678696e9b973e9eac5a1bff552d5a95ee48f9027"}
Jul 28 04:46:40.667200 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:40.667127    2566 scope.go:117] "RemoveContainer" containerID="f57ecba6e7d3cba305f00c01acd54c32d7d9ba7605b5b133a5173d03a86911eb"
Jul 28 04:46:40.676330 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:40.676311    2566 scope.go:117] "RemoveContainer" containerID="73f08a25aa75aeaceece5c96c11b859461049bf85013b08b8a8edfd29bb073b3"
Jul 28 04:46:40.683751 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:40.683733    2566 scope.go:117] "RemoveContainer" containerID="f57ecba6e7d3cba305f00c01acd54c32d7d9ba7605b5b133a5173d03a86911eb"
Jul 28 04:46:40.684027 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:46:40.684008    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f57ecba6e7d3cba305f00c01acd54c32d7d9ba7605b5b133a5173d03a86911eb\": container with ID starting with f57ecba6e7d3cba305f00c01acd54c32d7d9ba7605b5b133a5173d03a86911eb not found: ID does not exist" containerID="f57ecba6e7d3cba305f00c01acd54c32d7d9ba7605b5b133a5173d03a86911eb"
Jul 28 04:46:40.684098 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:40.684035    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f57ecba6e7d3cba305f00c01acd54c32d7d9ba7605b5b133a5173d03a86911eb"} err="failed to get container status \"f57ecba6e7d3cba305f00c01acd54c32d7d9ba7605b5b133a5173d03a86911eb\": rpc error: code = NotFound desc = could not find container \"f57ecba6e7d3cba305f00c01acd54c32d7d9ba7605b5b133a5173d03a86911eb\": container with ID starting with f57ecba6e7d3cba305f00c01acd54c32d7d9ba7605b5b133a5173d03a86911eb not found: ID does not exist"
Jul 28 04:46:40.684098 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:40.684052    2566 scope.go:117] "RemoveContainer" containerID="73f08a25aa75aeaceece5c96c11b859461049bf85013b08b8a8edfd29bb073b3"
Jul 28 04:46:40.684306 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:46:40.684285    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73f08a25aa75aeaceece5c96c11b859461049bf85013b08b8a8edfd29bb073b3\": container with ID starting with 73f08a25aa75aeaceece5c96c11b859461049bf85013b08b8a8edfd29bb073b3 not found: ID does not exist" containerID="73f08a25aa75aeaceece5c96c11b859461049bf85013b08b8a8edfd29bb073b3"
Jul 28 04:46:40.684352 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:40.684312    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73f08a25aa75aeaceece5c96c11b859461049bf85013b08b8a8edfd29bb073b3"} err="failed to get container status \"73f08a25aa75aeaceece5c96c11b859461049bf85013b08b8a8edfd29bb073b3\": rpc error: code = NotFound desc = could not find container \"73f08a25aa75aeaceece5c96c11b859461049bf85013b08b8a8edfd29bb073b3\": container with ID starting with 73f08a25aa75aeaceece5c96c11b859461049bf85013b08b8a8edfd29bb073b3 not found: ID does not exist"
Jul 28 04:46:40.688933 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:40.688912    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw"]
Jul 28 04:46:40.694523 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:40.694502    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-ac8148f2-openshift-default-6d4cbf9bdd-2kktw"]
Jul 28 04:46:42.088711 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:42.088678    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da666aad-410e-4a82-bdf9-aa79cd238520" path="/var/lib/kubelet/pods/da666aad-410e-4a82-bdf9-aa79cd238520/volumes"
Jul 28 04:46:43.695693 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:43.695661    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4glpr_bfd2d3d8-f287-42d8-83c7-f82185c96735/console-operator/2.log"
Jul 28 04:46:43.701448 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:46:43.701424    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4glpr_bfd2d3d8-f287-42d8-83c7-f82185c96735/console-operator/2.log"
Jul 28 04:48:25.100450 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.100418    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9"]
Jul 28 04:48:25.100942 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.100771    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a62608d-9359-43c9-9cfc-b13b26bb00b8" containerName="istio-proxy"
Jul 28 04:48:25.100942 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.100782    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a62608d-9359-43c9-9cfc-b13b26bb00b8" containerName="istio-proxy"
Jul 28 04:48:25.100942 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.100794    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a62608d-9359-43c9-9cfc-b13b26bb00b8" containerName="istio-proxy"
Jul 28 04:48:25.100942 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.100801    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a62608d-9359-43c9-9cfc-b13b26bb00b8" containerName="istio-proxy"
Jul 28 04:48:25.100942 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.100809    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ab32ed5-3b6e-489d-b5a4-74f14daa836c" containerName="payload-pre-processing"
Jul 28 04:48:25.100942 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.100814    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ab32ed5-3b6e-489d-b5a4-74f14daa836c" containerName="payload-pre-processing"
Jul 28 04:48:25.100942 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.100827    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da666aad-410e-4a82-bdf9-aa79cd238520" containerName="istio-proxy"
Jul 28 04:48:25.100942 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.100832    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="da666aad-410e-4a82-bdf9-aa79cd238520" containerName="istio-proxy"
Jul 28 04:48:25.100942 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.100865    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcda0637-d837-4a0d-819a-0dcc264c3ddb" containerName="maas-api"
Jul 28 04:48:25.100942 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.100870    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcda0637-d837-4a0d-819a-0dcc264c3ddb" containerName="maas-api"
Jul 28 04:48:25.100942 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.100876    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da666aad-410e-4a82-bdf9-aa79cd238520" containerName="istio-proxy"
Jul 28 04:48:25.100942 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.100881    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="da666aad-410e-4a82-bdf9-aa79cd238520" containerName="istio-proxy"
Jul 28 04:48:25.100942 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.100928    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ab32ed5-3b6e-489d-b5a4-74f14daa836c" containerName="payload-pre-processing"
Jul 28 04:48:25.100942 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.100936    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="da666aad-410e-4a82-bdf9-aa79cd238520" containerName="istio-proxy"
Jul 28 04:48:25.100942 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.100943    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="da666aad-410e-4a82-bdf9-aa79cd238520" containerName="istio-proxy"
Jul 28 04:48:25.101414 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.100952    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a62608d-9359-43c9-9cfc-b13b26bb00b8" containerName="istio-proxy"
Jul 28 04:48:25.101414 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.100957    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcda0637-d837-4a0d-819a-0dcc264c3ddb" containerName="maas-api"
Jul 28 04:48:25.101414 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.100963    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a62608d-9359-43c9-9cfc-b13b26bb00b8" containerName="istio-proxy"
Jul 28 04:48:25.109094 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.109060    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9"
Jul 28 04:48:25.111971 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.111947    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 28 04:48:25.112342 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.112319    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-ae9f1f22-openshift-default-dockercfg-xb9vn\""
Jul 28 04:48:25.132896 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.132871    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9"]
Jul 28 04:48:25.282514 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.282476    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpxzn\" (UniqueName: \"kubernetes.io/projected/0c3cbb68-4002-4255-84e4-b83fca6162e8-kube-api-access-hpxzn\") pod \"e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9\" (UID: \"0c3cbb68-4002-4255-84e4-b83fca6162e8\") " pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9"
Jul 28 04:48:25.282711 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.282532    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c3cbb68-4002-4255-84e4-b83fca6162e8-workload-certs\") pod \"e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9\" (UID: \"0c3cbb68-4002-4255-84e4-b83fca6162e8\") " pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9"
Jul 28 04:48:25.282711 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.282584    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c3cbb68-4002-4255-84e4-b83fca6162e8-istio-data\") pod \"e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9\" (UID: \"0c3cbb68-4002-4255-84e4-b83fca6162e8\") " pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9"
Jul 28 04:48:25.282711 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.282678    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c3cbb68-4002-4255-84e4-b83fca6162e8-istio-podinfo\") pod \"e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9\" (UID: \"0c3cbb68-4002-4255-84e4-b83fca6162e8\") " pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9"
Jul 28 04:48:25.282933 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.282764    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c3cbb68-4002-4255-84e4-b83fca6162e8-credential-socket\") pod \"e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9\" (UID: \"0c3cbb68-4002-4255-84e4-b83fca6162e8\") " pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9"
Jul 28 04:48:25.282933 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.282796    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c3cbb68-4002-4255-84e4-b83fca6162e8-workload-socket\") pod \"e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9\" (UID: \"0c3cbb68-4002-4255-84e4-b83fca6162e8\") " pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9"
Jul 28 04:48:25.282933 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.282823    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c3cbb68-4002-4255-84e4-b83fca6162e8-istio-envoy\") pod \"e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9\" (UID: \"0c3cbb68-4002-4255-84e4-b83fca6162e8\") " pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9"
Jul 28 04:48:25.282933 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.282925    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c3cbb68-4002-4255-84e4-b83fca6162e8-istiod-ca-cert\") pod \"e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9\" (UID: \"0c3cbb68-4002-4255-84e4-b83fca6162e8\") " pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9"
Jul 28 04:48:25.283115 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.282984    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c3cbb68-4002-4255-84e4-b83fca6162e8-istio-token\") pod \"e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9\" (UID: \"0c3cbb68-4002-4255-84e4-b83fca6162e8\") " pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9"
Jul 28 04:48:25.384093 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.383981    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c3cbb68-4002-4255-84e4-b83fca6162e8-credential-socket\") pod \"e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9\" (UID: \"0c3cbb68-4002-4255-84e4-b83fca6162e8\") " pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9"
Jul 28 04:48:25.384093 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.384033    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c3cbb68-4002-4255-84e4-b83fca6162e8-workload-socket\") pod \"e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9\" (UID: \"0c3cbb68-4002-4255-84e4-b83fca6162e8\") " pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9"
Jul 28 04:48:25.384340 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.384217    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c3cbb68-4002-4255-84e4-b83fca6162e8-istio-envoy\") pod \"e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9\" (UID: \"0c3cbb68-4002-4255-84e4-b83fca6162e8\") " pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9"
Jul 28 04:48:25.384340 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.384278    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c3cbb68-4002-4255-84e4-b83fca6162e8-istiod-ca-cert\") pod \"e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9\" (UID: \"0c3cbb68-4002-4255-84e4-b83fca6162e8\") " pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9"
Jul 28 04:48:25.384340 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.384337    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c3cbb68-4002-4255-84e4-b83fca6162e8-istio-token\") pod \"e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9\" (UID: \"0c3cbb68-4002-4255-84e4-b83fca6162e8\") " pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9"
Jul 28 04:48:25.384508 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.384373    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpxzn\" (UniqueName: \"kubernetes.io/projected/0c3cbb68-4002-4255-84e4-b83fca6162e8-kube-api-access-hpxzn\") pod \"e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9\" (UID: \"0c3cbb68-4002-4255-84e4-b83fca6162e8\") " pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9"
Jul 28 04:48:25.384508 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.384419    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c3cbb68-4002-4255-84e4-b83fca6162e8-workload-certs\") pod \"e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9\" (UID: \"0c3cbb68-4002-4255-84e4-b83fca6162e8\") " pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9"
Jul 28 04:48:25.384508 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.384449    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c3cbb68-4002-4255-84e4-b83fca6162e8-credential-socket\") pod \"e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9\" (UID: \"0c3cbb68-4002-4255-84e4-b83fca6162e8\") " pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9"
Jul 28 04:48:25.384508 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.384464    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c3cbb68-4002-4255-84e4-b83fca6162e8-istio-data\") pod \"e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9\" (UID: \"0c3cbb68-4002-4255-84e4-b83fca6162e8\") " pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9"
Jul 28 04:48:25.384508 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.384478    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c3cbb68-4002-4255-84e4-b83fca6162e8-workload-socket\") pod \"e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9\" (UID: \"0c3cbb68-4002-4255-84e4-b83fca6162e8\") " pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9"
Jul 28 04:48:25.384774 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.384723    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c3cbb68-4002-4255-84e4-b83fca6162e8-workload-certs\") pod \"e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9\" (UID: \"0c3cbb68-4002-4255-84e4-b83fca6162e8\") " pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9"
Jul 28 04:48:25.384774 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.384755    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c3cbb68-4002-4255-84e4-b83fca6162e8-istio-podinfo\") pod \"e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9\" (UID: \"0c3cbb68-4002-4255-84e4-b83fca6162e8\") " pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9"
Jul 28 04:48:25.385150 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.385126    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c3cbb68-4002-4255-84e4-b83fca6162e8-istiod-ca-cert\") pod \"e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9\" (UID: \"0c3cbb68-4002-4255-84e4-b83fca6162e8\") " pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9"
Jul 28 04:48:25.385241 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.385146    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c3cbb68-4002-4255-84e4-b83fca6162e8-istio-data\") pod \"e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9\" (UID: \"0c3cbb68-4002-4255-84e4-b83fca6162e8\") " pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9"
Jul 28 04:48:25.387102 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.387078    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c3cbb68-4002-4255-84e4-b83fca6162e8-istio-envoy\") pod \"e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9\" (UID: \"0c3cbb68-4002-4255-84e4-b83fca6162e8\") " pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9"
Jul 28 04:48:25.387285 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.387266    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c3cbb68-4002-4255-84e4-b83fca6162e8-istio-podinfo\") pod \"e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9\" (UID: \"0c3cbb68-4002-4255-84e4-b83fca6162e8\") " pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9"
Jul 28 04:48:25.392155 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.392132    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c3cbb68-4002-4255-84e4-b83fca6162e8-istio-token\") pod \"e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9\" (UID: \"0c3cbb68-4002-4255-84e4-b83fca6162e8\") " pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9"
Jul 28 04:48:25.392282 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.392264    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpxzn\" (UniqueName: \"kubernetes.io/projected/0c3cbb68-4002-4255-84e4-b83fca6162e8-kube-api-access-hpxzn\") pod \"e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9\" (UID: \"0c3cbb68-4002-4255-84e4-b83fca6162e8\") " pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9"
Jul 28 04:48:25.420918 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.420885    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9"
Jul 28 04:48:25.560488 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.560465    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9"]
Jul 28 04:48:25.562364 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:48:25.562333    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c3cbb68_4002_4255_84e4_b83fca6162e8.slice/crio-a9983e3f55521935613311a590e504a9e8765602366dbe6fc1fb48b55aea91f2 WatchSource:0}: Error finding container a9983e3f55521935613311a590e504a9e8765602366dbe6fc1fb48b55aea91f2: Status 404 returned error can't find the container with id a9983e3f55521935613311a590e504a9e8765602366dbe6fc1fb48b55aea91f2
Jul 28 04:48:25.564578 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.564545    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:48:25.564703 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.564606    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:48:25.564703 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:25.564638    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:48:26.167246 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:26.167214    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9" event={"ID":"0c3cbb68-4002-4255-84e4-b83fca6162e8","Type":"ContainerStarted","Data":"f9768be2633b9b8841a6192be837d32a5736b66a6142d4a1f15349f9bc670ee9"}
Jul 28 04:48:26.167654 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:26.167267    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9" event={"ID":"0c3cbb68-4002-4255-84e4-b83fca6162e8","Type":"ContainerStarted","Data":"a9983e3f55521935613311a590e504a9e8765602366dbe6fc1fb48b55aea91f2"}
Jul 28 04:48:26.187269 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:26.187216    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9" podStartSLOduration=1.1872013909999999 podStartE2EDuration="1.187201391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:48:26.18571353 +0000 UTC m=+2206.575889552" watchObservedRunningTime="2026-07-28 04:48:26.187201391 +0000 UTC m=+2206.577377412"
Jul 28 04:48:26.421637 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:26.421540    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9"
Jul 28 04:48:26.427422 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:26.427392    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9"
Jul 28 04:48:27.172602 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:27.172571    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9"
Jul 28 04:48:27.173574 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:27.173556    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9"
Jul 28 04:48:28.369331 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:28.369302    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ae9f1f22-864f4688b9-dpmr2"]
Jul 28 04:48:28.373288 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:28.373268    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae9f1f22-864f4688b9-dpmr2"
Jul 28 04:48:28.376335 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:28.376309    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-ae9f1f22-serving-cert\""
Jul 28 04:48:28.376335 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:28.376325    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 04:48:28.376524 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:28.376356    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 04:48:28.377237 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:28.377216    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gfn77\""
Jul 28 04:48:28.385396 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:28.385376    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ae9f1f22-864f4688b9-dpmr2"]
Jul 28 04:48:28.411550 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:28.411520    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnvqh\" (UniqueName: \"kubernetes.io/projected/1051c09d-0b3e-4fdd-85cf-94b6c3e7f646-kube-api-access-fnvqh\") pod \"maas-api-e2e-mt-ae9f1f22-864f4688b9-dpmr2\" (UID: \"1051c09d-0b3e-4fdd-85cf-94b6c3e7f646\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae9f1f22-864f4688b9-dpmr2"
Jul 28 04:48:28.411713 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:28.411575    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1051c09d-0b3e-4fdd-85cf-94b6c3e7f646-maas-api-tls\") pod \"maas-api-e2e-mt-ae9f1f22-864f4688b9-dpmr2\" (UID: \"1051c09d-0b3e-4fdd-85cf-94b6c3e7f646\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae9f1f22-864f4688b9-dpmr2"
Jul 28 04:48:28.411777 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:28.411703    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1051c09d-0b3e-4fdd-85cf-94b6c3e7f646-openshift-service-ca\") pod \"maas-api-e2e-mt-ae9f1f22-864f4688b9-dpmr2\" (UID: \"1051c09d-0b3e-4fdd-85cf-94b6c3e7f646\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae9f1f22-864f4688b9-dpmr2"
Jul 28 04:48:28.513127 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:28.513097    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnvqh\" (UniqueName: \"kubernetes.io/projected/1051c09d-0b3e-4fdd-85cf-94b6c3e7f646-kube-api-access-fnvqh\") pod \"maas-api-e2e-mt-ae9f1f22-864f4688b9-dpmr2\" (UID: \"1051c09d-0b3e-4fdd-85cf-94b6c3e7f646\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae9f1f22-864f4688b9-dpmr2"
Jul 28 04:48:28.513287 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:28.513159    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1051c09d-0b3e-4fdd-85cf-94b6c3e7f646-maas-api-tls\") pod \"maas-api-e2e-mt-ae9f1f22-864f4688b9-dpmr2\" (UID: \"1051c09d-0b3e-4fdd-85cf-94b6c3e7f646\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae9f1f22-864f4688b9-dpmr2"
Jul 28 04:48:28.513287 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:28.513246    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1051c09d-0b3e-4fdd-85cf-94b6c3e7f646-openshift-service-ca\") pod \"maas-api-e2e-mt-ae9f1f22-864f4688b9-dpmr2\" (UID: \"1051c09d-0b3e-4fdd-85cf-94b6c3e7f646\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae9f1f22-864f4688b9-dpmr2"
Jul 28 04:48:28.513287 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:48:28.513257    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-ae9f1f22-serving-cert: secret "maas-api-e2e-mt-ae9f1f22-serving-cert" not found
Jul 28 04:48:28.513390 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:48:28.513319    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1051c09d-0b3e-4fdd-85cf-94b6c3e7f646-maas-api-tls podName:1051c09d-0b3e-4fdd-85cf-94b6c3e7f646 nodeName:}" failed. No retries permitted until 2026-07-28 04:48:29.013298853 +0000 UTC m=+2209.403474858 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1051c09d-0b3e-4fdd-85cf-94b6c3e7f646-maas-api-tls") pod "maas-api-e2e-mt-ae9f1f22-864f4688b9-dpmr2" (UID: "1051c09d-0b3e-4fdd-85cf-94b6c3e7f646") : secret "maas-api-e2e-mt-ae9f1f22-serving-cert" not found
Jul 28 04:48:28.513874 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:28.513854    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1051c09d-0b3e-4fdd-85cf-94b6c3e7f646-openshift-service-ca\") pod \"maas-api-e2e-mt-ae9f1f22-864f4688b9-dpmr2\" (UID: \"1051c09d-0b3e-4fdd-85cf-94b6c3e7f646\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae9f1f22-864f4688b9-dpmr2"
Jul 28 04:48:28.521885 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:28.521865    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnvqh\" (UniqueName: \"kubernetes.io/projected/1051c09d-0b3e-4fdd-85cf-94b6c3e7f646-kube-api-access-fnvqh\") pod \"maas-api-e2e-mt-ae9f1f22-864f4688b9-dpmr2\" (UID: \"1051c09d-0b3e-4fdd-85cf-94b6c3e7f646\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae9f1f22-864f4688b9-dpmr2"
Jul 28 04:48:28.996903 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:28.996864    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ae9f1f22-fc7fc5cbd-xcrck"]
Jul 28 04:48:29.002863 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:29.002196    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae9f1f22-fc7fc5cbd-xcrck"
Jul 28 04:48:29.006006 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:29.005981    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ae9f1f22\""
Jul 28 04:48:29.006231 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:29.006136    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-ae9f1f22-dockercfg-trwj2\""
Jul 28 04:48:29.012638 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:29.012614    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ae9f1f22-fc7fc5cbd-xcrck"]
Jul 28 04:48:29.018593 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:29.018564    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lspxw\" (UniqueName: \"kubernetes.io/projected/9ba02699-ef32-49a9-ac5e-c96c2049b024-kube-api-access-lspxw\") pod \"payload-pre-processing-e2e-mt-ae9f1f22-fc7fc5cbd-xcrck\" (UID: \"9ba02699-ef32-49a9-ac5e-c96c2049b024\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ae9f1f22-fc7fc5cbd-xcrck"
Jul 28 04:48:29.018691 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:29.018623    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1051c09d-0b3e-4fdd-85cf-94b6c3e7f646-maas-api-tls\") pod \"maas-api-e2e-mt-ae9f1f22-864f4688b9-dpmr2\" (UID: \"1051c09d-0b3e-4fdd-85cf-94b6c3e7f646\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae9f1f22-864f4688b9-dpmr2"
Jul 28 04:48:29.018691 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:29.018645    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ba02699-ef32-49a9-ac5e-c96c2049b024-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ae9f1f22-fc7fc5cbd-xcrck\" (UID: \"9ba02699-ef32-49a9-ac5e-c96c2049b024\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ae9f1f22-fc7fc5cbd-xcrck"
Jul 28 04:48:29.021028 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:29.021006    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1051c09d-0b3e-4fdd-85cf-94b6c3e7f646-maas-api-tls\") pod \"maas-api-e2e-mt-ae9f1f22-864f4688b9-dpmr2\" (UID: \"1051c09d-0b3e-4fdd-85cf-94b6c3e7f646\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae9f1f22-864f4688b9-dpmr2"
Jul 28 04:48:29.022257 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:29.022236    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ae9f1f22-b68f48b94-x68n8"]
Jul 28 04:48:29.025714 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:29.025698    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ae9f1f22-b68f48b94-x68n8"
Jul 28 04:48:29.035572 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:29.035542    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ae9f1f22-b68f48b94-x68n8"]
Jul 28 04:48:29.119643 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:29.119606    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52fb7\" (UniqueName: \"kubernetes.io/projected/19c645e3-83f9-4fa4-96b3-73277c1e78c5-kube-api-access-52fb7\") pod \"payload-processing-e2e-mt-ae9f1f22-b68f48b94-x68n8\" (UID: \"19c645e3-83f9-4fa4-96b3-73277c1e78c5\") " pod="openshift-ingress/payload-processing-e2e-mt-ae9f1f22-b68f48b94-x68n8"
Jul 28 04:48:29.119643 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:29.119644    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19c645e3-83f9-4fa4-96b3-73277c1e78c5-plugins-config-volume\") pod \"payload-processing-e2e-mt-ae9f1f22-b68f48b94-x68n8\" (UID: \"19c645e3-83f9-4fa4-96b3-73277c1e78c5\") " pod="openshift-ingress/payload-processing-e2e-mt-ae9f1f22-b68f48b94-x68n8"
Jul 28 04:48:29.119869 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:29.119736    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ba02699-ef32-49a9-ac5e-c96c2049b024-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ae9f1f22-fc7fc5cbd-xcrck\" (UID: \"9ba02699-ef32-49a9-ac5e-c96c2049b024\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ae9f1f22-fc7fc5cbd-xcrck"
Jul 28 04:48:29.119869 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:29.119862    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lspxw\" (UniqueName: \"kubernetes.io/projected/9ba02699-ef32-49a9-ac5e-c96c2049b024-kube-api-access-lspxw\") pod \"payload-pre-processing-e2e-mt-ae9f1f22-fc7fc5cbd-xcrck\" (UID: \"9ba02699-ef32-49a9-ac5e-c96c2049b024\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ae9f1f22-fc7fc5cbd-xcrck"
Jul 28 04:48:29.120447 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:29.120421    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ba02699-ef32-49a9-ac5e-c96c2049b024-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ae9f1f22-fc7fc5cbd-xcrck\" (UID: \"9ba02699-ef32-49a9-ac5e-c96c2049b024\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ae9f1f22-fc7fc5cbd-xcrck"
Jul 28 04:48:29.130956 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:29.130929    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lspxw\" (UniqueName: \"kubernetes.io/projected/9ba02699-ef32-49a9-ac5e-c96c2049b024-kube-api-access-lspxw\") pod \"payload-pre-processing-e2e-mt-ae9f1f22-fc7fc5cbd-xcrck\" (UID: \"9ba02699-ef32-49a9-ac5e-c96c2049b024\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ae9f1f22-fc7fc5cbd-xcrck"
Jul 28 04:48:29.221146 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:29.221097    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52fb7\" (UniqueName: \"kubernetes.io/projected/19c645e3-83f9-4fa4-96b3-73277c1e78c5-kube-api-access-52fb7\") pod \"payload-processing-e2e-mt-ae9f1f22-b68f48b94-x68n8\" (UID: \"19c645e3-83f9-4fa4-96b3-73277c1e78c5\") " pod="openshift-ingress/payload-processing-e2e-mt-ae9f1f22-b68f48b94-x68n8"
Jul 28 04:48:29.221311 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:29.221163    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19c645e3-83f9-4fa4-96b3-73277c1e78c5-plugins-config-volume\") pod \"payload-processing-e2e-mt-ae9f1f22-b68f48b94-x68n8\" (UID: \"19c645e3-83f9-4fa4-96b3-73277c1e78c5\") " pod="openshift-ingress/payload-processing-e2e-mt-ae9f1f22-b68f48b94-x68n8"
Jul 28 04:48:29.221925 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:29.221906    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19c645e3-83f9-4fa4-96b3-73277c1e78c5-plugins-config-volume\") pod \"payload-processing-e2e-mt-ae9f1f22-b68f48b94-x68n8\" (UID: \"19c645e3-83f9-4fa4-96b3-73277c1e78c5\") " pod="openshift-ingress/payload-processing-e2e-mt-ae9f1f22-b68f48b94-x68n8"
Jul 28 04:48:29.228902 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:29.228874    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52fb7\" (UniqueName: \"kubernetes.io/projected/19c645e3-83f9-4fa4-96b3-73277c1e78c5-kube-api-access-52fb7\") pod \"payload-processing-e2e-mt-ae9f1f22-b68f48b94-x68n8\" (UID: \"19c645e3-83f9-4fa4-96b3-73277c1e78c5\") " pod="openshift-ingress/payload-processing-e2e-mt-ae9f1f22-b68f48b94-x68n8"
Jul 28 04:48:29.286383 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:29.286363    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae9f1f22-864f4688b9-dpmr2"
Jul 28 04:48:29.315199 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:29.315172    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae9f1f22-fc7fc5cbd-xcrck"
Jul 28 04:48:29.342961 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:29.342932    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ae9f1f22-b68f48b94-x68n8"
Jul 28 04:48:29.454044 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:29.453990    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ae9f1f22-864f4688b9-dpmr2"]
Jul 28 04:48:29.456783 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:48:29.456737    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1051c09d_0b3e_4fdd_85cf_94b6c3e7f646.slice/crio-21cd1e940ea22415b7abaf3262b32249ecd37b55dccfb2a2a9d737b0910ea14c WatchSource:0}: Error finding container 21cd1e940ea22415b7abaf3262b32249ecd37b55dccfb2a2a9d737b0910ea14c: Status 404 returned error can't find the container with id 21cd1e940ea22415b7abaf3262b32249ecd37b55dccfb2a2a9d737b0910ea14c
Jul 28 04:48:29.515006 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:29.514970    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ae9f1f22-b68f48b94-x68n8"]
Jul 28 04:48:29.516541 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:48:29.516516    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19c645e3_83f9_4fa4_96b3_73277c1e78c5.slice/crio-ded40eec990fc1d02d48808cde535e71c05207b7508437110be9aaae138dec83 WatchSource:0}: Error finding container ded40eec990fc1d02d48808cde535e71c05207b7508437110be9aaae138dec83: Status 404 returned error can't find the container with id ded40eec990fc1d02d48808cde535e71c05207b7508437110be9aaae138dec83
Jul 28 04:48:29.683741 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:29.683710    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ae9f1f22-fc7fc5cbd-xcrck"]
Jul 28 04:48:29.685569 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:48:29.685537    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ba02699_ef32_49a9_ac5e_c96c2049b024.slice/crio-45b0ccb44f409967c1670a5a326d3f53b8e7f681cc7d93533d7cb452d89a8c28 WatchSource:0}: Error finding container 45b0ccb44f409967c1670a5a326d3f53b8e7f681cc7d93533d7cb452d89a8c28: Status 404 returned error can't find the container with id 45b0ccb44f409967c1670a5a326d3f53b8e7f681cc7d93533d7cb452d89a8c28
Jul 28 04:48:30.188520 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:30.188489    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae9f1f22-fc7fc5cbd-xcrck" event={"ID":"9ba02699-ef32-49a9-ac5e-c96c2049b024","Type":"ContainerStarted","Data":"08a682291e12529c543b7d881de537a7eaf8cb443835b31bf579381efa9f692a"}
Jul 28 04:48:30.188706 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:30.188545    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae9f1f22-fc7fc5cbd-xcrck" event={"ID":"9ba02699-ef32-49a9-ac5e-c96c2049b024","Type":"ContainerStarted","Data":"45b0ccb44f409967c1670a5a326d3f53b8e7f681cc7d93533d7cb452d89a8c28"}
Jul 28 04:48:30.188706 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:30.188670    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae9f1f22-fc7fc5cbd-xcrck"
Jul 28 04:48:30.190170 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:30.190145    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae9f1f22-864f4688b9-dpmr2" event={"ID":"1051c09d-0b3e-4fdd-85cf-94b6c3e7f646","Type":"ContainerStarted","Data":"a31da1e32cdda6fd4d21a1ad380d749cf339f2280fd2cd65fcb595ce7834cea2"}
Jul 28 04:48:30.190312 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:30.190187    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae9f1f22-864f4688b9-dpmr2" event={"ID":"1051c09d-0b3e-4fdd-85cf-94b6c3e7f646","Type":"ContainerStarted","Data":"21cd1e940ea22415b7abaf3262b32249ecd37b55dccfb2a2a9d737b0910ea14c"}
Jul 28 04:48:30.190312 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:30.190252    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae9f1f22-864f4688b9-dpmr2"
Jul 28 04:48:30.191644 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:30.191624    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ae9f1f22-b68f48b94-x68n8" event={"ID":"19c645e3-83f9-4fa4-96b3-73277c1e78c5","Type":"ContainerStarted","Data":"8c6a010a3f0a03815a21b0bf10a11dc5856ce41fb6afc51e025dcc50fb325c86"}
Jul 28 04:48:30.191734 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:30.191667    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ae9f1f22-b68f48b94-x68n8" event={"ID":"19c645e3-83f9-4fa4-96b3-73277c1e78c5","Type":"ContainerStarted","Data":"ded40eec990fc1d02d48808cde535e71c05207b7508437110be9aaae138dec83"}
Jul 28 04:48:30.191734 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:30.191707    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-ae9f1f22-b68f48b94-x68n8"
Jul 28 04:48:30.204478 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:30.204396    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae9f1f22-fc7fc5cbd-xcrck" podStartSLOduration=2.204383863 podStartE2EDuration="2.204383863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:48:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:48:30.202400401 +0000 UTC m=+2210.592576425" watchObservedRunningTime="2026-07-28 04:48:30.204383863 +0000 UTC m=+2210.594559884"
Jul 28 04:48:30.217989 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:30.217940    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae9f1f22-864f4688b9-dpmr2" podStartSLOduration=1.7220048380000001 podStartE2EDuration="2.217924874s" totalImagesPullingTime="495.920036ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:48:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:48:30.216798557 +0000 UTC m=+2210.606974579" watchObservedRunningTime="2026-07-28 04:48:30.217924874 +0000 UTC m=+2210.608100894"
Jul 28 04:48:30.232509 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:30.232461    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-ae9f1f22-b68f48b94-x68n8" podStartSLOduration=1.232446897 podStartE2EDuration="1.232446897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:48:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:48:30.231119173 +0000 UTC m=+2210.621295222" watchObservedRunningTime="2026-07-28 04:48:30.232446897 +0000 UTC m=+2210.622622922"
Jul 28 04:48:36.202254 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:36.202224    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae9f1f22-864f4688b9-dpmr2"
Jul 28 04:48:41.198413 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:41.198315    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-ae9f1f22-b68f48b94-x68n8"
Jul 28 04:48:41.198413 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:41.198404    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae9f1f22-fc7fc5cbd-xcrck"
Jul 28 04:48:41.258589 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:41.258495    2566 generic.go:356] "Generic (PLEG): container finished" podID="0c3cbb68-4002-4255-84e4-b83fca6162e8" containerID="f9768be2633b9b8841a6192be837d32a5736b66a6142d4a1f15349f9bc670ee9" exitCode=0
Jul 28 04:48:41.258753 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:41.258580    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9" event={"ID":"0c3cbb68-4002-4255-84e4-b83fca6162e8","Type":"ContainerDied","Data":"f9768be2633b9b8841a6192be837d32a5736b66a6142d4a1f15349f9bc670ee9"}
Jul 28 04:48:41.259131 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:41.259114    2566 scope.go:117] "RemoveContainer" containerID="f9768be2633b9b8841a6192be837d32a5736b66a6142d4a1f15349f9bc670ee9"
Jul 28 04:48:41.260182 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:41.260143    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:48:41.260292 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:41.260221    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:48:41.260292 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:41.260264    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:48:42.173439 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:42.173405    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9"
Jul 28 04:48:42.265883 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:42.265828    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9" event={"ID":"0c3cbb68-4002-4255-84e4-b83fca6162e8","Type":"ContainerStarted","Data":"46a0b325ed2e9c751b0ed55275fe72c34cf3363777ed4383d88884c83bd89553"}
Jul 28 04:48:42.421952 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:42.421903    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9"
Jul 28 04:48:42.423325 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:42.423287    2566 patch_prober.go:28] interesting pod/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Jul 28 04:48:42.423506 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:42.423379    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9" podUID="0c3cbb68-4002-4255-84e4-b83fca6162e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Jul 28 04:48:43.422259 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:43.422210    2566 patch_prober.go:28] interesting pod/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Jul 28 04:48:43.422793 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:43.422282    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9" podUID="0c3cbb68-4002-4255-84e4-b83fca6162e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Jul 28 04:48:44.421420 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:44.421372    2566 patch_prober.go:28] interesting pod/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Jul 28 04:48:44.421636 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:44.421443    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9" podUID="0c3cbb68-4002-4255-84e4-b83fca6162e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Jul 28 04:48:45.421643 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:45.421600    2566 patch_prober.go:28] interesting pod/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Jul 28 04:48:45.422281 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:45.422248    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9" podUID="0c3cbb68-4002-4255-84e4-b83fca6162e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Jul 28 04:48:46.422332 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:46.422287    2566 patch_prober.go:28] interesting pod/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Jul 28 04:48:46.422831 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:46.422360    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9" podUID="0c3cbb68-4002-4255-84e4-b83fca6162e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Jul 28 04:48:47.426137 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:47.426106    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9"
Jul 28 04:48:47.426531 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:47.426392    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9"
Jul 28 04:48:47.427088 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:47.427064    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9"
Jul 28 04:48:50.166525 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:50.166488    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ae9f1f22-864f4688b9-dpmr2"]
Jul 28 04:48:50.167009 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:50.166754    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae9f1f22-864f4688b9-dpmr2" podUID="1051c09d-0b3e-4fdd-85cf-94b6c3e7f646" containerName="maas-api" containerID="cri-o://a31da1e32cdda6fd4d21a1ad380d749cf339f2280fd2cd65fcb595ce7834cea2" gracePeriod=30
Jul 28 04:48:50.220479 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:50.220444    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ae9f1f22-b68f48b94-x68n8"]
Jul 28 04:48:50.220748 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:50.220717    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-ae9f1f22-b68f48b94-x68n8" podUID="19c645e3-83f9-4fa4-96b3-73277c1e78c5" containerName="payload-processing" containerID="cri-o://8c6a010a3f0a03815a21b0bf10a11dc5856ce41fb6afc51e025dcc50fb325c86" gracePeriod=30
Jul 28 04:48:50.229031 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:50.229003    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ae9f1f22-fc7fc5cbd-xcrck"]
Jul 28 04:48:50.229266 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:50.229241    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae9f1f22-fc7fc5cbd-xcrck" podUID="9ba02699-ef32-49a9-ac5e-c96c2049b024" containerName="payload-pre-processing" containerID="cri-o://08a682291e12529c543b7d881de537a7eaf8cb443835b31bf579381efa9f692a" gracePeriod=30
Jul 28 04:48:50.310224 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:50.310194    2566 generic.go:356] "Generic (PLEG): container finished" podID="1051c09d-0b3e-4fdd-85cf-94b6c3e7f646" containerID="a31da1e32cdda6fd4d21a1ad380d749cf339f2280fd2cd65fcb595ce7834cea2" exitCode=0
Jul 28 04:48:50.310357 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:50.310233    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae9f1f22-864f4688b9-dpmr2" event={"ID":"1051c09d-0b3e-4fdd-85cf-94b6c3e7f646","Type":"ContainerDied","Data":"a31da1e32cdda6fd4d21a1ad380d749cf339f2280fd2cd65fcb595ce7834cea2"}
Jul 28 04:48:50.509436 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:50.509409    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae9f1f22-864f4688b9-dpmr2"
Jul 28 04:48:50.512765 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:50.512741    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ae9f1f22-b68f48b94-x68n8"
Jul 28 04:48:50.526388 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:50.526369    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae9f1f22-fc7fc5cbd-xcrck"
Jul 28 04:48:50.634705 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:50.634671    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ba02699-ef32-49a9-ac5e-c96c2049b024-plugins-config-volume\") pod \"9ba02699-ef32-49a9-ac5e-c96c2049b024\" (UID: \"9ba02699-ef32-49a9-ac5e-c96c2049b024\") "
Jul 28 04:48:50.634934 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:50.634735    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lspxw\" (UniqueName: \"kubernetes.io/projected/9ba02699-ef32-49a9-ac5e-c96c2049b024-kube-api-access-lspxw\") pod \"9ba02699-ef32-49a9-ac5e-c96c2049b024\" (UID: \"9ba02699-ef32-49a9-ac5e-c96c2049b024\") "
Jul 28 04:48:50.634934 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:50.634792    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1051c09d-0b3e-4fdd-85cf-94b6c3e7f646-maas-api-tls\") pod \"1051c09d-0b3e-4fdd-85cf-94b6c3e7f646\" (UID: \"1051c09d-0b3e-4fdd-85cf-94b6c3e7f646\") "
Jul 28 04:48:50.634934 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:50.634816    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1051c09d-0b3e-4fdd-85cf-94b6c3e7f646-openshift-service-ca\") pod \"1051c09d-0b3e-4fdd-85cf-94b6c3e7f646\" (UID: \"1051c09d-0b3e-4fdd-85cf-94b6c3e7f646\") "
Jul 28 04:48:50.634934 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:50.634905    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fnvqh\" (UniqueName: \"kubernetes.io/projected/1051c09d-0b3e-4fdd-85cf-94b6c3e7f646-kube-api-access-fnvqh\") pod \"1051c09d-0b3e-4fdd-85cf-94b6c3e7f646\" (UID: \"1051c09d-0b3e-4fdd-85cf-94b6c3e7f646\") "
Jul 28 04:48:50.634934 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:50.634932    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52fb7\" (UniqueName: \"kubernetes.io/projected/19c645e3-83f9-4fa4-96b3-73277c1e78c5-kube-api-access-52fb7\") pod \"19c645e3-83f9-4fa4-96b3-73277c1e78c5\" (UID: \"19c645e3-83f9-4fa4-96b3-73277c1e78c5\") "
Jul 28 04:48:50.635235 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:50.634986    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19c645e3-83f9-4fa4-96b3-73277c1e78c5-plugins-config-volume\") pod \"19c645e3-83f9-4fa4-96b3-73277c1e78c5\" (UID: \"19c645e3-83f9-4fa4-96b3-73277c1e78c5\") "
Jul 28 04:48:50.635287 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:50.635228    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ba02699-ef32-49a9-ac5e-c96c2049b024-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9ba02699-ef32-49a9-ac5e-c96c2049b024" (UID: "9ba02699-ef32-49a9-ac5e-c96c2049b024"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 04:48:50.635341 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:50.635311    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1051c09d-0b3e-4fdd-85cf-94b6c3e7f646-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1051c09d-0b3e-4fdd-85cf-94b6c3e7f646" (UID: "1051c09d-0b3e-4fdd-85cf-94b6c3e7f646"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 04:48:50.635507 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:50.635489    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ba02699-ef32-49a9-ac5e-c96c2049b024-plugins-config-volume\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:48:50.635576 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:50.635514    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1051c09d-0b3e-4fdd-85cf-94b6c3e7f646-openshift-service-ca\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:48:50.635636 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:50.635597    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19c645e3-83f9-4fa4-96b3-73277c1e78c5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "19c645e3-83f9-4fa4-96b3-73277c1e78c5" (UID: "19c645e3-83f9-4fa4-96b3-73277c1e78c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 04:48:50.637107 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:50.637080    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1051c09d-0b3e-4fdd-85cf-94b6c3e7f646-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1051c09d-0b3e-4fdd-85cf-94b6c3e7f646" (UID: "1051c09d-0b3e-4fdd-85cf-94b6c3e7f646"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 04:48:50.637209 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:50.637124    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1051c09d-0b3e-4fdd-85cf-94b6c3e7f646-kube-api-access-fnvqh" (OuterVolumeSpecName: "kube-api-access-fnvqh") pod "1051c09d-0b3e-4fdd-85cf-94b6c3e7f646" (UID: "1051c09d-0b3e-4fdd-85cf-94b6c3e7f646"). InnerVolumeSpecName "kube-api-access-fnvqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:48:50.637428 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:50.637408    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ba02699-ef32-49a9-ac5e-c96c2049b024-kube-api-access-lspxw" (OuterVolumeSpecName: "kube-api-access-lspxw") pod "9ba02699-ef32-49a9-ac5e-c96c2049b024" (UID: "9ba02699-ef32-49a9-ac5e-c96c2049b024"). InnerVolumeSpecName "kube-api-access-lspxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:48:50.637485 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:50.637436    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19c645e3-83f9-4fa4-96b3-73277c1e78c5-kube-api-access-52fb7" (OuterVolumeSpecName: "kube-api-access-52fb7") pod "19c645e3-83f9-4fa4-96b3-73277c1e78c5" (UID: "19c645e3-83f9-4fa4-96b3-73277c1e78c5"). InnerVolumeSpecName "kube-api-access-52fb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:48:50.736147 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:50.736058    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnvqh\" (UniqueName: \"kubernetes.io/projected/1051c09d-0b3e-4fdd-85cf-94b6c3e7f646-kube-api-access-fnvqh\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:48:50.736147 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:50.736089    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52fb7\" (UniqueName: \"kubernetes.io/projected/19c645e3-83f9-4fa4-96b3-73277c1e78c5-kube-api-access-52fb7\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:48:50.736147 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:50.736098    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19c645e3-83f9-4fa4-96b3-73277c1e78c5-plugins-config-volume\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:48:50.736147 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:50.736107    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lspxw\" (UniqueName: \"kubernetes.io/projected/9ba02699-ef32-49a9-ac5e-c96c2049b024-kube-api-access-lspxw\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:48:50.736147 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:50.736117    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1051c09d-0b3e-4fdd-85cf-94b6c3e7f646-maas-api-tls\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:48:51.316296 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:51.316258    2566 generic.go:356] "Generic (PLEG): container finished" podID="19c645e3-83f9-4fa4-96b3-73277c1e78c5" containerID="8c6a010a3f0a03815a21b0bf10a11dc5856ce41fb6afc51e025dcc50fb325c86" exitCode=0
Jul 28 04:48:51.316757 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:51.316308    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ae9f1f22-b68f48b94-x68n8"
Jul 28 04:48:51.316757 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:51.316343    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ae9f1f22-b68f48b94-x68n8" event={"ID":"19c645e3-83f9-4fa4-96b3-73277c1e78c5","Type":"ContainerDied","Data":"8c6a010a3f0a03815a21b0bf10a11dc5856ce41fb6afc51e025dcc50fb325c86"}
Jul 28 04:48:51.316757 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:51.316406    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ae9f1f22-b68f48b94-x68n8" event={"ID":"19c645e3-83f9-4fa4-96b3-73277c1e78c5","Type":"ContainerDied","Data":"ded40eec990fc1d02d48808cde535e71c05207b7508437110be9aaae138dec83"}
Jul 28 04:48:51.316757 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:51.316451    2566 scope.go:117] "RemoveContainer" containerID="8c6a010a3f0a03815a21b0bf10a11dc5856ce41fb6afc51e025dcc50fb325c86"
Jul 28 04:48:51.317792 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:51.317737    2566 generic.go:356] "Generic (PLEG): container finished" podID="9ba02699-ef32-49a9-ac5e-c96c2049b024" containerID="08a682291e12529c543b7d881de537a7eaf8cb443835b31bf579381efa9f692a" exitCode=0
Jul 28 04:48:51.317792 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:51.317785    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae9f1f22-fc7fc5cbd-xcrck"
Jul 28 04:48:51.317950 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:51.317829    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae9f1f22-fc7fc5cbd-xcrck" event={"ID":"9ba02699-ef32-49a9-ac5e-c96c2049b024","Type":"ContainerDied","Data":"08a682291e12529c543b7d881de537a7eaf8cb443835b31bf579381efa9f692a"}
Jul 28 04:48:51.317950 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:51.317891    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ae9f1f22-fc7fc5cbd-xcrck" event={"ID":"9ba02699-ef32-49a9-ac5e-c96c2049b024","Type":"ContainerDied","Data":"45b0ccb44f409967c1670a5a326d3f53b8e7f681cc7d93533d7cb452d89a8c28"}
Jul 28 04:48:51.319257 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:51.319234    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae9f1f22-864f4688b9-dpmr2" event={"ID":"1051c09d-0b3e-4fdd-85cf-94b6c3e7f646","Type":"ContainerDied","Data":"21cd1e940ea22415b7abaf3262b32249ecd37b55dccfb2a2a9d737b0910ea14c"}
Jul 28 04:48:51.319375 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:51.319261    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-ae9f1f22-864f4688b9-dpmr2"
Jul 28 04:48:51.325724 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:51.325693    2566 scope.go:117] "RemoveContainer" containerID="8c6a010a3f0a03815a21b0bf10a11dc5856ce41fb6afc51e025dcc50fb325c86"
Jul 28 04:48:51.326060 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:48:51.326041    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c6a010a3f0a03815a21b0bf10a11dc5856ce41fb6afc51e025dcc50fb325c86\": container with ID starting with 8c6a010a3f0a03815a21b0bf10a11dc5856ce41fb6afc51e025dcc50fb325c86 not found: ID does not exist" containerID="8c6a010a3f0a03815a21b0bf10a11dc5856ce41fb6afc51e025dcc50fb325c86"
Jul 28 04:48:51.326161 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:51.326083    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c6a010a3f0a03815a21b0bf10a11dc5856ce41fb6afc51e025dcc50fb325c86"} err="failed to get container status \"8c6a010a3f0a03815a21b0bf10a11dc5856ce41fb6afc51e025dcc50fb325c86\": rpc error: code = NotFound desc = could not find container \"8c6a010a3f0a03815a21b0bf10a11dc5856ce41fb6afc51e025dcc50fb325c86\": container with ID starting with 8c6a010a3f0a03815a21b0bf10a11dc5856ce41fb6afc51e025dcc50fb325c86 not found: ID does not exist"
Jul 28 04:48:51.326161 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:51.326106    2566 scope.go:117] "RemoveContainer" containerID="08a682291e12529c543b7d881de537a7eaf8cb443835b31bf579381efa9f692a"
Jul 28 04:48:51.335055 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:51.335037    2566 scope.go:117] "RemoveContainer" containerID="08a682291e12529c543b7d881de537a7eaf8cb443835b31bf579381efa9f692a"
Jul 28 04:48:51.335289 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:48:51.335271    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08a682291e12529c543b7d881de537a7eaf8cb443835b31bf579381efa9f692a\": container with ID starting with 08a682291e12529c543b7d881de537a7eaf8cb443835b31bf579381efa9f692a not found: ID does not exist" containerID="08a682291e12529c543b7d881de537a7eaf8cb443835b31bf579381efa9f692a"
Jul 28 04:48:51.335335 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:51.335297    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08a682291e12529c543b7d881de537a7eaf8cb443835b31bf579381efa9f692a"} err="failed to get container status \"08a682291e12529c543b7d881de537a7eaf8cb443835b31bf579381efa9f692a\": rpc error: code = NotFound desc = could not find container \"08a682291e12529c543b7d881de537a7eaf8cb443835b31bf579381efa9f692a\": container with ID starting with 08a682291e12529c543b7d881de537a7eaf8cb443835b31bf579381efa9f692a not found: ID does not exist"
Jul 28 04:48:51.335335 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:51.335314    2566 scope.go:117] "RemoveContainer" containerID="a31da1e32cdda6fd4d21a1ad380d749cf339f2280fd2cd65fcb595ce7834cea2"
Jul 28 04:48:51.342901 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:51.342878    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ae9f1f22-b68f48b94-x68n8"]
Jul 28 04:48:51.345732 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:51.345711    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ae9f1f22-b68f48b94-x68n8"]
Jul 28 04:48:51.355369 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:51.355349    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ae9f1f22-fc7fc5cbd-xcrck"]
Jul 28 04:48:51.358906 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:51.358886    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ae9f1f22-fc7fc5cbd-xcrck"]
Jul 28 04:48:51.368555 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:51.368531    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ae9f1f22-864f4688b9-dpmr2"]
Jul 28 04:48:51.372483 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:51.372463    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-ae9f1f22-864f4688b9-dpmr2"]
Jul 28 04:48:52.088596 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:52.088562    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1051c09d-0b3e-4fdd-85cf-94b6c3e7f646" path="/var/lib/kubelet/pods/1051c09d-0b3e-4fdd-85cf-94b6c3e7f646/volumes"
Jul 28 04:48:52.088988 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:52.088972    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19c645e3-83f9-4fa4-96b3-73277c1e78c5" path="/var/lib/kubelet/pods/19c645e3-83f9-4fa4-96b3-73277c1e78c5/volumes"
Jul 28 04:48:52.089332 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:48:52.089316    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ba02699-ef32-49a9-ac5e-c96c2049b024" path="/var/lib/kubelet/pods/9ba02699-ef32-49a9-ac5e-c96c2049b024/volumes"
Jul 28 04:49:01.053094 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:01.053049    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9"]
Jul 28 04:49:01.053716 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:01.053294    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9" podUID="0c3cbb68-4002-4255-84e4-b83fca6162e8" containerName="istio-proxy" containerID="cri-o://46a0b325ed2e9c751b0ed55275fe72c34cf3363777ed4383d88884c83bd89553" gracePeriod=30
Jul 28 04:49:02.429579 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:02.429485    2566 patch_prober.go:28] interesting pod/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 28 04:49:02.429579 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:02.429555    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9" podUID="0c3cbb68-4002-4255-84e4-b83fca6162e8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 04:49:06.297429 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:06.297406    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9"
Jul 28 04:49:06.385649 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:06.385558    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c3cbb68-4002-4255-84e4-b83fca6162e8-istio-data\") pod \"0c3cbb68-4002-4255-84e4-b83fca6162e8\" (UID: \"0c3cbb68-4002-4255-84e4-b83fca6162e8\") "
Jul 28 04:49:06.385649 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:06.385594    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c3cbb68-4002-4255-84e4-b83fca6162e8-istiod-ca-cert\") pod \"0c3cbb68-4002-4255-84e4-b83fca6162e8\" (UID: \"0c3cbb68-4002-4255-84e4-b83fca6162e8\") "
Jul 28 04:49:06.385649 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:06.385626    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c3cbb68-4002-4255-84e4-b83fca6162e8-workload-certs\") pod \"0c3cbb68-4002-4255-84e4-b83fca6162e8\" (UID: \"0c3cbb68-4002-4255-84e4-b83fca6162e8\") "
Jul 28 04:49:06.385979 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:06.385651    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpxzn\" (UniqueName: \"kubernetes.io/projected/0c3cbb68-4002-4255-84e4-b83fca6162e8-kube-api-access-hpxzn\") pod \"0c3cbb68-4002-4255-84e4-b83fca6162e8\" (UID: \"0c3cbb68-4002-4255-84e4-b83fca6162e8\") "
Jul 28 04:49:06.385979 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:06.385676    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c3cbb68-4002-4255-84e4-b83fca6162e8-workload-socket\") pod \"0c3cbb68-4002-4255-84e4-b83fca6162e8\" (UID: \"0c3cbb68-4002-4255-84e4-b83fca6162e8\") "
Jul 28 04:49:06.385979 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:06.385711    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c3cbb68-4002-4255-84e4-b83fca6162e8-istio-envoy\") pod \"0c3cbb68-4002-4255-84e4-b83fca6162e8\" (UID: \"0c3cbb68-4002-4255-84e4-b83fca6162e8\") "
Jul 28 04:49:06.385979 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:06.385860    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c3cbb68-4002-4255-84e4-b83fca6162e8-credential-socket\") pod \"0c3cbb68-4002-4255-84e4-b83fca6162e8\" (UID: \"0c3cbb68-4002-4255-84e4-b83fca6162e8\") "
Jul 28 04:49:06.385979 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:06.385901    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c3cbb68-4002-4255-84e4-b83fca6162e8-istio-token\") pod \"0c3cbb68-4002-4255-84e4-b83fca6162e8\" (UID: \"0c3cbb68-4002-4255-84e4-b83fca6162e8\") "
Jul 28 04:49:06.385979 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:06.385942    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c3cbb68-4002-4255-84e4-b83fca6162e8-istio-podinfo\") pod \"0c3cbb68-4002-4255-84e4-b83fca6162e8\" (UID: \"0c3cbb68-4002-4255-84e4-b83fca6162e8\") "
Jul 28 04:49:06.386287 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:06.386056    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c3cbb68-4002-4255-84e4-b83fca6162e8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0c3cbb68-4002-4255-84e4-b83fca6162e8" (UID: "0c3cbb68-4002-4255-84e4-b83fca6162e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:49:06.386287 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:06.386074    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c3cbb68-4002-4255-84e4-b83fca6162e8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0c3cbb68-4002-4255-84e4-b83fca6162e8" (UID: "0c3cbb68-4002-4255-84e4-b83fca6162e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:49:06.386287 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:06.386222    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c3cbb68-4002-4255-84e4-b83fca6162e8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0c3cbb68-4002-4255-84e4-b83fca6162e8" (UID: "0c3cbb68-4002-4255-84e4-b83fca6162e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:49:06.386287 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:06.386255    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c3cbb68-4002-4255-84e4-b83fca6162e8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0c3cbb68-4002-4255-84e4-b83fca6162e8" (UID: "0c3cbb68-4002-4255-84e4-b83fca6162e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 04:49:06.386515 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:06.386390    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c3cbb68-4002-4255-84e4-b83fca6162e8-credential-socket\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:49:06.386515 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:06.386412    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c3cbb68-4002-4255-84e4-b83fca6162e8-istiod-ca-cert\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:49:06.386515 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:06.386427    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c3cbb68-4002-4255-84e4-b83fca6162e8-workload-certs\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:49:06.386515 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:06.386442    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c3cbb68-4002-4255-84e4-b83fca6162e8-workload-socket\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:49:06.386722 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:06.386510    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c3cbb68-4002-4255-84e4-b83fca6162e8-istio-data" (OuterVolumeSpecName: "istio-data") pod "0c3cbb68-4002-4255-84e4-b83fca6162e8" (UID: "0c3cbb68-4002-4255-84e4-b83fca6162e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:49:06.388206 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:06.388174    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c3cbb68-4002-4255-84e4-b83fca6162e8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0c3cbb68-4002-4255-84e4-b83fca6162e8" (UID: "0c3cbb68-4002-4255-84e4-b83fca6162e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 04:49:06.388326 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:06.388206    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c3cbb68-4002-4255-84e4-b83fca6162e8-istio-token" (OuterVolumeSpecName: "istio-token") pod "0c3cbb68-4002-4255-84e4-b83fca6162e8" (UID: "0c3cbb68-4002-4255-84e4-b83fca6162e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:49:06.388326 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:06.388279    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c3cbb68-4002-4255-84e4-b83fca6162e8-kube-api-access-hpxzn" (OuterVolumeSpecName: "kube-api-access-hpxzn") pod "0c3cbb68-4002-4255-84e4-b83fca6162e8" (UID: "0c3cbb68-4002-4255-84e4-b83fca6162e8"). InnerVolumeSpecName "kube-api-access-hpxzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:49:06.388423 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:06.388374    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c3cbb68-4002-4255-84e4-b83fca6162e8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0c3cbb68-4002-4255-84e4-b83fca6162e8" (UID: "0c3cbb68-4002-4255-84e4-b83fca6162e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:49:06.393410 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:06.393382    2566 generic.go:356] "Generic (PLEG): container finished" podID="0c3cbb68-4002-4255-84e4-b83fca6162e8" containerID="46a0b325ed2e9c751b0ed55275fe72c34cf3363777ed4383d88884c83bd89553" exitCode=0
Jul 28 04:49:06.393531 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:06.393437    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9"
Jul 28 04:49:06.393531 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:06.393465    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9" event={"ID":"0c3cbb68-4002-4255-84e4-b83fca6162e8","Type":"ContainerDied","Data":"46a0b325ed2e9c751b0ed55275fe72c34cf3363777ed4383d88884c83bd89553"}
Jul 28 04:49:06.393531 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:06.393528    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9" event={"ID":"0c3cbb68-4002-4255-84e4-b83fca6162e8","Type":"ContainerDied","Data":"a9983e3f55521935613311a590e504a9e8765602366dbe6fc1fb48b55aea91f2"}
Jul 28 04:49:06.393703 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:06.393560    2566 scope.go:117] "RemoveContainer" containerID="46a0b325ed2e9c751b0ed55275fe72c34cf3363777ed4383d88884c83bd89553"
Jul 28 04:49:06.408938 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:06.408915    2566 scope.go:117] "RemoveContainer" containerID="f9768be2633b9b8841a6192be837d32a5736b66a6142d4a1f15349f9bc670ee9"
Jul 28 04:49:06.417535 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:06.417514    2566 scope.go:117] "RemoveContainer" containerID="46a0b325ed2e9c751b0ed55275fe72c34cf3363777ed4383d88884c83bd89553"
Jul 28 04:49:06.417815 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:49:06.417795    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46a0b325ed2e9c751b0ed55275fe72c34cf3363777ed4383d88884c83bd89553\": container with ID starting with 46a0b325ed2e9c751b0ed55275fe72c34cf3363777ed4383d88884c83bd89553 not found: ID does not exist" containerID="46a0b325ed2e9c751b0ed55275fe72c34cf3363777ed4383d88884c83bd89553"
Jul 28 04:49:06.417880 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:06.417825    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46a0b325ed2e9c751b0ed55275fe72c34cf3363777ed4383d88884c83bd89553"} err="failed to get container status \"46a0b325ed2e9c751b0ed55275fe72c34cf3363777ed4383d88884c83bd89553\": rpc error: code = NotFound desc = could not find container \"46a0b325ed2e9c751b0ed55275fe72c34cf3363777ed4383d88884c83bd89553\": container with ID starting with 46a0b325ed2e9c751b0ed55275fe72c34cf3363777ed4383d88884c83bd89553 not found: ID does not exist"
Jul 28 04:49:06.417880 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:06.417865    2566 scope.go:117] "RemoveContainer" containerID="f9768be2633b9b8841a6192be837d32a5736b66a6142d4a1f15349f9bc670ee9"
Jul 28 04:49:06.418180 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:49:06.418161    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9768be2633b9b8841a6192be837d32a5736b66a6142d4a1f15349f9bc670ee9\": container with ID starting with f9768be2633b9b8841a6192be837d32a5736b66a6142d4a1f15349f9bc670ee9 not found: ID does not exist" containerID="f9768be2633b9b8841a6192be837d32a5736b66a6142d4a1f15349f9bc670ee9"
Jul 28 04:49:06.418222 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:06.418188    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9768be2633b9b8841a6192be837d32a5736b66a6142d4a1f15349f9bc670ee9"} err="failed to get container status \"f9768be2633b9b8841a6192be837d32a5736b66a6142d4a1f15349f9bc670ee9\": rpc error: code = NotFound desc = could not find container \"f9768be2633b9b8841a6192be837d32a5736b66a6142d4a1f15349f9bc670ee9\": container with ID starting with f9768be2633b9b8841a6192be837d32a5736b66a6142d4a1f15349f9bc670ee9 not found: ID does not exist"
Jul 28 04:49:06.422202 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:06.422179    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9"]
Jul 28 04:49:06.426365 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:06.426343    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ae9f1f22-openshift-default-7744576bb4-vplf9"]
Jul 28 04:49:06.487340 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:06.487313    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c3cbb68-4002-4255-84e4-b83fca6162e8-istio-token\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:49:06.487340 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:06.487335    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c3cbb68-4002-4255-84e4-b83fca6162e8-istio-podinfo\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:49:06.487340 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:06.487345    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c3cbb68-4002-4255-84e4-b83fca6162e8-istio-data\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:49:06.487540 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:06.487356    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpxzn\" (UniqueName: \"kubernetes.io/projected/0c3cbb68-4002-4255-84e4-b83fca6162e8-kube-api-access-hpxzn\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:49:06.487540 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:06.487367    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c3cbb68-4002-4255-84e4-b83fca6162e8-istio-envoy\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:49:08.088788 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:08.088743    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c3cbb68-4002-4255-84e4-b83fca6162e8" path="/var/lib/kubelet/pods/0c3cbb68-4002-4255-84e4-b83fca6162e8/volumes"
Jul 28 04:49:20.765286 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:20.765250    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh"]
Jul 28 04:49:20.765876 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:20.765853    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c3cbb68-4002-4255-84e4-b83fca6162e8" containerName="istio-proxy"
Jul 28 04:49:20.765951 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:20.765880    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c3cbb68-4002-4255-84e4-b83fca6162e8" containerName="istio-proxy"
Jul 28 04:49:20.765951 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:20.765900    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19c645e3-83f9-4fa4-96b3-73277c1e78c5" containerName="payload-processing"
Jul 28 04:49:20.765951 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:20.765910    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="19c645e3-83f9-4fa4-96b3-73277c1e78c5" containerName="payload-processing"
Jul 28 04:49:20.766110 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:20.765962    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ba02699-ef32-49a9-ac5e-c96c2049b024" containerName="payload-pre-processing"
Jul 28 04:49:20.766110 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:20.765971    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ba02699-ef32-49a9-ac5e-c96c2049b024" containerName="payload-pre-processing"
Jul 28 04:49:20.766110 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:20.765982    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1051c09d-0b3e-4fdd-85cf-94b6c3e7f646" containerName="maas-api"
Jul 28 04:49:20.766110 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:20.765991    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1051c09d-0b3e-4fdd-85cf-94b6c3e7f646" containerName="maas-api"
Jul 28 04:49:20.766110 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:20.766110    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1051c09d-0b3e-4fdd-85cf-94b6c3e7f646" containerName="maas-api"
Jul 28 04:49:20.766363 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:20.766123    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c3cbb68-4002-4255-84e4-b83fca6162e8" containerName="istio-proxy"
Jul 28 04:49:20.766363 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:20.766132    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="19c645e3-83f9-4fa4-96b3-73277c1e78c5" containerName="payload-processing"
Jul 28 04:49:20.766363 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:20.766147    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ba02699-ef32-49a9-ac5e-c96c2049b024" containerName="payload-pre-processing"
Jul 28 04:49:20.766363 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:20.766159    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c3cbb68-4002-4255-84e4-b83fca6162e8" containerName="istio-proxy"
Jul 28 04:49:20.769523 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:20.769502    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh"
Jul 28 04:49:20.772187 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:20.772160    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 28 04:49:20.772299 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:20.772274    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-890677f3-openshift-default-dockercfg-7t6n4\""
Jul 28 04:49:20.779090 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:20.779068    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh"]
Jul 28 04:49:20.819485 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:20.819456    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-istio-data\") pod \"e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh\" (UID: \"81b65161-13f4-49ad-b8c1-6dbe36ccab7a\") " pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh"
Jul 28 04:49:20.819652 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:20.819500    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-credential-socket\") pod \"e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh\" (UID: \"81b65161-13f4-49ad-b8c1-6dbe36ccab7a\") " pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh"
Jul 28 04:49:20.819652 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:20.819524    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-istio-envoy\") pod \"e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh\" (UID: \"81b65161-13f4-49ad-b8c1-6dbe36ccab7a\") " pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh"
Jul 28 04:49:20.819652 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:20.819556    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-istio-token\") pod \"e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh\" (UID: \"81b65161-13f4-49ad-b8c1-6dbe36ccab7a\") " pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh"
Jul 28 04:49:20.819652 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:20.819613    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-istiod-ca-cert\") pod \"e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh\" (UID: \"81b65161-13f4-49ad-b8c1-6dbe36ccab7a\") " pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh"
Jul 28 04:49:20.819786 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:20.819660    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-workload-certs\") pod \"e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh\" (UID: \"81b65161-13f4-49ad-b8c1-6dbe36ccab7a\") " pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh"
Jul 28 04:49:20.819786 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:20.819684    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-istio-podinfo\") pod \"e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh\" (UID: \"81b65161-13f4-49ad-b8c1-6dbe36ccab7a\") " pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh"
Jul 28 04:49:20.819786 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:20.819762    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq8vn\" (UniqueName: \"kubernetes.io/projected/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-kube-api-access-sq8vn\") pod \"e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh\" (UID: \"81b65161-13f4-49ad-b8c1-6dbe36ccab7a\") " pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh"
Jul 28 04:49:20.819895 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:20.819809    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-workload-socket\") pod \"e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh\" (UID: \"81b65161-13f4-49ad-b8c1-6dbe36ccab7a\") " pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh"
Jul 28 04:49:20.920609 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:20.920569    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-istio-envoy\") pod \"e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh\" (UID: \"81b65161-13f4-49ad-b8c1-6dbe36ccab7a\") " pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh"
Jul 28 04:49:20.920802 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:20.920615    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-istio-token\") pod \"e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh\" (UID: \"81b65161-13f4-49ad-b8c1-6dbe36ccab7a\") " pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh"
Jul 28 04:49:20.920802 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:20.920649    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-istiod-ca-cert\") pod \"e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh\" (UID: \"81b65161-13f4-49ad-b8c1-6dbe36ccab7a\") " pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh"
Jul 28 04:49:20.920802 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:20.920687    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-workload-certs\") pod \"e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh\" (UID: \"81b65161-13f4-49ad-b8c1-6dbe36ccab7a\") " pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh"
Jul 28 04:49:20.920802 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:20.920717    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-istio-podinfo\") pod \"e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh\" (UID: \"81b65161-13f4-49ad-b8c1-6dbe36ccab7a\") " pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh"
Jul 28 04:49:20.920802 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:20.920760    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq8vn\" (UniqueName: \"kubernetes.io/projected/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-kube-api-access-sq8vn\") pod \"e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh\" (UID: \"81b65161-13f4-49ad-b8c1-6dbe36ccab7a\") " pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh"
Jul 28 04:49:20.921112 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:20.920824    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-workload-socket\") pod \"e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh\" (UID: \"81b65161-13f4-49ad-b8c1-6dbe36ccab7a\") " pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh"
Jul 28 04:49:20.921112 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:20.920917    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-istio-data\") pod \"e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh\" (UID: \"81b65161-13f4-49ad-b8c1-6dbe36ccab7a\") " pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh"
Jul 28 04:49:20.921112 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:20.920967    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-credential-socket\") pod \"e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh\" (UID: \"81b65161-13f4-49ad-b8c1-6dbe36ccab7a\") " pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh"
Jul 28 04:49:20.921282 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:20.921225    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-workload-socket\") pod \"e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh\" (UID: \"81b65161-13f4-49ad-b8c1-6dbe36ccab7a\") " pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh"
Jul 28 04:49:20.921348 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:20.921329    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-credential-socket\") pod \"e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh\" (UID: \"81b65161-13f4-49ad-b8c1-6dbe36ccab7a\") " pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh"
Jul 28 04:49:20.921395 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:20.921357    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-istio-data\") pod \"e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh\" (UID: \"81b65161-13f4-49ad-b8c1-6dbe36ccab7a\") " pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh"
Jul 28 04:49:20.921431 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:20.921401    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-workload-certs\") pod \"e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh\" (UID: \"81b65161-13f4-49ad-b8c1-6dbe36ccab7a\") " pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh"
Jul 28 04:49:20.921546 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:20.921524    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-istiod-ca-cert\") pod \"e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh\" (UID: \"81b65161-13f4-49ad-b8c1-6dbe36ccab7a\") " pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh"
Jul 28 04:49:20.923351 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:20.923330    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-istio-podinfo\") pod \"e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh\" (UID: \"81b65161-13f4-49ad-b8c1-6dbe36ccab7a\") " pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh"
Jul 28 04:49:20.923458 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:20.923395    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-istio-envoy\") pod \"e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh\" (UID: \"81b65161-13f4-49ad-b8c1-6dbe36ccab7a\") " pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh"
Jul 28 04:49:20.928765 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:20.928747    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-istio-token\") pod \"e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh\" (UID: \"81b65161-13f4-49ad-b8c1-6dbe36ccab7a\") " pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh"
Jul 28 04:49:20.929189 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:20.929169    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq8vn\" (UniqueName: \"kubernetes.io/projected/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-kube-api-access-sq8vn\") pod \"e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh\" (UID: \"81b65161-13f4-49ad-b8c1-6dbe36ccab7a\") " pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh"
Jul 28 04:49:21.081810 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:21.081710    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh"
Jul 28 04:49:21.219993 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:21.219965    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh"]
Jul 28 04:49:21.221288 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:49:21.221259    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81b65161_13f4_49ad_b8c1_6dbe36ccab7a.slice/crio-051f7b904f530239b46bffd196fa9f1abd88ae139192e0748cda7563d635fbe3 WatchSource:0}: Error finding container 051f7b904f530239b46bffd196fa9f1abd88ae139192e0748cda7563d635fbe3: Status 404 returned error can't find the container with id 051f7b904f530239b46bffd196fa9f1abd88ae139192e0748cda7563d635fbe3
Jul 28 04:49:21.223449 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:21.223419    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:49:21.223562 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:21.223500    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:49:21.223562 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:21.223544    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:49:21.477260 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:21.477174    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh" event={"ID":"81b65161-13f4-49ad-b8c1-6dbe36ccab7a","Type":"ContainerStarted","Data":"1d95eaf9590db9417f3f91c025837a246a38e9081a30242dedcc2b4ae30a9abe"}
Jul 28 04:49:21.477260 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:21.477237    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh" event={"ID":"81b65161-13f4-49ad-b8c1-6dbe36ccab7a","Type":"ContainerStarted","Data":"051f7b904f530239b46bffd196fa9f1abd88ae139192e0748cda7563d635fbe3"}
Jul 28 04:49:21.498706 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:21.498654    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh" podStartSLOduration=1.4986394760000001 podStartE2EDuration="1.498639476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:49:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:49:21.496672112 +0000 UTC m=+2261.886848127" watchObservedRunningTime="2026-07-28 04:49:21.498639476 +0000 UTC m=+2261.888815496"
Jul 28 04:49:22.088945 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:22.088919    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh"
Jul 28 04:49:22.089294 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:22.089009    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh"
Jul 28 04:49:22.481876 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:22.481772    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh"
Jul 28 04:49:22.482655 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:22.482637    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh"
Jul 28 04:49:23.669348 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:23.669291    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-890677f3-579d87745f-s5k6c"]
Jul 28 04:49:23.669949 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:23.669926    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c3cbb68-4002-4255-84e4-b83fca6162e8" containerName="istio-proxy"
Jul 28 04:49:23.669949 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:23.669951    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c3cbb68-4002-4255-84e4-b83fca6162e8" containerName="istio-proxy"
Jul 28 04:49:23.673742 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:23.673723    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-890677f3-579d87745f-s5k6c"
Jul 28 04:49:23.676572 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:23.676509    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-890677f3-serving-cert\""
Jul 28 04:49:23.676572 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:23.676528    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gfn77\""
Jul 28 04:49:23.676820 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:23.676803    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 04:49:23.677775 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:23.677752    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 04:49:23.680884 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:23.680862    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-890677f3-579d87745f-s5k6c"]
Jul 28 04:49:23.750197 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:23.750163    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbpc7\" (UniqueName: \"kubernetes.io/projected/b96ba11a-88ff-49c1-8eaf-c0463ace7ac5-kube-api-access-hbpc7\") pod \"maas-api-e2e-mt-890677f3-579d87745f-s5k6c\" (UID: \"b96ba11a-88ff-49c1-8eaf-c0463ace7ac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-890677f3-579d87745f-s5k6c"
Jul 28 04:49:23.750370 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:23.750220    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b96ba11a-88ff-49c1-8eaf-c0463ace7ac5-openshift-service-ca\") pod \"maas-api-e2e-mt-890677f3-579d87745f-s5k6c\" (UID: \"b96ba11a-88ff-49c1-8eaf-c0463ace7ac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-890677f3-579d87745f-s5k6c"
Jul 28 04:49:23.750370 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:23.750283    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b96ba11a-88ff-49c1-8eaf-c0463ace7ac5-maas-api-tls\") pod \"maas-api-e2e-mt-890677f3-579d87745f-s5k6c\" (UID: \"b96ba11a-88ff-49c1-8eaf-c0463ace7ac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-890677f3-579d87745f-s5k6c"
Jul 28 04:49:23.851542 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:23.851508    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbpc7\" (UniqueName: \"kubernetes.io/projected/b96ba11a-88ff-49c1-8eaf-c0463ace7ac5-kube-api-access-hbpc7\") pod \"maas-api-e2e-mt-890677f3-579d87745f-s5k6c\" (UID: \"b96ba11a-88ff-49c1-8eaf-c0463ace7ac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-890677f3-579d87745f-s5k6c"
Jul 28 04:49:23.851744 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:23.851564    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b96ba11a-88ff-49c1-8eaf-c0463ace7ac5-openshift-service-ca\") pod \"maas-api-e2e-mt-890677f3-579d87745f-s5k6c\" (UID: \"b96ba11a-88ff-49c1-8eaf-c0463ace7ac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-890677f3-579d87745f-s5k6c"
Jul 28 04:49:23.851744 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:23.851587    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b96ba11a-88ff-49c1-8eaf-c0463ace7ac5-maas-api-tls\") pod \"maas-api-e2e-mt-890677f3-579d87745f-s5k6c\" (UID: \"b96ba11a-88ff-49c1-8eaf-c0463ace7ac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-890677f3-579d87745f-s5k6c"
Jul 28 04:49:23.852312 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:23.852289    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b96ba11a-88ff-49c1-8eaf-c0463ace7ac5-openshift-service-ca\") pod \"maas-api-e2e-mt-890677f3-579d87745f-s5k6c\" (UID: \"b96ba11a-88ff-49c1-8eaf-c0463ace7ac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-890677f3-579d87745f-s5k6c"
Jul 28 04:49:23.854071 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:23.854052    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b96ba11a-88ff-49c1-8eaf-c0463ace7ac5-maas-api-tls\") pod \"maas-api-e2e-mt-890677f3-579d87745f-s5k6c\" (UID: \"b96ba11a-88ff-49c1-8eaf-c0463ace7ac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-890677f3-579d87745f-s5k6c"
Jul 28 04:49:23.861021 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:23.860991    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbpc7\" (UniqueName: \"kubernetes.io/projected/b96ba11a-88ff-49c1-8eaf-c0463ace7ac5-kube-api-access-hbpc7\") pod \"maas-api-e2e-mt-890677f3-579d87745f-s5k6c\" (UID: \"b96ba11a-88ff-49c1-8eaf-c0463ace7ac5\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-890677f3-579d87745f-s5k6c"
Jul 28 04:49:23.987053 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:23.986976    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-890677f3-579d87745f-s5k6c"
Jul 28 04:49:24.122408 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:24.122380    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-890677f3-579d87745f-s5k6c"]
Jul 28 04:49:24.124375 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:49:24.124334    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb96ba11a_88ff_49c1_8eaf_c0463ace7ac5.slice/crio-8d2ffaec88133649d430b8407c81887bebc869c0078c1de73424beb0e5bcf575 WatchSource:0}: Error finding container 8d2ffaec88133649d430b8407c81887bebc869c0078c1de73424beb0e5bcf575: Status 404 returned error can't find the container with id 8d2ffaec88133649d430b8407c81887bebc869c0078c1de73424beb0e5bcf575
Jul 28 04:49:24.127661 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:24.127521    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 04:49:24.351508 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:24.351473    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-890677f3-794b65987b-k9ndm"]
Jul 28 04:49:24.356308 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:24.356287    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-890677f3-794b65987b-k9ndm"
Jul 28 04:49:24.359101 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:24.359081    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-890677f3\""
Jul 28 04:49:24.365626 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:24.365600    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-890677f3-794b65987b-k9ndm"]
Jul 28 04:49:24.387942 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:24.387915    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-890677f3-787d64964b-894m7"]
Jul 28 04:49:24.391282 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:24.391267    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-890677f3-787d64964b-894m7"
Jul 28 04:49:24.394000 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:24.393977    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-890677f3-dockercfg-vq2vl\""
Jul 28 04:49:24.399794 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:24.399753    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-890677f3-787d64964b-894m7"]
Jul 28 04:49:24.457971 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:24.457935    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85f1753b-ab48-427f-9066-14e141955bec-plugins-config-volume\") pod \"payload-processing-e2e-mt-890677f3-787d64964b-894m7\" (UID: \"85f1753b-ab48-427f-9066-14e141955bec\") " pod="openshift-ingress/payload-processing-e2e-mt-890677f3-787d64964b-894m7"
Jul 28 04:49:24.458151 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:24.458015    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/336d91ce-9e94-4706-a454-5198148b416e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-890677f3-794b65987b-k9ndm\" (UID: \"336d91ce-9e94-4706-a454-5198148b416e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-890677f3-794b65987b-k9ndm"
Jul 28 04:49:24.458151 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:24.458042    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc7wn\" (UniqueName: \"kubernetes.io/projected/85f1753b-ab48-427f-9066-14e141955bec-kube-api-access-fc7wn\") pod \"payload-processing-e2e-mt-890677f3-787d64964b-894m7\" (UID: \"85f1753b-ab48-427f-9066-14e141955bec\") " pod="openshift-ingress/payload-processing-e2e-mt-890677f3-787d64964b-894m7"
Jul 28 04:49:24.458151 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:24.458105    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckshf\" (UniqueName: \"kubernetes.io/projected/336d91ce-9e94-4706-a454-5198148b416e-kube-api-access-ckshf\") pod \"payload-pre-processing-e2e-mt-890677f3-794b65987b-k9ndm\" (UID: \"336d91ce-9e94-4706-a454-5198148b416e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-890677f3-794b65987b-k9ndm"
Jul 28 04:49:24.493512 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:24.493478    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-890677f3-579d87745f-s5k6c" event={"ID":"b96ba11a-88ff-49c1-8eaf-c0463ace7ac5","Type":"ContainerStarted","Data":"8d2ffaec88133649d430b8407c81887bebc869c0078c1de73424beb0e5bcf575"}
Jul 28 04:49:24.559235 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:24.559210    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85f1753b-ab48-427f-9066-14e141955bec-plugins-config-volume\") pod \"payload-processing-e2e-mt-890677f3-787d64964b-894m7\" (UID: \"85f1753b-ab48-427f-9066-14e141955bec\") " pod="openshift-ingress/payload-processing-e2e-mt-890677f3-787d64964b-894m7"
Jul 28 04:49:24.559350 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:24.559270    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/336d91ce-9e94-4706-a454-5198148b416e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-890677f3-794b65987b-k9ndm\" (UID: \"336d91ce-9e94-4706-a454-5198148b416e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-890677f3-794b65987b-k9ndm"
Jul 28 04:49:24.559350 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:24.559292    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc7wn\" (UniqueName: \"kubernetes.io/projected/85f1753b-ab48-427f-9066-14e141955bec-kube-api-access-fc7wn\") pod \"payload-processing-e2e-mt-890677f3-787d64964b-894m7\" (UID: \"85f1753b-ab48-427f-9066-14e141955bec\") " pod="openshift-ingress/payload-processing-e2e-mt-890677f3-787d64964b-894m7"
Jul 28 04:49:24.559350 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:24.559315    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckshf\" (UniqueName: \"kubernetes.io/projected/336d91ce-9e94-4706-a454-5198148b416e-kube-api-access-ckshf\") pod \"payload-pre-processing-e2e-mt-890677f3-794b65987b-k9ndm\" (UID: \"336d91ce-9e94-4706-a454-5198148b416e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-890677f3-794b65987b-k9ndm"
Jul 28 04:49:24.560019 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:24.559994    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85f1753b-ab48-427f-9066-14e141955bec-plugins-config-volume\") pod \"payload-processing-e2e-mt-890677f3-787d64964b-894m7\" (UID: \"85f1753b-ab48-427f-9066-14e141955bec\") " pod="openshift-ingress/payload-processing-e2e-mt-890677f3-787d64964b-894m7"
Jul 28 04:49:24.560128 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:24.560109    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/336d91ce-9e94-4706-a454-5198148b416e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-890677f3-794b65987b-k9ndm\" (UID: \"336d91ce-9e94-4706-a454-5198148b416e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-890677f3-794b65987b-k9ndm"
Jul 28 04:49:24.567987 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:24.567969    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc7wn\" (UniqueName: \"kubernetes.io/projected/85f1753b-ab48-427f-9066-14e141955bec-kube-api-access-fc7wn\") pod \"payload-processing-e2e-mt-890677f3-787d64964b-894m7\" (UID: \"85f1753b-ab48-427f-9066-14e141955bec\") " pod="openshift-ingress/payload-processing-e2e-mt-890677f3-787d64964b-894m7"
Jul 28 04:49:24.568096 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:24.568080    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckshf\" (UniqueName: \"kubernetes.io/projected/336d91ce-9e94-4706-a454-5198148b416e-kube-api-access-ckshf\") pod \"payload-pre-processing-e2e-mt-890677f3-794b65987b-k9ndm\" (UID: \"336d91ce-9e94-4706-a454-5198148b416e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-890677f3-794b65987b-k9ndm"
Jul 28 04:49:24.668174 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:24.668090    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-890677f3-794b65987b-k9ndm"
Jul 28 04:49:24.702207 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:24.702161    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-890677f3-787d64964b-894m7"
Jul 28 04:49:24.818940 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:24.818913    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-890677f3-794b65987b-k9ndm"]
Jul 28 04:49:24.821069 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:49:24.821027    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod336d91ce_9e94_4706_a454_5198148b416e.slice/crio-31bc3abf0e6691ce8c910b446ebd97513264e3f1ae60fdc86d9674429a3500bd WatchSource:0}: Error finding container 31bc3abf0e6691ce8c910b446ebd97513264e3f1ae60fdc86d9674429a3500bd: Status 404 returned error can't find the container with id 31bc3abf0e6691ce8c910b446ebd97513264e3f1ae60fdc86d9674429a3500bd
Jul 28 04:49:25.060322 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:25.060294    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-890677f3-787d64964b-894m7"]
Jul 28 04:49:25.061851 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:49:25.061806    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85f1753b_ab48_427f_9066_14e141955bec.slice/crio-9b749b4fe2d2e5acaa525f03eb6e8b63e17827ef70aff75c594c1462419c6f5c WatchSource:0}: Error finding container 9b749b4fe2d2e5acaa525f03eb6e8b63e17827ef70aff75c594c1462419c6f5c: Status 404 returned error can't find the container with id 9b749b4fe2d2e5acaa525f03eb6e8b63e17827ef70aff75c594c1462419c6f5c
Jul 28 04:49:25.500337 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:25.500247    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-890677f3-787d64964b-894m7" event={"ID":"85f1753b-ab48-427f-9066-14e141955bec","Type":"ContainerStarted","Data":"f732a76151c453db8ed6da1c557e135868a4d4140b24b6b28da224a8d600eaa5"}
Jul 28 04:49:25.500337 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:25.500313    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-890677f3-787d64964b-894m7" event={"ID":"85f1753b-ab48-427f-9066-14e141955bec","Type":"ContainerStarted","Data":"9b749b4fe2d2e5acaa525f03eb6e8b63e17827ef70aff75c594c1462419c6f5c"}
Jul 28 04:49:25.500536 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:25.500340    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-890677f3-787d64964b-894m7"
Jul 28 04:49:25.501920 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:25.501865    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-890677f3-794b65987b-k9ndm" event={"ID":"336d91ce-9e94-4706-a454-5198148b416e","Type":"ContainerStarted","Data":"b8c3a022b0bf464c325bd15544c51e4c87180b37d9ad9381be9c18ad3e1c7196"}
Jul 28 04:49:25.501920 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:25.501919    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-890677f3-794b65987b-k9ndm"
Jul 28 04:49:25.502076 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:25.501944    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-890677f3-794b65987b-k9ndm" event={"ID":"336d91ce-9e94-4706-a454-5198148b416e","Type":"ContainerStarted","Data":"31bc3abf0e6691ce8c910b446ebd97513264e3f1ae60fdc86d9674429a3500bd"}
Jul 28 04:49:25.503363 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:25.503336    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-890677f3-579d87745f-s5k6c" event={"ID":"b96ba11a-88ff-49c1-8eaf-c0463ace7ac5","Type":"ContainerStarted","Data":"4c39d7c2e4036aea36040f16b4fc3b92e0a154e4916b7a12f6ec9dd8a3262a2b"}
Jul 28 04:49:25.503482 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:25.503385    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-890677f3-579d87745f-s5k6c"
Jul 28 04:49:25.514905 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:25.514454    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-890677f3-787d64964b-894m7" podStartSLOduration=1.514438204 podStartE2EDuration="1.514438204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:49:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:49:25.514087363 +0000 UTC m=+2265.904263384" watchObservedRunningTime="2026-07-28 04:49:25.514438204 +0000 UTC m=+2265.904614226"
Jul 28 04:49:25.532330 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:25.532282    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-890677f3-794b65987b-k9ndm" podStartSLOduration=1.5322708889999999 podStartE2EDuration="1.532270889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:49:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:49:25.530705078 +0000 UTC m=+2265.920881110" watchObservedRunningTime="2026-07-28 04:49:25.532270889 +0000 UTC m=+2265.922446910"
Jul 28 04:49:25.546427 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:25.546363    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-890677f3-579d87745f-s5k6c" podStartSLOduration=2.138529649 podStartE2EDuration="2.546350177s" totalImagesPullingTime="407.820528ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:49:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:49:25.545257872 +0000 UTC m=+2265.935433893" watchObservedRunningTime="2026-07-28 04:49:25.546350177 +0000 UTC m=+2265.936526198"
Jul 28 04:49:31.522191 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:31.522155    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-890677f3-579d87745f-s5k6c"
Jul 28 04:49:36.510423 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:36.510392    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-890677f3-794b65987b-k9ndm"
Jul 28 04:49:36.510946 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:36.510661    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-890677f3-787d64964b-894m7"
Jul 28 04:49:38.051518 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:38.050186    2566 patch_prober.go:28] interesting pod/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": read tcp 10.132.0.2:39290->10.132.0.66:15021: read: connection reset by peer" start-of-body=
Jul 28 04:49:38.051518 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:38.050268    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh" podUID="81b65161-13f4-49ad-b8c1-6dbe36ccab7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": read tcp 10.132.0.2:39290->10.132.0.66:15021: read: connection reset by peer"
Jul 28 04:49:38.575429 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:38.575396    2566 generic.go:356] "Generic (PLEG): container finished" podID="81b65161-13f4-49ad-b8c1-6dbe36ccab7a" containerID="1d95eaf9590db9417f3f91c025837a246a38e9081a30242dedcc2b4ae30a9abe" exitCode=0
Jul 28 04:49:38.575600 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:38.575451    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh" event={"ID":"81b65161-13f4-49ad-b8c1-6dbe36ccab7a","Type":"ContainerDied","Data":"1d95eaf9590db9417f3f91c025837a246a38e9081a30242dedcc2b4ae30a9abe"}
Jul 28 04:49:38.575882 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:38.575864    2566 scope.go:117] "RemoveContainer" containerID="1d95eaf9590db9417f3f91c025837a246a38e9081a30242dedcc2b4ae30a9abe"
Jul 28 04:49:38.576900 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:38.576867    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:49:38.577010 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:38.576942    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:49:38.577010 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:38.576976    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:49:39.582189 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:39.582154    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh" event={"ID":"81b65161-13f4-49ad-b8c1-6dbe36ccab7a","Type":"ContainerStarted","Data":"c16cdf1dfd6abff988273d3b320ff811f880503521fd6bdc7e29c790c56abfa9"}
Jul 28 04:49:40.082613 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:40.082575    2566 patch_prober.go:28] interesting pod/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Jul 28 04:49:40.082804 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:40.082630    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh" podUID="81b65161-13f4-49ad-b8c1-6dbe36ccab7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Jul 28 04:49:40.084624 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:40.084602    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh"
Jul 28 04:49:41.082742 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:41.082701    2566 patch_prober.go:28] interesting pod/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Jul 28 04:49:41.083263 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:41.082777    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh" podUID="81b65161-13f4-49ad-b8c1-6dbe36ccab7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Jul 28 04:49:42.082926 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:42.082885    2566 patch_prober.go:28] interesting pod/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Jul 28 04:49:42.083340 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:42.082943    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh" podUID="81b65161-13f4-49ad-b8c1-6dbe36ccab7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Jul 28 04:49:43.082329 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:43.082292    2566 patch_prober.go:28] interesting pod/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Jul 28 04:49:43.082506 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:43.082348    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh" podUID="81b65161-13f4-49ad-b8c1-6dbe36ccab7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Jul 28 04:49:44.088628 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:44.088601    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh"
Jul 28 04:49:44.089016 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:44.088861    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh"
Jul 28 04:49:44.089483 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:49:44.089467    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh"
Jul 28 04:50:23.341639 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.341595    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-890677f3-579d87745f-s5k6c"]
Jul 28 04:50:23.342284 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.341915    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-890677f3-579d87745f-s5k6c" podUID="b96ba11a-88ff-49c1-8eaf-c0463ace7ac5" containerName="maas-api" containerID="cri-o://4c39d7c2e4036aea36040f16b4fc3b92e0a154e4916b7a12f6ec9dd8a3262a2b" gracePeriod=30
Jul 28 04:50:23.398102 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.397756    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-890677f3-787d64964b-894m7"]
Jul 28 04:50:23.398578 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.398543    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-890677f3-787d64964b-894m7" podUID="85f1753b-ab48-427f-9066-14e141955bec" containerName="payload-processing" containerID="cri-o://f732a76151c453db8ed6da1c557e135868a4d4140b24b6b28da224a8d600eaa5" gracePeriod=30
Jul 28 04:50:23.406312 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.406280    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-890677f3-794b65987b-k9ndm"]
Jul 28 04:50:23.406859 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.406535    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-890677f3-794b65987b-k9ndm" podUID="336d91ce-9e94-4706-a454-5198148b416e" containerName="payload-pre-processing" containerID="cri-o://b8c3a022b0bf464c325bd15544c51e4c87180b37d9ad9381be9c18ad3e1c7196" gracePeriod=30
Jul 28 04:50:23.653180 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.653137    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-890677f3-579d87745f-s5k6c"
Jul 28 04:50:23.691645 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.691614    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-890677f3-787d64964b-894m7"
Jul 28 04:50:23.720795 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.720759    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-890677f3-794b65987b-k9ndm"
Jul 28 04:50:23.760624 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.760593    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b96ba11a-88ff-49c1-8eaf-c0463ace7ac5-maas-api-tls\") pod \"b96ba11a-88ff-49c1-8eaf-c0463ace7ac5\" (UID: \"b96ba11a-88ff-49c1-8eaf-c0463ace7ac5\") "
Jul 28 04:50:23.760787 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.760637    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fc7wn\" (UniqueName: \"kubernetes.io/projected/85f1753b-ab48-427f-9066-14e141955bec-kube-api-access-fc7wn\") pod \"85f1753b-ab48-427f-9066-14e141955bec\" (UID: \"85f1753b-ab48-427f-9066-14e141955bec\") "
Jul 28 04:50:23.760787 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.760664    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b96ba11a-88ff-49c1-8eaf-c0463ace7ac5-openshift-service-ca\") pod \"b96ba11a-88ff-49c1-8eaf-c0463ace7ac5\" (UID: \"b96ba11a-88ff-49c1-8eaf-c0463ace7ac5\") "
Jul 28 04:50:23.760787 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.760710    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85f1753b-ab48-427f-9066-14e141955bec-plugins-config-volume\") pod \"85f1753b-ab48-427f-9066-14e141955bec\" (UID: \"85f1753b-ab48-427f-9066-14e141955bec\") "
Jul 28 04:50:23.760787 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.760743    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbpc7\" (UniqueName: \"kubernetes.io/projected/b96ba11a-88ff-49c1-8eaf-c0463ace7ac5-kube-api-access-hbpc7\") pod \"b96ba11a-88ff-49c1-8eaf-c0463ace7ac5\" (UID: \"b96ba11a-88ff-49c1-8eaf-c0463ace7ac5\") "
Jul 28 04:50:23.761049 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.760800    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/336d91ce-9e94-4706-a454-5198148b416e-plugins-config-volume\") pod \"336d91ce-9e94-4706-a454-5198148b416e\" (UID: \"336d91ce-9e94-4706-a454-5198148b416e\") "
Jul 28 04:50:23.761158 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.761131    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b96ba11a-88ff-49c1-8eaf-c0463ace7ac5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b96ba11a-88ff-49c1-8eaf-c0463ace7ac5" (UID: "b96ba11a-88ff-49c1-8eaf-c0463ace7ac5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 04:50:23.761229 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.761190    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ckshf\" (UniqueName: \"kubernetes.io/projected/336d91ce-9e94-4706-a454-5198148b416e-kube-api-access-ckshf\") pod \"336d91ce-9e94-4706-a454-5198148b416e\" (UID: \"336d91ce-9e94-4706-a454-5198148b416e\") "
Jul 28 04:50:23.761336 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.761252    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85f1753b-ab48-427f-9066-14e141955bec-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "85f1753b-ab48-427f-9066-14e141955bec" (UID: "85f1753b-ab48-427f-9066-14e141955bec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 04:50:23.761336 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.761294    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/336d91ce-9e94-4706-a454-5198148b416e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "336d91ce-9e94-4706-a454-5198148b416e" (UID: "336d91ce-9e94-4706-a454-5198148b416e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 04:50:23.761546 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.761527    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b96ba11a-88ff-49c1-8eaf-c0463ace7ac5-openshift-service-ca\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:50:23.761608 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.761551    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85f1753b-ab48-427f-9066-14e141955bec-plugins-config-volume\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:50:23.761608 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.761564    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/336d91ce-9e94-4706-a454-5198148b416e-plugins-config-volume\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:50:23.762924 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.762890    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b96ba11a-88ff-49c1-8eaf-c0463ace7ac5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b96ba11a-88ff-49c1-8eaf-c0463ace7ac5" (UID: "b96ba11a-88ff-49c1-8eaf-c0463ace7ac5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 04:50:23.763312 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.763277    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/336d91ce-9e94-4706-a454-5198148b416e-kube-api-access-ckshf" (OuterVolumeSpecName: "kube-api-access-ckshf") pod "336d91ce-9e94-4706-a454-5198148b416e" (UID: "336d91ce-9e94-4706-a454-5198148b416e"). InnerVolumeSpecName "kube-api-access-ckshf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:50:23.763385 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.763363    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85f1753b-ab48-427f-9066-14e141955bec-kube-api-access-fc7wn" (OuterVolumeSpecName: "kube-api-access-fc7wn") pod "85f1753b-ab48-427f-9066-14e141955bec" (UID: "85f1753b-ab48-427f-9066-14e141955bec"). InnerVolumeSpecName "kube-api-access-fc7wn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:50:23.763385 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.763361    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b96ba11a-88ff-49c1-8eaf-c0463ace7ac5-kube-api-access-hbpc7" (OuterVolumeSpecName: "kube-api-access-hbpc7") pod "b96ba11a-88ff-49c1-8eaf-c0463ace7ac5" (UID: "b96ba11a-88ff-49c1-8eaf-c0463ace7ac5"). InnerVolumeSpecName "kube-api-access-hbpc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:50:23.799276 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.799241    2566 generic.go:356] "Generic (PLEG): container finished" podID="b96ba11a-88ff-49c1-8eaf-c0463ace7ac5" containerID="4c39d7c2e4036aea36040f16b4fc3b92e0a154e4916b7a12f6ec9dd8a3262a2b" exitCode=0
Jul 28 04:50:23.799439 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.799291    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-890677f3-579d87745f-s5k6c"
Jul 28 04:50:23.799439 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.799331    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-890677f3-579d87745f-s5k6c" event={"ID":"b96ba11a-88ff-49c1-8eaf-c0463ace7ac5","Type":"ContainerDied","Data":"4c39d7c2e4036aea36040f16b4fc3b92e0a154e4916b7a12f6ec9dd8a3262a2b"}
Jul 28 04:50:23.799439 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.799387    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-890677f3-579d87745f-s5k6c" event={"ID":"b96ba11a-88ff-49c1-8eaf-c0463ace7ac5","Type":"ContainerDied","Data":"8d2ffaec88133649d430b8407c81887bebc869c0078c1de73424beb0e5bcf575"}
Jul 28 04:50:23.799439 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.799419    2566 scope.go:117] "RemoveContainer" containerID="4c39d7c2e4036aea36040f16b4fc3b92e0a154e4916b7a12f6ec9dd8a3262a2b"
Jul 28 04:50:23.800697 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.800673    2566 generic.go:356] "Generic (PLEG): container finished" podID="85f1753b-ab48-427f-9066-14e141955bec" containerID="f732a76151c453db8ed6da1c557e135868a4d4140b24b6b28da224a8d600eaa5" exitCode=0
Jul 28 04:50:23.800818 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.800705    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-890677f3-787d64964b-894m7" event={"ID":"85f1753b-ab48-427f-9066-14e141955bec","Type":"ContainerDied","Data":"f732a76151c453db8ed6da1c557e135868a4d4140b24b6b28da224a8d600eaa5"}
Jul 28 04:50:23.800818 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.800740    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-890677f3-787d64964b-894m7"
Jul 28 04:50:23.800818 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.800752    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-890677f3-787d64964b-894m7" event={"ID":"85f1753b-ab48-427f-9066-14e141955bec","Type":"ContainerDied","Data":"9b749b4fe2d2e5acaa525f03eb6e8b63e17827ef70aff75c594c1462419c6f5c"}
Jul 28 04:50:23.802314 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.802285    2566 generic.go:356] "Generic (PLEG): container finished" podID="336d91ce-9e94-4706-a454-5198148b416e" containerID="b8c3a022b0bf464c325bd15544c51e4c87180b37d9ad9381be9c18ad3e1c7196" exitCode=0
Jul 28 04:50:23.802314 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.802322    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-890677f3-794b65987b-k9ndm"
Jul 28 04:50:23.802541 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.802357    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-890677f3-794b65987b-k9ndm" event={"ID":"336d91ce-9e94-4706-a454-5198148b416e","Type":"ContainerDied","Data":"b8c3a022b0bf464c325bd15544c51e4c87180b37d9ad9381be9c18ad3e1c7196"}
Jul 28 04:50:23.802541 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.802411    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-890677f3-794b65987b-k9ndm" event={"ID":"336d91ce-9e94-4706-a454-5198148b416e","Type":"ContainerDied","Data":"31bc3abf0e6691ce8c910b446ebd97513264e3f1ae60fdc86d9674429a3500bd"}
Jul 28 04:50:23.810442 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.810420    2566 scope.go:117] "RemoveContainer" containerID="4c39d7c2e4036aea36040f16b4fc3b92e0a154e4916b7a12f6ec9dd8a3262a2b"
Jul 28 04:50:23.810794 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:50:23.810774    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c39d7c2e4036aea36040f16b4fc3b92e0a154e4916b7a12f6ec9dd8a3262a2b\": container with ID starting with 4c39d7c2e4036aea36040f16b4fc3b92e0a154e4916b7a12f6ec9dd8a3262a2b not found: ID does not exist" containerID="4c39d7c2e4036aea36040f16b4fc3b92e0a154e4916b7a12f6ec9dd8a3262a2b"
Jul 28 04:50:23.810879 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.810805    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c39d7c2e4036aea36040f16b4fc3b92e0a154e4916b7a12f6ec9dd8a3262a2b"} err="failed to get container status \"4c39d7c2e4036aea36040f16b4fc3b92e0a154e4916b7a12f6ec9dd8a3262a2b\": rpc error: code = NotFound desc = could not find container \"4c39d7c2e4036aea36040f16b4fc3b92e0a154e4916b7a12f6ec9dd8a3262a2b\": container with ID starting with 4c39d7c2e4036aea36040f16b4fc3b92e0a154e4916b7a12f6ec9dd8a3262a2b not found: ID does not exist"
Jul 28 04:50:23.810879 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.810823    2566 scope.go:117] "RemoveContainer" containerID="f732a76151c453db8ed6da1c557e135868a4d4140b24b6b28da224a8d600eaa5"
Jul 28 04:50:23.819176 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.819159    2566 scope.go:117] "RemoveContainer" containerID="f732a76151c453db8ed6da1c557e135868a4d4140b24b6b28da224a8d600eaa5"
Jul 28 04:50:23.819444 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:50:23.819423    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f732a76151c453db8ed6da1c557e135868a4d4140b24b6b28da224a8d600eaa5\": container with ID starting with f732a76151c453db8ed6da1c557e135868a4d4140b24b6b28da224a8d600eaa5 not found: ID does not exist" containerID="f732a76151c453db8ed6da1c557e135868a4d4140b24b6b28da224a8d600eaa5"
Jul 28 04:50:23.819488 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.819451    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f732a76151c453db8ed6da1c557e135868a4d4140b24b6b28da224a8d600eaa5"} err="failed to get container status \"f732a76151c453db8ed6da1c557e135868a4d4140b24b6b28da224a8d600eaa5\": rpc error: code = NotFound desc = could not find container \"f732a76151c453db8ed6da1c557e135868a4d4140b24b6b28da224a8d600eaa5\": container with ID starting with f732a76151c453db8ed6da1c557e135868a4d4140b24b6b28da224a8d600eaa5 not found: ID does not exist"
Jul 28 04:50:23.819488 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.819470    2566 scope.go:117] "RemoveContainer" containerID="b8c3a022b0bf464c325bd15544c51e4c87180b37d9ad9381be9c18ad3e1c7196"
Jul 28 04:50:23.825540 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.825521    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-890677f3-579d87745f-s5k6c"]
Jul 28 04:50:23.834727 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.829959    2566 scope.go:117] "RemoveContainer" containerID="b8c3a022b0bf464c325bd15544c51e4c87180b37d9ad9381be9c18ad3e1c7196"
Jul 28 04:50:23.834727 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.831714    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-890677f3-579d87745f-s5k6c"]
Jul 28 04:50:23.835073 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:50:23.835050    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8c3a022b0bf464c325bd15544c51e4c87180b37d9ad9381be9c18ad3e1c7196\": container with ID starting with b8c3a022b0bf464c325bd15544c51e4c87180b37d9ad9381be9c18ad3e1c7196 not found: ID does not exist" containerID="b8c3a022b0bf464c325bd15544c51e4c87180b37d9ad9381be9c18ad3e1c7196"
Jul 28 04:50:23.835145 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.835081    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8c3a022b0bf464c325bd15544c51e4c87180b37d9ad9381be9c18ad3e1c7196"} err="failed to get container status \"b8c3a022b0bf464c325bd15544c51e4c87180b37d9ad9381be9c18ad3e1c7196\": rpc error: code = NotFound desc = could not find container \"b8c3a022b0bf464c325bd15544c51e4c87180b37d9ad9381be9c18ad3e1c7196\": container with ID starting with b8c3a022b0bf464c325bd15544c51e4c87180b37d9ad9381be9c18ad3e1c7196 not found: ID does not exist"
Jul 28 04:50:23.839773 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.839752    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-890677f3-787d64964b-894m7"]
Jul 28 04:50:23.841403 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.841384    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-890677f3-787d64964b-894m7"]
Jul 28 04:50:23.851440 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.851416    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-890677f3-794b65987b-k9ndm"]
Jul 28 04:50:23.853249 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.853232    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-890677f3-794b65987b-k9ndm"]
Jul 28 04:50:23.862919 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.862899    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbpc7\" (UniqueName: \"kubernetes.io/projected/b96ba11a-88ff-49c1-8eaf-c0463ace7ac5-kube-api-access-hbpc7\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:50:23.862919 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.862918    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckshf\" (UniqueName: \"kubernetes.io/projected/336d91ce-9e94-4706-a454-5198148b416e-kube-api-access-ckshf\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:50:23.863031 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.862927    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b96ba11a-88ff-49c1-8eaf-c0463ace7ac5-maas-api-tls\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:50:23.863031 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:23.862936    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fc7wn\" (UniqueName: \"kubernetes.io/projected/85f1753b-ab48-427f-9066-14e141955bec-kube-api-access-fc7wn\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:50:24.088481 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:24.088450    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="336d91ce-9e94-4706-a454-5198148b416e" path="/var/lib/kubelet/pods/336d91ce-9e94-4706-a454-5198148b416e/volumes"
Jul 28 04:50:24.088828 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:24.088815    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85f1753b-ab48-427f-9066-14e141955bec" path="/var/lib/kubelet/pods/85f1753b-ab48-427f-9066-14e141955bec/volumes"
Jul 28 04:50:24.089218 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:50:24.089197    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b96ba11a-88ff-49c1-8eaf-c0463ace7ac5" path="/var/lib/kubelet/pods/b96ba11a-88ff-49c1-8eaf-c0463ace7ac5/volumes"
Jul 28 04:51:18.125879 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:51:18.125826    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh"]
Jul 28 04:51:18.126501 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:51:18.126146    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh" podUID="81b65161-13f4-49ad-b8c1-6dbe36ccab7a" containerName="istio-proxy" containerID="cri-o://c16cdf1dfd6abff988273d3b320ff811f880503521fd6bdc7e29c790c56abfa9" gracePeriod=30
Jul 28 04:51:23.390024 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:51:23.389960    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh"
Jul 28 04:51:23.553578 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:51:23.553543    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-istiod-ca-cert\") pod \"81b65161-13f4-49ad-b8c1-6dbe36ccab7a\" (UID: \"81b65161-13f4-49ad-b8c1-6dbe36ccab7a\") "
Jul 28 04:51:23.553578 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:51:23.553591    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-istio-podinfo\") pod \"81b65161-13f4-49ad-b8c1-6dbe36ccab7a\" (UID: \"81b65161-13f4-49ad-b8c1-6dbe36ccab7a\") "
Jul 28 04:51:23.553879 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:51:23.553616    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-credential-socket\") pod \"81b65161-13f4-49ad-b8c1-6dbe36ccab7a\" (UID: \"81b65161-13f4-49ad-b8c1-6dbe36ccab7a\") "
Jul 28 04:51:23.553879 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:51:23.553647    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-istio-token\") pod \"81b65161-13f4-49ad-b8c1-6dbe36ccab7a\" (UID: \"81b65161-13f4-49ad-b8c1-6dbe36ccab7a\") "
Jul 28 04:51:23.553879 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:51:23.553689    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-workload-certs\") pod \"81b65161-13f4-49ad-b8c1-6dbe36ccab7a\" (UID: \"81b65161-13f4-49ad-b8c1-6dbe36ccab7a\") "
Jul 28 04:51:23.553879 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:51:23.553755    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-workload-socket\") pod \"81b65161-13f4-49ad-b8c1-6dbe36ccab7a\" (UID: \"81b65161-13f4-49ad-b8c1-6dbe36ccab7a\") "
Jul 28 04:51:23.553879 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:51:23.553804    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-istio-envoy\") pod \"81b65161-13f4-49ad-b8c1-6dbe36ccab7a\" (UID: \"81b65161-13f4-49ad-b8c1-6dbe36ccab7a\") "
Jul 28 04:51:23.553879 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:51:23.553871    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-istio-data\") pod \"81b65161-13f4-49ad-b8c1-6dbe36ccab7a\" (UID: \"81b65161-13f4-49ad-b8c1-6dbe36ccab7a\") "
Jul 28 04:51:23.554206 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:51:23.553931    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sq8vn\" (UniqueName: \"kubernetes.io/projected/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-kube-api-access-sq8vn\") pod \"81b65161-13f4-49ad-b8c1-6dbe36ccab7a\" (UID: \"81b65161-13f4-49ad-b8c1-6dbe36ccab7a\") "
Jul 28 04:51:23.554206 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:51:23.554022    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "81b65161-13f4-49ad-b8c1-6dbe36ccab7a" (UID: "81b65161-13f4-49ad-b8c1-6dbe36ccab7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:51:23.554206 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:51:23.554130    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "81b65161-13f4-49ad-b8c1-6dbe36ccab7a" (UID: "81b65161-13f4-49ad-b8c1-6dbe36ccab7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:51:23.554206 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:51:23.554144    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "81b65161-13f4-49ad-b8c1-6dbe36ccab7a" (UID: "81b65161-13f4-49ad-b8c1-6dbe36ccab7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:51:23.554206 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:51:23.554053    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "81b65161-13f4-49ad-b8c1-6dbe36ccab7a" (UID: "81b65161-13f4-49ad-b8c1-6dbe36ccab7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 04:51:23.554511 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:51:23.554342    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-workload-certs\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:51:23.554511 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:51:23.554361    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-workload-socket\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:51:23.554511 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:51:23.554374    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-istiod-ca-cert\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:51:23.554511 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:51:23.554382    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-credential-socket\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:51:23.554936 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:51:23.554914    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-istio-data" (OuterVolumeSpecName: "istio-data") pod "81b65161-13f4-49ad-b8c1-6dbe36ccab7a" (UID: "81b65161-13f4-49ad-b8c1-6dbe36ccab7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:51:23.555996 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:51:23.555967    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "81b65161-13f4-49ad-b8c1-6dbe36ccab7a" (UID: "81b65161-13f4-49ad-b8c1-6dbe36ccab7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:51:23.555996 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:51:23.555990    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "81b65161-13f4-49ad-b8c1-6dbe36ccab7a" (UID: "81b65161-13f4-49ad-b8c1-6dbe36ccab7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 04:51:23.556484 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:51:23.556467    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-istio-token" (OuterVolumeSpecName: "istio-token") pod "81b65161-13f4-49ad-b8c1-6dbe36ccab7a" (UID: "81b65161-13f4-49ad-b8c1-6dbe36ccab7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:51:23.556553 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:51:23.556490    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-kube-api-access-sq8vn" (OuterVolumeSpecName: "kube-api-access-sq8vn") pod "81b65161-13f4-49ad-b8c1-6dbe36ccab7a" (UID: "81b65161-13f4-49ad-b8c1-6dbe36ccab7a"). InnerVolumeSpecName "kube-api-access-sq8vn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:51:23.655466 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:51:23.655381    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-istio-podinfo\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:51:23.655466 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:51:23.655409    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-istio-token\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:51:23.655466 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:51:23.655419    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-istio-envoy\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:51:23.655466 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:51:23.655427    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-istio-data\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:51:23.655466 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:51:23.655434    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq8vn\" (UniqueName: \"kubernetes.io/projected/81b65161-13f4-49ad-b8c1-6dbe36ccab7a-kube-api-access-sq8vn\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:51:24.099186 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:51:24.099157    2566 generic.go:356] "Generic (PLEG): container finished" podID="81b65161-13f4-49ad-b8c1-6dbe36ccab7a" containerID="c16cdf1dfd6abff988273d3b320ff811f880503521fd6bdc7e29c790c56abfa9" exitCode=0
Jul 28 04:51:24.099322 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:51:24.099209    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh"
Jul 28 04:51:24.099322 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:51:24.099239    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh" event={"ID":"81b65161-13f4-49ad-b8c1-6dbe36ccab7a","Type":"ContainerDied","Data":"c16cdf1dfd6abff988273d3b320ff811f880503521fd6bdc7e29c790c56abfa9"}
Jul 28 04:51:24.099322 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:51:24.099304    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh" event={"ID":"81b65161-13f4-49ad-b8c1-6dbe36ccab7a","Type":"ContainerDied","Data":"051f7b904f530239b46bffd196fa9f1abd88ae139192e0748cda7563d635fbe3"}
Jul 28 04:51:24.099440 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:51:24.099331    2566 scope.go:117] "RemoveContainer" containerID="c16cdf1dfd6abff988273d3b320ff811f880503521fd6bdc7e29c790c56abfa9"
Jul 28 04:51:24.110052 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:51:24.109924    2566 scope.go:117] "RemoveContainer" containerID="1d95eaf9590db9417f3f91c025837a246a38e9081a30242dedcc2b4ae30a9abe"
Jul 28 04:51:24.118421 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:51:24.118401    2566 scope.go:117] "RemoveContainer" containerID="c16cdf1dfd6abff988273d3b320ff811f880503521fd6bdc7e29c790c56abfa9"
Jul 28 04:51:24.118724 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:51:24.118698    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c16cdf1dfd6abff988273d3b320ff811f880503521fd6bdc7e29c790c56abfa9\": container with ID starting with c16cdf1dfd6abff988273d3b320ff811f880503521fd6bdc7e29c790c56abfa9 not found: ID does not exist" containerID="c16cdf1dfd6abff988273d3b320ff811f880503521fd6bdc7e29c790c56abfa9"
Jul 28 04:51:24.118862 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:51:24.118729    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c16cdf1dfd6abff988273d3b320ff811f880503521fd6bdc7e29c790c56abfa9"} err="failed to get container status \"c16cdf1dfd6abff988273d3b320ff811f880503521fd6bdc7e29c790c56abfa9\": rpc error: code = NotFound desc = could not find container \"c16cdf1dfd6abff988273d3b320ff811f880503521fd6bdc7e29c790c56abfa9\": container with ID starting with c16cdf1dfd6abff988273d3b320ff811f880503521fd6bdc7e29c790c56abfa9 not found: ID does not exist"
Jul 28 04:51:24.118862 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:51:24.118748    2566 scope.go:117] "RemoveContainer" containerID="1d95eaf9590db9417f3f91c025837a246a38e9081a30242dedcc2b4ae30a9abe"
Jul 28 04:51:24.119027 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:51:24.119004    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d95eaf9590db9417f3f91c025837a246a38e9081a30242dedcc2b4ae30a9abe\": container with ID starting with 1d95eaf9590db9417f3f91c025837a246a38e9081a30242dedcc2b4ae30a9abe not found: ID does not exist" containerID="1d95eaf9590db9417f3f91c025837a246a38e9081a30242dedcc2b4ae30a9abe"
Jul 28 04:51:24.119076 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:51:24.119034    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d95eaf9590db9417f3f91c025837a246a38e9081a30242dedcc2b4ae30a9abe"} err="failed to get container status \"1d95eaf9590db9417f3f91c025837a246a38e9081a30242dedcc2b4ae30a9abe\": rpc error: code = NotFound desc = could not find container \"1d95eaf9590db9417f3f91c025837a246a38e9081a30242dedcc2b4ae30a9abe\": container with ID starting with 1d95eaf9590db9417f3f91c025837a246a38e9081a30242dedcc2b4ae30a9abe not found: ID does not exist"
Jul 28 04:51:24.121920 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:51:24.121898    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh"]
Jul 28 04:51:24.128174 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:51:24.128150    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-890677f3-openshift-default-ddcfb5984-mprsh"]
Jul 28 04:51:26.088370 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:51:26.088336    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81b65161-13f4-49ad-b8c1-6dbe36ccab7a" path="/var/lib/kubelet/pods/81b65161-13f4-49ad-b8c1-6dbe36ccab7a/volumes"
Jul 28 04:51:43.725457 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:51:43.725428    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4glpr_bfd2d3d8-f287-42d8-83c7-f82185c96735/console-operator/2.log"
Jul 28 04:51:43.730637 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:51:43.730615    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4glpr_bfd2d3d8-f287-42d8-83c7-f82185c96735/console-operator/2.log"
Jul 28 04:52:13.954527 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:13.954438    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588"]
Jul 28 04:52:13.955010 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:13.954989    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85f1753b-ab48-427f-9066-14e141955bec" containerName="payload-processing"
Jul 28 04:52:13.955010 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:13.955005    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="85f1753b-ab48-427f-9066-14e141955bec" containerName="payload-processing"
Jul 28 04:52:13.955086 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:13.955020    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b96ba11a-88ff-49c1-8eaf-c0463ace7ac5" containerName="maas-api"
Jul 28 04:52:13.955086 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:13.955029    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b96ba11a-88ff-49c1-8eaf-c0463ace7ac5" containerName="maas-api"
Jul 28 04:52:13.955086 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:13.955051    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="336d91ce-9e94-4706-a454-5198148b416e" containerName="payload-pre-processing"
Jul 28 04:52:13.955086 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:13.955060    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="336d91ce-9e94-4706-a454-5198148b416e" containerName="payload-pre-processing"
Jul 28 04:52:13.955086 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:13.955080    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81b65161-13f4-49ad-b8c1-6dbe36ccab7a" containerName="istio-proxy"
Jul 28 04:52:13.955241 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:13.955089    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="81b65161-13f4-49ad-b8c1-6dbe36ccab7a" containerName="istio-proxy"
Jul 28 04:52:13.955241 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:13.955099    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81b65161-13f4-49ad-b8c1-6dbe36ccab7a" containerName="istio-proxy"
Jul 28 04:52:13.955241 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:13.955108    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="81b65161-13f4-49ad-b8c1-6dbe36ccab7a" containerName="istio-proxy"
Jul 28 04:52:13.955241 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:13.955196    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b96ba11a-88ff-49c1-8eaf-c0463ace7ac5" containerName="maas-api"
Jul 28 04:52:13.955241 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:13.955208    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="336d91ce-9e94-4706-a454-5198148b416e" containerName="payload-pre-processing"
Jul 28 04:52:13.955241 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:13.955219    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="81b65161-13f4-49ad-b8c1-6dbe36ccab7a" containerName="istio-proxy"
Jul 28 04:52:13.955241 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:13.955230    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="85f1753b-ab48-427f-9066-14e141955bec" containerName="payload-processing"
Jul 28 04:52:13.958730 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:13.958709    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588"
Jul 28 04:52:13.961294 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:13.961270    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 28 04:52:13.961422 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:13.961315    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-20ba8c-openshift-default-dockercfg-jjb9w\""
Jul 28 04:52:13.968224 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:13.967581    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588"]
Jul 28 04:52:14.028976 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:14.028937    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dd74d22-d2b4-47ec-acac-5d60279c7c59-istio-token\") pod \"e2e-inf-a-20ba8c-openshift-default-68c89854-kx588\" (UID: \"2dd74d22-d2b4-47ec-acac-5d60279c7c59\") " pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588"
Jul 28 04:52:14.029175 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:14.028995    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dd74d22-d2b4-47ec-acac-5d60279c7c59-istiod-ca-cert\") pod \"e2e-inf-a-20ba8c-openshift-default-68c89854-kx588\" (UID: \"2dd74d22-d2b4-47ec-acac-5d60279c7c59\") " pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588"
Jul 28 04:52:14.029175 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:14.029110    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd74d22-d2b4-47ec-acac-5d60279c7c59-workload-socket\") pod \"e2e-inf-a-20ba8c-openshift-default-68c89854-kx588\" (UID: \"2dd74d22-d2b4-47ec-acac-5d60279c7c59\") " pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588"
Jul 28 04:52:14.029175 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:14.029149    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dd74d22-d2b4-47ec-acac-5d60279c7c59-istio-envoy\") pod \"e2e-inf-a-20ba8c-openshift-default-68c89854-kx588\" (UID: \"2dd74d22-d2b4-47ec-acac-5d60279c7c59\") " pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588"
Jul 28 04:52:14.029326 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:14.029176    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v797q\" (UniqueName: \"kubernetes.io/projected/2dd74d22-d2b4-47ec-acac-5d60279c7c59-kube-api-access-v797q\") pod \"e2e-inf-a-20ba8c-openshift-default-68c89854-kx588\" (UID: \"2dd74d22-d2b4-47ec-acac-5d60279c7c59\") " pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588"
Jul 28 04:52:14.029326 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:14.029233    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dd74d22-d2b4-47ec-acac-5d60279c7c59-istio-data\") pod \"e2e-inf-a-20ba8c-openshift-default-68c89854-kx588\" (UID: \"2dd74d22-d2b4-47ec-acac-5d60279c7c59\") " pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588"
Jul 28 04:52:14.029326 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:14.029293    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dd74d22-d2b4-47ec-acac-5d60279c7c59-workload-certs\") pod \"e2e-inf-a-20ba8c-openshift-default-68c89854-kx588\" (UID: \"2dd74d22-d2b4-47ec-acac-5d60279c7c59\") " pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588"
Jul 28 04:52:14.029326 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:14.029312    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dd74d22-d2b4-47ec-acac-5d60279c7c59-istio-podinfo\") pod \"e2e-inf-a-20ba8c-openshift-default-68c89854-kx588\" (UID: \"2dd74d22-d2b4-47ec-acac-5d60279c7c59\") " pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588"
Jul 28 04:52:14.029479 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:14.029338    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd74d22-d2b4-47ec-acac-5d60279c7c59-credential-socket\") pod \"e2e-inf-a-20ba8c-openshift-default-68c89854-kx588\" (UID: \"2dd74d22-d2b4-47ec-acac-5d60279c7c59\") " pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588"
Jul 28 04:52:14.130415 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:14.130373    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd74d22-d2b4-47ec-acac-5d60279c7c59-workload-socket\") pod \"e2e-inf-a-20ba8c-openshift-default-68c89854-kx588\" (UID: \"2dd74d22-d2b4-47ec-acac-5d60279c7c59\") " pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588"
Jul 28 04:52:14.130618 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:14.130422    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dd74d22-d2b4-47ec-acac-5d60279c7c59-istio-envoy\") pod \"e2e-inf-a-20ba8c-openshift-default-68c89854-kx588\" (UID: \"2dd74d22-d2b4-47ec-acac-5d60279c7c59\") " pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588"
Jul 28 04:52:14.130618 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:14.130454    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v797q\" (UniqueName: \"kubernetes.io/projected/2dd74d22-d2b4-47ec-acac-5d60279c7c59-kube-api-access-v797q\") pod \"e2e-inf-a-20ba8c-openshift-default-68c89854-kx588\" (UID: \"2dd74d22-d2b4-47ec-acac-5d60279c7c59\") " pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588"
Jul 28 04:52:14.130618 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:14.130588    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dd74d22-d2b4-47ec-acac-5d60279c7c59-istio-data\") pod \"e2e-inf-a-20ba8c-openshift-default-68c89854-kx588\" (UID: \"2dd74d22-d2b4-47ec-acac-5d60279c7c59\") " pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588"
Jul 28 04:52:14.130790 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:14.130683    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dd74d22-d2b4-47ec-acac-5d60279c7c59-workload-certs\") pod \"e2e-inf-a-20ba8c-openshift-default-68c89854-kx588\" (UID: \"2dd74d22-d2b4-47ec-acac-5d60279c7c59\") " pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588"
Jul 28 04:52:14.130790 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:14.130714    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dd74d22-d2b4-47ec-acac-5d60279c7c59-istio-podinfo\") pod \"e2e-inf-a-20ba8c-openshift-default-68c89854-kx588\" (UID: \"2dd74d22-d2b4-47ec-acac-5d60279c7c59\") " pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588"
Jul 28 04:52:14.130790 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:14.130760    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd74d22-d2b4-47ec-acac-5d60279c7c59-credential-socket\") pod \"e2e-inf-a-20ba8c-openshift-default-68c89854-kx588\" (UID: \"2dd74d22-d2b4-47ec-acac-5d60279c7c59\") " pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588"
Jul 28 04:52:14.131006 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:14.130818    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dd74d22-d2b4-47ec-acac-5d60279c7c59-istio-token\") pod \"e2e-inf-a-20ba8c-openshift-default-68c89854-kx588\" (UID: \"2dd74d22-d2b4-47ec-acac-5d60279c7c59\") " pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588"
Jul 28 04:52:14.131006 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:14.130954    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd74d22-d2b4-47ec-acac-5d60279c7c59-workload-socket\") pod \"e2e-inf-a-20ba8c-openshift-default-68c89854-kx588\" (UID: \"2dd74d22-d2b4-47ec-acac-5d60279c7c59\") " pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588"
Jul 28 04:52:14.131119 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:14.131000    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dd74d22-d2b4-47ec-acac-5d60279c7c59-istio-data\") pod \"e2e-inf-a-20ba8c-openshift-default-68c89854-kx588\" (UID: \"2dd74d22-d2b4-47ec-acac-5d60279c7c59\") " pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588"
Jul 28 04:52:14.131119 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:14.131028    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dd74d22-d2b4-47ec-acac-5d60279c7c59-workload-certs\") pod \"e2e-inf-a-20ba8c-openshift-default-68c89854-kx588\" (UID: \"2dd74d22-d2b4-47ec-acac-5d60279c7c59\") " pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588"
Jul 28 04:52:14.131224 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:14.131154    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd74d22-d2b4-47ec-acac-5d60279c7c59-credential-socket\") pod \"e2e-inf-a-20ba8c-openshift-default-68c89854-kx588\" (UID: \"2dd74d22-d2b4-47ec-acac-5d60279c7c59\") " pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588"
Jul 28 04:52:14.131282 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:14.131234    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dd74d22-d2b4-47ec-acac-5d60279c7c59-istiod-ca-cert\") pod \"e2e-inf-a-20ba8c-openshift-default-68c89854-kx588\" (UID: \"2dd74d22-d2b4-47ec-acac-5d60279c7c59\") " pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588"
Jul 28 04:52:14.131881 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:14.131856    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dd74d22-d2b4-47ec-acac-5d60279c7c59-istiod-ca-cert\") pod \"e2e-inf-a-20ba8c-openshift-default-68c89854-kx588\" (UID: \"2dd74d22-d2b4-47ec-acac-5d60279c7c59\") " pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588"
Jul 28 04:52:14.132922 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:14.132903    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dd74d22-d2b4-47ec-acac-5d60279c7c59-istio-envoy\") pod \"e2e-inf-a-20ba8c-openshift-default-68c89854-kx588\" (UID: \"2dd74d22-d2b4-47ec-acac-5d60279c7c59\") " pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588"
Jul 28 04:52:14.133338 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:14.133314    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dd74d22-d2b4-47ec-acac-5d60279c7c59-istio-podinfo\") pod \"e2e-inf-a-20ba8c-openshift-default-68c89854-kx588\" (UID: \"2dd74d22-d2b4-47ec-acac-5d60279c7c59\") " pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588"
Jul 28 04:52:14.139589 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:14.139565    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v797q\" (UniqueName: \"kubernetes.io/projected/2dd74d22-d2b4-47ec-acac-5d60279c7c59-kube-api-access-v797q\") pod \"e2e-inf-a-20ba8c-openshift-default-68c89854-kx588\" (UID: \"2dd74d22-d2b4-47ec-acac-5d60279c7c59\") " pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588"
Jul 28 04:52:14.139589 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:14.139578    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dd74d22-d2b4-47ec-acac-5d60279c7c59-istio-token\") pod \"e2e-inf-a-20ba8c-openshift-default-68c89854-kx588\" (UID: \"2dd74d22-d2b4-47ec-acac-5d60279c7c59\") " pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588"
Jul 28 04:52:14.273227 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:14.273198    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588"
Jul 28 04:52:14.412011 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:14.411984    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588"]
Jul 28 04:52:14.413894 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:52:14.413868    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd74d22_d2b4_47ec_acac_5d60279c7c59.slice/crio-ea227a02c3e7df85d8973ba84764f69c4cf437b6158499a1e36a975c92679a4e WatchSource:0}: Error finding container ea227a02c3e7df85d8973ba84764f69c4cf437b6158499a1e36a975c92679a4e: Status 404 returned error can't find the container with id ea227a02c3e7df85d8973ba84764f69c4cf437b6158499a1e36a975c92679a4e
Jul 28 04:52:14.416140 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:14.416100    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:52:14.416214 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:14.416196    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:52:14.416260 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:14.416248    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:52:15.344811 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:15.344772    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588" event={"ID":"2dd74d22-d2b4-47ec-acac-5d60279c7c59","Type":"ContainerStarted","Data":"d30beb77beb43961d5a4ae6c025b99b2702bd69e2730ba444a28ea756c13197e"}
Jul 28 04:52:15.345223 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:15.344875    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588" event={"ID":"2dd74d22-d2b4-47ec-acac-5d60279c7c59","Type":"ContainerStarted","Data":"ea227a02c3e7df85d8973ba84764f69c4cf437b6158499a1e36a975c92679a4e"}
Jul 28 04:52:15.366462 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:15.366414    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588" podStartSLOduration=2.366401329 podStartE2EDuration="2.366401329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:52:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:52:15.363456281 +0000 UTC m=+2435.753632302" watchObservedRunningTime="2026-07-28 04:52:15.366401329 +0000 UTC m=+2435.756577351"
Jul 28 04:52:16.274334 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:16.274302    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588"
Jul 28 04:52:16.278901 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:16.278876    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588"
Jul 28 04:52:16.349766 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:16.349740    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588"
Jul 28 04:52:16.350707 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:16.350689    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588"
Jul 28 04:52:17.431293 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:17.431255    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-20ba8c-74d7cbfdd7-czd8x"]
Jul 28 04:52:17.431991 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:17.431972    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="81b65161-13f4-49ad-b8c1-6dbe36ccab7a" containerName="istio-proxy"
Jul 28 04:52:17.435243 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:17.435222    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-20ba8c-74d7cbfdd7-czd8x"
Jul 28 04:52:17.437784 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:17.437758    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-20ba8c-serving-cert\""
Jul 28 04:52:17.437976 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:17.437953    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 04:52:17.438133 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:17.438026    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gfn77\""
Jul 28 04:52:17.438955 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:17.438936    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 04:52:17.443472 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:17.443374    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-20ba8c-74d7cbfdd7-czd8x"]
Jul 28 04:52:17.562082 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:17.562048    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cd86c82-7f61-4828-b7c3-1be9fac77fc4-maas-api-tls\") pod \"maas-api-e2e-inf-a-20ba8c-74d7cbfdd7-czd8x\" (UID: \"3cd86c82-7f61-4828-b7c3-1be9fac77fc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-20ba8c-74d7cbfdd7-czd8x"
Jul 28 04:52:17.562278 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:17.562125    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftvww\" (UniqueName: \"kubernetes.io/projected/3cd86c82-7f61-4828-b7c3-1be9fac77fc4-kube-api-access-ftvww\") pod \"maas-api-e2e-inf-a-20ba8c-74d7cbfdd7-czd8x\" (UID: \"3cd86c82-7f61-4828-b7c3-1be9fac77fc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-20ba8c-74d7cbfdd7-czd8x"
Jul 28 04:52:17.562278 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:17.562155    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cd86c82-7f61-4828-b7c3-1be9fac77fc4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-20ba8c-74d7cbfdd7-czd8x\" (UID: \"3cd86c82-7f61-4828-b7c3-1be9fac77fc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-20ba8c-74d7cbfdd7-czd8x"
Jul 28 04:52:17.663148 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:17.663117    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftvww\" (UniqueName: \"kubernetes.io/projected/3cd86c82-7f61-4828-b7c3-1be9fac77fc4-kube-api-access-ftvww\") pod \"maas-api-e2e-inf-a-20ba8c-74d7cbfdd7-czd8x\" (UID: \"3cd86c82-7f61-4828-b7c3-1be9fac77fc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-20ba8c-74d7cbfdd7-czd8x"
Jul 28 04:52:17.663338 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:17.663164    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cd86c82-7f61-4828-b7c3-1be9fac77fc4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-20ba8c-74d7cbfdd7-czd8x\" (UID: \"3cd86c82-7f61-4828-b7c3-1be9fac77fc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-20ba8c-74d7cbfdd7-czd8x"
Jul 28 04:52:17.663338 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:17.663208    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cd86c82-7f61-4828-b7c3-1be9fac77fc4-maas-api-tls\") pod \"maas-api-e2e-inf-a-20ba8c-74d7cbfdd7-czd8x\" (UID: \"3cd86c82-7f61-4828-b7c3-1be9fac77fc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-20ba8c-74d7cbfdd7-czd8x"
Jul 28 04:52:17.663902 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:17.663865    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cd86c82-7f61-4828-b7c3-1be9fac77fc4-openshift-service-ca\") pod \"maas-api-e2e-inf-a-20ba8c-74d7cbfdd7-czd8x\" (UID: \"3cd86c82-7f61-4828-b7c3-1be9fac77fc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-20ba8c-74d7cbfdd7-czd8x"
Jul 28 04:52:17.665570 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:17.665552    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cd86c82-7f61-4828-b7c3-1be9fac77fc4-maas-api-tls\") pod \"maas-api-e2e-inf-a-20ba8c-74d7cbfdd7-czd8x\" (UID: \"3cd86c82-7f61-4828-b7c3-1be9fac77fc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-20ba8c-74d7cbfdd7-czd8x"
Jul 28 04:52:17.671335 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:17.671315    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftvww\" (UniqueName: \"kubernetes.io/projected/3cd86c82-7f61-4828-b7c3-1be9fac77fc4-kube-api-access-ftvww\") pod \"maas-api-e2e-inf-a-20ba8c-74d7cbfdd7-czd8x\" (UID: \"3cd86c82-7f61-4828-b7c3-1be9fac77fc4\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-20ba8c-74d7cbfdd7-czd8x"
Jul 28 04:52:17.750867 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:17.750750    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-20ba8c-74d7cbfdd7-czd8x"
Jul 28 04:52:17.889162 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:17.889134    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-20ba8c-74d7cbfdd7-czd8x"]
Jul 28 04:52:17.891609 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:52:17.891580    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cd86c82_7f61_4828_b7c3_1be9fac77fc4.slice/crio-cb8dade7571b35e1d82a6601d270077d4c606bb7472fe99bdf446d5f9b57e99c WatchSource:0}: Error finding container cb8dade7571b35e1d82a6601d270077d4c606bb7472fe99bdf446d5f9b57e99c: Status 404 returned error can't find the container with id cb8dade7571b35e1d82a6601d270077d4c606bb7472fe99bdf446d5f9b57e99c
Jul 28 04:52:18.155639 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:18.155605    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-20ba8c-6dc48cf4c8-qw7nc"]
Jul 28 04:52:18.160169 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:18.160152    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-20ba8c-6dc48cf4c8-qw7nc"
Jul 28 04:52:18.162810 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:18.162786    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-20ba8c\""
Jul 28 04:52:18.162945 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:18.162886    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-20ba8c-dockercfg-fhk9n\""
Jul 28 04:52:18.166426 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:18.166399    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-20ba8c-6dc48cf4c8-qw7nc"]
Jul 28 04:52:18.182706 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:18.182680    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-20ba8c-f5b55757b-w6nkt"]
Jul 28 04:52:18.186571 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:18.186555    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-20ba8c-f5b55757b-w6nkt"
Jul 28 04:52:18.195255 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:18.195238    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-20ba8c-f5b55757b-w6nkt"]
Jul 28 04:52:18.269190 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:18.269151    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4342a471-1d30-4a2f-8a62-ed51288f8385-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-20ba8c-6dc48cf4c8-qw7nc\" (UID: \"4342a471-1d30-4a2f-8a62-ed51288f8385\") " pod="openshift-ingress/payload-processing-e2e-inf-a-20ba8c-6dc48cf4c8-qw7nc"
Jul 28 04:52:18.269362 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:18.269206    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbdb2eb2-865e-4bdf-8bd5-d851edd6c10f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-20ba8c-f5b55757b-w6nkt\" (UID: \"dbdb2eb2-865e-4bdf-8bd5-d851edd6c10f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-20ba8c-f5b55757b-w6nkt"
Jul 28 04:52:18.269362 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:18.269324    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t6lk\" (UniqueName: \"kubernetes.io/projected/4342a471-1d30-4a2f-8a62-ed51288f8385-kube-api-access-9t6lk\") pod \"payload-processing-e2e-inf-a-20ba8c-6dc48cf4c8-qw7nc\" (UID: \"4342a471-1d30-4a2f-8a62-ed51288f8385\") " pod="openshift-ingress/payload-processing-e2e-inf-a-20ba8c-6dc48cf4c8-qw7nc"
Jul 28 04:52:18.269441 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:18.269405    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czkvq\" (UniqueName: \"kubernetes.io/projected/dbdb2eb2-865e-4bdf-8bd5-d851edd6c10f-kube-api-access-czkvq\") pod \"payload-pre-processing-e2e-inf-a-20ba8c-f5b55757b-w6nkt\" (UID: \"dbdb2eb2-865e-4bdf-8bd5-d851edd6c10f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-20ba8c-f5b55757b-w6nkt"
Jul 28 04:52:18.361107 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:18.361068    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-20ba8c-74d7cbfdd7-czd8x" event={"ID":"3cd86c82-7f61-4828-b7c3-1be9fac77fc4","Type":"ContainerStarted","Data":"cb8dade7571b35e1d82a6601d270077d4c606bb7472fe99bdf446d5f9b57e99c"}
Jul 28 04:52:18.370872 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:18.370831    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czkvq\" (UniqueName: \"kubernetes.io/projected/dbdb2eb2-865e-4bdf-8bd5-d851edd6c10f-kube-api-access-czkvq\") pod \"payload-pre-processing-e2e-inf-a-20ba8c-f5b55757b-w6nkt\" (UID: \"dbdb2eb2-865e-4bdf-8bd5-d851edd6c10f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-20ba8c-f5b55757b-w6nkt"
Jul 28 04:52:18.370999 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:18.370896    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4342a471-1d30-4a2f-8a62-ed51288f8385-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-20ba8c-6dc48cf4c8-qw7nc\" (UID: \"4342a471-1d30-4a2f-8a62-ed51288f8385\") " pod="openshift-ingress/payload-processing-e2e-inf-a-20ba8c-6dc48cf4c8-qw7nc"
Jul 28 04:52:18.370999 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:18.370944    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbdb2eb2-865e-4bdf-8bd5-d851edd6c10f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-20ba8c-f5b55757b-w6nkt\" (UID: \"dbdb2eb2-865e-4bdf-8bd5-d851edd6c10f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-20ba8c-f5b55757b-w6nkt"
Jul 28 04:52:18.371081 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:18.371007    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t6lk\" (UniqueName: \"kubernetes.io/projected/4342a471-1d30-4a2f-8a62-ed51288f8385-kube-api-access-9t6lk\") pod \"payload-processing-e2e-inf-a-20ba8c-6dc48cf4c8-qw7nc\" (UID: \"4342a471-1d30-4a2f-8a62-ed51288f8385\") " pod="openshift-ingress/payload-processing-e2e-inf-a-20ba8c-6dc48cf4c8-qw7nc"
Jul 28 04:52:18.371781 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:18.371754    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbdb2eb2-865e-4bdf-8bd5-d851edd6c10f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-20ba8c-f5b55757b-w6nkt\" (UID: \"dbdb2eb2-865e-4bdf-8bd5-d851edd6c10f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-20ba8c-f5b55757b-w6nkt"
Jul 28 04:52:18.371926 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:18.371786    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4342a471-1d30-4a2f-8a62-ed51288f8385-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-20ba8c-6dc48cf4c8-qw7nc\" (UID: \"4342a471-1d30-4a2f-8a62-ed51288f8385\") " pod="openshift-ingress/payload-processing-e2e-inf-a-20ba8c-6dc48cf4c8-qw7nc"
Jul 28 04:52:18.378429 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:18.378410    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t6lk\" (UniqueName: \"kubernetes.io/projected/4342a471-1d30-4a2f-8a62-ed51288f8385-kube-api-access-9t6lk\") pod \"payload-processing-e2e-inf-a-20ba8c-6dc48cf4c8-qw7nc\" (UID: \"4342a471-1d30-4a2f-8a62-ed51288f8385\") " pod="openshift-ingress/payload-processing-e2e-inf-a-20ba8c-6dc48cf4c8-qw7nc"
Jul 28 04:52:18.378533 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:18.378513    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czkvq\" (UniqueName: \"kubernetes.io/projected/dbdb2eb2-865e-4bdf-8bd5-d851edd6c10f-kube-api-access-czkvq\") pod \"payload-pre-processing-e2e-inf-a-20ba8c-f5b55757b-w6nkt\" (UID: \"dbdb2eb2-865e-4bdf-8bd5-d851edd6c10f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-20ba8c-f5b55757b-w6nkt"
Jul 28 04:52:18.472193 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:18.472168    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-20ba8c-6dc48cf4c8-qw7nc"
Jul 28 04:52:18.497008 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:18.496981    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-20ba8c-f5b55757b-w6nkt"
Jul 28 04:52:18.651700 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:18.651675    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-20ba8c-f5b55757b-w6nkt"]
Jul 28 04:52:18.653596 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:52:18.653569    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbdb2eb2_865e_4bdf_8bd5_d851edd6c10f.slice/crio-ea82d4dbff8495f0a883cc54f2a41f3c76d4db8962d692e448315e47097b0f99 WatchSource:0}: Error finding container ea82d4dbff8495f0a883cc54f2a41f3c76d4db8962d692e448315e47097b0f99: Status 404 returned error can't find the container with id ea82d4dbff8495f0a883cc54f2a41f3c76d4db8962d692e448315e47097b0f99
Jul 28 04:52:18.825783 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:18.825762    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-20ba8c-6dc48cf4c8-qw7nc"]
Jul 28 04:52:18.827875 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:52:18.827824    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4342a471_1d30_4a2f_8a62_ed51288f8385.slice/crio-2fce3def5cad6507b5dbc81925e5a689e7be4d7be1de840bbb24712b59162465 WatchSource:0}: Error finding container 2fce3def5cad6507b5dbc81925e5a689e7be4d7be1de840bbb24712b59162465: Status 404 returned error can't find the container with id 2fce3def5cad6507b5dbc81925e5a689e7be4d7be1de840bbb24712b59162465
Jul 28 04:52:19.368112 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:19.368079    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-20ba8c-f5b55757b-w6nkt" event={"ID":"dbdb2eb2-865e-4bdf-8bd5-d851edd6c10f","Type":"ContainerStarted","Data":"07353ef95c329cb08719652aaf766ce43679461735fbd12214e4830ecf8bcdd8"}
Jul 28 04:52:19.368402 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:19.368139    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-20ba8c-f5b55757b-w6nkt"
Jul 28 04:52:19.368402 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:19.368162    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-20ba8c-f5b55757b-w6nkt" event={"ID":"dbdb2eb2-865e-4bdf-8bd5-d851edd6c10f","Type":"ContainerStarted","Data":"ea82d4dbff8495f0a883cc54f2a41f3c76d4db8962d692e448315e47097b0f99"}
Jul 28 04:52:19.370051 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:19.370017    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-20ba8c-6dc48cf4c8-qw7nc" event={"ID":"4342a471-1d30-4a2f-8a62-ed51288f8385","Type":"ContainerStarted","Data":"4a07d02a97fc6e22ed18e8a7ee04682b95335523bbf5e717b9b3a44afe9e6dc8"}
Jul 28 04:52:19.370186 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:19.370067    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-20ba8c-6dc48cf4c8-qw7nc" event={"ID":"4342a471-1d30-4a2f-8a62-ed51288f8385","Type":"ContainerStarted","Data":"2fce3def5cad6507b5dbc81925e5a689e7be4d7be1de840bbb24712b59162465"}
Jul 28 04:52:19.370186 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:19.370112    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-20ba8c-6dc48cf4c8-qw7nc"
Jul 28 04:52:19.371817 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:19.371796    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-20ba8c-74d7cbfdd7-czd8x" event={"ID":"3cd86c82-7f61-4828-b7c3-1be9fac77fc4","Type":"ContainerStarted","Data":"35a593f3b6a66611ac34625eaab26847f78b6560e1a77f62472a0680a19aba65"}
Jul 28 04:52:19.371959 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:19.371942    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-20ba8c-74d7cbfdd7-czd8x"
Jul 28 04:52:19.383174 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:19.383131    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-20ba8c-f5b55757b-w6nkt" podStartSLOduration=1.383113944 podStartE2EDuration="1.383113944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:52:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:52:19.382333681 +0000 UTC m=+2439.772509699" watchObservedRunningTime="2026-07-28 04:52:19.383113944 +0000 UTC m=+2439.773289966"
Jul 28 04:52:19.399149 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:19.399102    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-20ba8c-6dc48cf4c8-qw7nc" podStartSLOduration=1.399087272 podStartE2EDuration="1.399087272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:52:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:52:19.397490418 +0000 UTC m=+2439.787666439" watchObservedRunningTime="2026-07-28 04:52:19.399087272 +0000 UTC m=+2439.789263293"
Jul 28 04:52:19.414032 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:19.413984    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-20ba8c-74d7cbfdd7-czd8x" podStartSLOduration=1.91309733 podStartE2EDuration="2.413967394s" totalImagesPullingTime="500.870064ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:52:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:52:19.412536133 +0000 UTC m=+2439.802712156" watchObservedRunningTime="2026-07-28 04:52:19.413967394 +0000 UTC m=+2439.804143429"
Jul 28 04:52:25.381643 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:25.381615    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-20ba8c-74d7cbfdd7-czd8x"
Jul 28 04:52:25.990192 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:25.990152    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1b1e7a-78679fcd47-whqrx"]
Jul 28 04:52:25.994498 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:25.994471    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1b1e7a-78679fcd47-whqrx"
Jul 28 04:52:25.997729 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:25.997705    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-1b1e7a-serving-cert\""
Jul 28 04:52:26.018304 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:26.005281    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1b1e7a-78679fcd47-whqrx"]
Jul 28 04:52:26.149997 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:26.149922    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7a32da6-b18e-4f7a-9fcd-60822e49a2f2-maas-api-tls\") pod \"maas-api-e2e-inf-b-1b1e7a-78679fcd47-whqrx\" (UID: \"e7a32da6-b18e-4f7a-9fcd-60822e49a2f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1b1e7a-78679fcd47-whqrx"
Jul 28 04:52:26.150178 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:26.150035    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7a32da6-b18e-4f7a-9fcd-60822e49a2f2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1b1e7a-78679fcd47-whqrx\" (UID: \"e7a32da6-b18e-4f7a-9fcd-60822e49a2f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1b1e7a-78679fcd47-whqrx"
Jul 28 04:52:26.150328 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:26.150304    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c8pl\" (UniqueName: \"kubernetes.io/projected/e7a32da6-b18e-4f7a-9fcd-60822e49a2f2-kube-api-access-7c8pl\") pod \"maas-api-e2e-inf-b-1b1e7a-78679fcd47-whqrx\" (UID: \"e7a32da6-b18e-4f7a-9fcd-60822e49a2f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1b1e7a-78679fcd47-whqrx"
Jul 28 04:52:26.251606 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:26.251517    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7a32da6-b18e-4f7a-9fcd-60822e49a2f2-maas-api-tls\") pod \"maas-api-e2e-inf-b-1b1e7a-78679fcd47-whqrx\" (UID: \"e7a32da6-b18e-4f7a-9fcd-60822e49a2f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1b1e7a-78679fcd47-whqrx"
Jul 28 04:52:26.251606 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:26.251569    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7a32da6-b18e-4f7a-9fcd-60822e49a2f2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1b1e7a-78679fcd47-whqrx\" (UID: \"e7a32da6-b18e-4f7a-9fcd-60822e49a2f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1b1e7a-78679fcd47-whqrx"
Jul 28 04:52:26.251792 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:26.251615    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c8pl\" (UniqueName: \"kubernetes.io/projected/e7a32da6-b18e-4f7a-9fcd-60822e49a2f2-kube-api-access-7c8pl\") pod \"maas-api-e2e-inf-b-1b1e7a-78679fcd47-whqrx\" (UID: \"e7a32da6-b18e-4f7a-9fcd-60822e49a2f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1b1e7a-78679fcd47-whqrx"
Jul 28 04:52:26.251995 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:52:26.251973    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-1b1e7a-serving-cert: secret "maas-api-e2e-inf-b-1b1e7a-serving-cert" not found
Jul 28 04:52:26.252061 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:52:26.252050    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e7a32da6-b18e-4f7a-9fcd-60822e49a2f2-maas-api-tls podName:e7a32da6-b18e-4f7a-9fcd-60822e49a2f2 nodeName:}" failed. No retries permitted until 2026-07-28 04:52:26.752025631 +0000 UTC m=+2447.142201649 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e7a32da6-b18e-4f7a-9fcd-60822e49a2f2-maas-api-tls") pod "maas-api-e2e-inf-b-1b1e7a-78679fcd47-whqrx" (UID: "e7a32da6-b18e-4f7a-9fcd-60822e49a2f2") : secret "maas-api-e2e-inf-b-1b1e7a-serving-cert" not found
Jul 28 04:52:26.252594 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:26.252574    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7a32da6-b18e-4f7a-9fcd-60822e49a2f2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1b1e7a-78679fcd47-whqrx\" (UID: \"e7a32da6-b18e-4f7a-9fcd-60822e49a2f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1b1e7a-78679fcd47-whqrx"
Jul 28 04:52:26.260171 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:26.260150    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c8pl\" (UniqueName: \"kubernetes.io/projected/e7a32da6-b18e-4f7a-9fcd-60822e49a2f2-kube-api-access-7c8pl\") pod \"maas-api-e2e-inf-b-1b1e7a-78679fcd47-whqrx\" (UID: \"e7a32da6-b18e-4f7a-9fcd-60822e49a2f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1b1e7a-78679fcd47-whqrx"
Jul 28 04:52:26.756747 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:26.756715    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7a32da6-b18e-4f7a-9fcd-60822e49a2f2-maas-api-tls\") pod \"maas-api-e2e-inf-b-1b1e7a-78679fcd47-whqrx\" (UID: \"e7a32da6-b18e-4f7a-9fcd-60822e49a2f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1b1e7a-78679fcd47-whqrx"
Jul 28 04:52:26.759200 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:26.759176    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7a32da6-b18e-4f7a-9fcd-60822e49a2f2-maas-api-tls\") pod \"maas-api-e2e-inf-b-1b1e7a-78679fcd47-whqrx\" (UID: \"e7a32da6-b18e-4f7a-9fcd-60822e49a2f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1b1e7a-78679fcd47-whqrx"
Jul 28 04:52:26.784699 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:26.784662    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1b1e7a-b555b7f7d-85gfb"]
Jul 28 04:52:26.788892 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:26.788876    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1b1e7a-b555b7f7d-85gfb"
Jul 28 04:52:26.791629 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:26.791609    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-1b1e7a\""
Jul 28 04:52:26.791739 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:26.791687    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-1b1e7a-dockercfg-vkp6b\""
Jul 28 04:52:26.799261 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:26.799234    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1b1e7a-b555b7f7d-85gfb"]
Jul 28 04:52:26.858089 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:26.858054    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65845d6f-9e4f-4150-85a8-bf22f2c6f47d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1b1e7a-b555b7f7d-85gfb\" (UID: \"65845d6f-9e4f-4150-85a8-bf22f2c6f47d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1b1e7a-b555b7f7d-85gfb"
Jul 28 04:52:26.858274 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:26.858210    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4q62\" (UniqueName: \"kubernetes.io/projected/65845d6f-9e4f-4150-85a8-bf22f2c6f47d-kube-api-access-c4q62\") pod \"payload-processing-e2e-inf-b-1b1e7a-b555b7f7d-85gfb\" (UID: \"65845d6f-9e4f-4150-85a8-bf22f2c6f47d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1b1e7a-b555b7f7d-85gfb"
Jul 28 04:52:26.918072 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:26.918040    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1b1e7a-78679fcd47-whqrx"
Jul 28 04:52:26.959564 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:26.959536    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4q62\" (UniqueName: \"kubernetes.io/projected/65845d6f-9e4f-4150-85a8-bf22f2c6f47d-kube-api-access-c4q62\") pod \"payload-processing-e2e-inf-b-1b1e7a-b555b7f7d-85gfb\" (UID: \"65845d6f-9e4f-4150-85a8-bf22f2c6f47d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1b1e7a-b555b7f7d-85gfb"
Jul 28 04:52:26.959810 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:26.959797    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65845d6f-9e4f-4150-85a8-bf22f2c6f47d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1b1e7a-b555b7f7d-85gfb\" (UID: \"65845d6f-9e4f-4150-85a8-bf22f2c6f47d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1b1e7a-b555b7f7d-85gfb"
Jul 28 04:52:26.960680 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:26.960659    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65845d6f-9e4f-4150-85a8-bf22f2c6f47d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1b1e7a-b555b7f7d-85gfb\" (UID: \"65845d6f-9e4f-4150-85a8-bf22f2c6f47d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1b1e7a-b555b7f7d-85gfb"
Jul 28 04:52:26.968738 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:26.968708    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4q62\" (UniqueName: \"kubernetes.io/projected/65845d6f-9e4f-4150-85a8-bf22f2c6f47d-kube-api-access-c4q62\") pod \"payload-processing-e2e-inf-b-1b1e7a-b555b7f7d-85gfb\" (UID: \"65845d6f-9e4f-4150-85a8-bf22f2c6f47d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1b1e7a-b555b7f7d-85gfb"
Jul 28 04:52:27.083200 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:27.083174    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1b1e7a-78679fcd47-whqrx"]
Jul 28 04:52:27.085345 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:52:27.085321    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7a32da6_b18e_4f7a_9fcd_60822e49a2f2.slice/crio-09a4a7894211585d6e9290ec876ba55363936fd15a7a3812e3de2507ed1dc115 WatchSource:0}: Error finding container 09a4a7894211585d6e9290ec876ba55363936fd15a7a3812e3de2507ed1dc115: Status 404 returned error can't find the container with id 09a4a7894211585d6e9290ec876ba55363936fd15a7a3812e3de2507ed1dc115
Jul 28 04:52:27.100992 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:27.100965    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1b1e7a-b555b7f7d-85gfb"
Jul 28 04:52:27.291885 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:27.291859    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1b1e7a-b555b7f7d-85gfb"]
Jul 28 04:52:27.293607 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:52:27.293576    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65845d6f_9e4f_4150_85a8_bf22f2c6f47d.slice/crio-d80437528976fe97897e64dd37e84178e55ab9ccebad5d1d2969683ac70d00f2 WatchSource:0}: Error finding container d80437528976fe97897e64dd37e84178e55ab9ccebad5d1d2969683ac70d00f2: Status 404 returned error can't find the container with id d80437528976fe97897e64dd37e84178e55ab9ccebad5d1d2969683ac70d00f2
Jul 28 04:52:27.428864 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:27.427965    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1b1e7a-78679fcd47-whqrx" event={"ID":"e7a32da6-b18e-4f7a-9fcd-60822e49a2f2","Type":"ContainerStarted","Data":"09a4a7894211585d6e9290ec876ba55363936fd15a7a3812e3de2507ed1dc115"}
Jul 28 04:52:27.432859 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:27.429888    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1b1e7a-b555b7f7d-85gfb" event={"ID":"65845d6f-9e4f-4150-85a8-bf22f2c6f47d","Type":"ContainerStarted","Data":"7a113745eb195f969f5ecadfc037a76e978bba5422add518e0b895ac30069a5d"}
Jul 28 04:52:27.432859 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:27.429937    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1b1e7a-b555b7f7d-85gfb" event={"ID":"65845d6f-9e4f-4150-85a8-bf22f2c6f47d","Type":"ContainerStarted","Data":"d80437528976fe97897e64dd37e84178e55ab9ccebad5d1d2969683ac70d00f2"}
Jul 28 04:52:27.432859 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:27.430692    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-1b1e7a-b555b7f7d-85gfb"
Jul 28 04:52:27.447369 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:27.447314    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-1b1e7a-b555b7f7d-85gfb" podStartSLOduration=1.447295287 podStartE2EDuration="1.447295287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:52:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:52:27.446619259 +0000 UTC m=+2447.836795277" watchObservedRunningTime="2026-07-28 04:52:27.447295287 +0000 UTC m=+2447.837471309"
Jul 28 04:52:28.437298 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:28.437259    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1b1e7a-78679fcd47-whqrx" event={"ID":"e7a32da6-b18e-4f7a-9fcd-60822e49a2f2","Type":"ContainerStarted","Data":"02b13fa66abf7922e1188372ea3dc66e6f813627846344dc315cda1ce7d915b5"}
Jul 28 04:52:28.437756 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:28.437503    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1b1e7a-78679fcd47-whqrx"
Jul 28 04:52:28.454144 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:28.454098    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1b1e7a-78679fcd47-whqrx" podStartSLOduration=3.088672268 podStartE2EDuration="3.454082191s" totalImagesPullingTime="365.409923ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:52:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:52:28.452177525 +0000 UTC m=+2448.842353546" watchObservedRunningTime="2026-07-28 04:52:28.454082191 +0000 UTC m=+2448.844258260"
Jul 28 04:52:30.379130 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:30.379099    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-20ba8c-f5b55757b-w6nkt"
Jul 28 04:52:30.379617 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:30.379261    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-20ba8c-6dc48cf4c8-qw7nc"
Jul 28 04:52:31.351016 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:31.350979    2566 patch_prober.go:28] interesting pod/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Jul 28 04:52:31.351244 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:31.351046    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588" podUID="2dd74d22-d2b4-47ec-acac-5d60279c7c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Jul 28 04:52:31.454553 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:31.454522    2566 generic.go:356] "Generic (PLEG): container finished" podID="2dd74d22-d2b4-47ec-acac-5d60279c7c59" containerID="d30beb77beb43961d5a4ae6c025b99b2702bd69e2730ba444a28ea756c13197e" exitCode=0
Jul 28 04:52:31.455009 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:31.454600    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588" event={"ID":"2dd74d22-d2b4-47ec-acac-5d60279c7c59","Type":"ContainerDied","Data":"d30beb77beb43961d5a4ae6c025b99b2702bd69e2730ba444a28ea756c13197e"}
Jul 28 04:52:31.455082 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:31.455012    2566 scope.go:117] "RemoveContainer" containerID="d30beb77beb43961d5a4ae6c025b99b2702bd69e2730ba444a28ea756c13197e"
Jul 28 04:52:31.455994 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:31.455970    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:52:31.456057 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:31.456042    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:52:31.456094 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:31.456076    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:52:32.462284 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:32.462250    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588" event={"ID":"2dd74d22-d2b4-47ec-acac-5d60279c7c59","Type":"ContainerStarted","Data":"7d745e49486875a349818d281d277ac8671ead2e4f17a42840dcd982ef073193"}
Jul 28 04:52:33.274155 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:33.274120    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588"
Jul 28 04:52:33.274358 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:33.274334    2566 patch_prober.go:28] interesting pod/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Jul 28 04:52:33.274405 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:33.274384    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588" podUID="2dd74d22-d2b4-47ec-acac-5d60279c7c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Jul 28 04:52:34.273783 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:34.273747    2566 patch_prober.go:28] interesting pod/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Jul 28 04:52:34.274201 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:34.273806    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588" podUID="2dd74d22-d2b4-47ec-acac-5d60279c7c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Jul 28 04:52:34.448438 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:34.448405    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1b1e7a-78679fcd47-whqrx"
Jul 28 04:52:35.274005 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:35.273964    2566 patch_prober.go:28] interesting pod/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Jul 28 04:52:35.274464 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:35.274027    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588" podUID="2dd74d22-d2b4-47ec-acac-5d60279c7c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Jul 28 04:52:36.274432 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:36.274397    2566 patch_prober.go:28] interesting pod/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Jul 28 04:52:36.274825 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:36.274454    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588" podUID="2dd74d22-d2b4-47ec-acac-5d60279c7c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Jul 28 04:52:37.277643 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:37.277608    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588"
Jul 28 04:52:37.278075 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:37.278023    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588"
Jul 28 04:52:37.278830 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:37.278808    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588"
Jul 28 04:52:39.445092 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:52:39.445065    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-1b1e7a-b555b7f7d-85gfb"
Jul 28 04:53:04.485466 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:04.485431    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-1b1e7a/test-model-1b1e7a-kserve-b4ff58867-bg5m9"]
Jul 28 04:53:04.488729 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:04.488710    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1b1e7a/test-model-1b1e7a-kserve-b4ff58867-bg5m9"
Jul 28 04:53:04.491536 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:04.491509    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1b1e7a\"/\"kube-root-ca.crt\""
Jul 28 04:53:04.492742 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:04.492721    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-1b1e7a\"/\"openshift-service-ca.crt\""
Jul 28 04:53:04.492888 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:04.492746    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1b1e7a\"/\"test-model-1b1e7a-kserve-self-signed-certs\""
Jul 28 04:53:04.492888 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:04.492729    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-1b1e7a\"/\"default-dockercfg-jf92q\""
Jul 28 04:53:04.501742 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:04.501714    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1b1e7a/test-model-1b1e7a-kserve-b4ff58867-bg5m9"]
Jul 28 04:53:04.635062 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:04.635032    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d-home\") pod \"test-model-1b1e7a-kserve-b4ff58867-bg5m9\" (UID: \"30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d\") " pod="ai-tenant-e2e-inf-b-1b1e7a/test-model-1b1e7a-kserve-b4ff58867-bg5m9"
Jul 28 04:53:04.635247 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:04.635073    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq9wk\" (UniqueName: \"kubernetes.io/projected/30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d-kube-api-access-lq9wk\") pod \"test-model-1b1e7a-kserve-b4ff58867-bg5m9\" (UID: \"30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d\") " pod="ai-tenant-e2e-inf-b-1b1e7a/test-model-1b1e7a-kserve-b4ff58867-bg5m9"
Jul 28 04:53:04.635247 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:04.635166    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d-tmp-dir\") pod \"test-model-1b1e7a-kserve-b4ff58867-bg5m9\" (UID: \"30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d\") " pod="ai-tenant-e2e-inf-b-1b1e7a/test-model-1b1e7a-kserve-b4ff58867-bg5m9"
Jul 28 04:53:04.635247 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:04.635203    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d-dshm\") pod \"test-model-1b1e7a-kserve-b4ff58867-bg5m9\" (UID: \"30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d\") " pod="ai-tenant-e2e-inf-b-1b1e7a/test-model-1b1e7a-kserve-b4ff58867-bg5m9"
Jul 28 04:53:04.635247 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:04.635224    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d-tls-certs\") pod \"test-model-1b1e7a-kserve-b4ff58867-bg5m9\" (UID: \"30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d\") " pod="ai-tenant-e2e-inf-b-1b1e7a/test-model-1b1e7a-kserve-b4ff58867-bg5m9"
Jul 28 04:53:04.635443 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:04.635292    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d-model-cache\") pod \"test-model-1b1e7a-kserve-b4ff58867-bg5m9\" (UID: \"30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d\") " pod="ai-tenant-e2e-inf-b-1b1e7a/test-model-1b1e7a-kserve-b4ff58867-bg5m9"
Jul 28 04:53:04.736439 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:04.736349    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d-tmp-dir\") pod \"test-model-1b1e7a-kserve-b4ff58867-bg5m9\" (UID: \"30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d\") " pod="ai-tenant-e2e-inf-b-1b1e7a/test-model-1b1e7a-kserve-b4ff58867-bg5m9"
Jul 28 04:53:04.736439 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:04.736413    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d-dshm\") pod \"test-model-1b1e7a-kserve-b4ff58867-bg5m9\" (UID: \"30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d\") " pod="ai-tenant-e2e-inf-b-1b1e7a/test-model-1b1e7a-kserve-b4ff58867-bg5m9"
Jul 28 04:53:04.736676 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:04.736444    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d-tls-certs\") pod \"test-model-1b1e7a-kserve-b4ff58867-bg5m9\" (UID: \"30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d\") " pod="ai-tenant-e2e-inf-b-1b1e7a/test-model-1b1e7a-kserve-b4ff58867-bg5m9"
Jul 28 04:53:04.736676 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:04.736494    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d-model-cache\") pod \"test-model-1b1e7a-kserve-b4ff58867-bg5m9\" (UID: \"30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d\") " pod="ai-tenant-e2e-inf-b-1b1e7a/test-model-1b1e7a-kserve-b4ff58867-bg5m9"
Jul 28 04:53:04.736676 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:04.736587    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d-home\") pod \"test-model-1b1e7a-kserve-b4ff58867-bg5m9\" (UID: \"30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d\") " pod="ai-tenant-e2e-inf-b-1b1e7a/test-model-1b1e7a-kserve-b4ff58867-bg5m9"
Jul 28 04:53:04.736676 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:04.736621    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq9wk\" (UniqueName: \"kubernetes.io/projected/30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d-kube-api-access-lq9wk\") pod \"test-model-1b1e7a-kserve-b4ff58867-bg5m9\" (UID: \"30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d\") " pod="ai-tenant-e2e-inf-b-1b1e7a/test-model-1b1e7a-kserve-b4ff58867-bg5m9"
Jul 28 04:53:04.736948 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:04.736805    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d-tmp-dir\") pod \"test-model-1b1e7a-kserve-b4ff58867-bg5m9\" (UID: \"30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d\") " pod="ai-tenant-e2e-inf-b-1b1e7a/test-model-1b1e7a-kserve-b4ff58867-bg5m9"
Jul 28 04:53:04.737029 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:04.736988    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d-model-cache\") pod \"test-model-1b1e7a-kserve-b4ff58867-bg5m9\" (UID: \"30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d\") " pod="ai-tenant-e2e-inf-b-1b1e7a/test-model-1b1e7a-kserve-b4ff58867-bg5m9"
Jul 28 04:53:04.737029 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:04.737013    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d-home\") pod \"test-model-1b1e7a-kserve-b4ff58867-bg5m9\" (UID: \"30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d\") " pod="ai-tenant-e2e-inf-b-1b1e7a/test-model-1b1e7a-kserve-b4ff58867-bg5m9"
Jul 28 04:53:04.738679 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:04.738658    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d-dshm\") pod \"test-model-1b1e7a-kserve-b4ff58867-bg5m9\" (UID: \"30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d\") " pod="ai-tenant-e2e-inf-b-1b1e7a/test-model-1b1e7a-kserve-b4ff58867-bg5m9"
Jul 28 04:53:04.738869 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:04.738826    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d-tls-certs\") pod \"test-model-1b1e7a-kserve-b4ff58867-bg5m9\" (UID: \"30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d\") " pod="ai-tenant-e2e-inf-b-1b1e7a/test-model-1b1e7a-kserve-b4ff58867-bg5m9"
Jul 28 04:53:04.747640 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:04.747612    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq9wk\" (UniqueName: \"kubernetes.io/projected/30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d-kube-api-access-lq9wk\") pod \"test-model-1b1e7a-kserve-b4ff58867-bg5m9\" (UID: \"30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d\") " pod="ai-tenant-e2e-inf-b-1b1e7a/test-model-1b1e7a-kserve-b4ff58867-bg5m9"
Jul 28 04:53:04.800342 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:04.800309    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1b1e7a/test-model-1b1e7a-kserve-b4ff58867-bg5m9"
Jul 28 04:53:04.935033 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:04.934999    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-1b1e7a/test-model-1b1e7a-kserve-b4ff58867-bg5m9"]
Jul 28 04:53:04.936178 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:53:04.936153    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30c5c4d6_fcb2_4fa1_9e7e_24d1b0eaf40d.slice/crio-d6ce48ce2d2b3f11dab52cc5bd998faefe969d034de30c07ede73c991280f195 WatchSource:0}: Error finding container d6ce48ce2d2b3f11dab52cc5bd998faefe969d034de30c07ede73c991280f195: Status 404 returned error can't find the container with id d6ce48ce2d2b3f11dab52cc5bd998faefe969d034de30c07ede73c991280f195
Jul 28 04:53:05.627935 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:05.627898    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1b1e7a/test-model-1b1e7a-kserve-b4ff58867-bg5m9" event={"ID":"30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d","Type":"ContainerStarted","Data":"3ed80a23a04048dbba7d80a8d5e9b77b40000bbe0e57a7fe9adc1893f949bdfb"}
Jul 28 04:53:05.627935 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:05.627941    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1b1e7a/test-model-1b1e7a-kserve-b4ff58867-bg5m9" event={"ID":"30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d","Type":"ContainerStarted","Data":"d6ce48ce2d2b3f11dab52cc5bd998faefe969d034de30c07ede73c991280f195"}
Jul 28 04:53:05.645964 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:05.645916    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-1b1e7a/test-model-1b1e7a-kserve-b4ff58867-bg5m9" podStartSLOduration=1.6459024169999998 podStartE2EDuration="1.645902417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:53:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:53:05.644231664 +0000 UTC m=+2486.034407683" watchObservedRunningTime="2026-07-28 04:53:05.645902417 +0000 UTC m=+2486.036078439"
Jul 28 04:53:14.801158 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:14.801122    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-1b1e7a/test-model-1b1e7a-kserve-b4ff58867-bg5m9"
Jul 28 04:53:14.801559 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:14.801182    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-1b1e7a/test-model-1b1e7a-kserve-b4ff58867-bg5m9"
Jul 28 04:53:14.814105 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:14.814081    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-1b1e7a/test-model-1b1e7a-kserve-b4ff58867-bg5m9"
Jul 28 04:53:15.691470 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:15.691444    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-1b1e7a/test-model-1b1e7a-kserve-b4ff58867-bg5m9"
Jul 28 04:53:46.018815 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:46.018781    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1b1e7a/test-model-1b1e7a-kserve-b4ff58867-bg5m9"]
Jul 28 04:53:56.599095 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:56.599057    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-20ba8c-74d7cbfdd7-czd8x"]
Jul 28 04:53:56.599650 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:56.599276    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-20ba8c-74d7cbfdd7-czd8x" podUID="3cd86c82-7f61-4828-b7c3-1be9fac77fc4" containerName="maas-api" containerID="cri-o://35a593f3b6a66611ac34625eaab26847f78b6560e1a77f62472a0680a19aba65" gracePeriod=30
Jul 28 04:53:56.654886 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:56.654851    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-20ba8c-6dc48cf4c8-qw7nc"]
Jul 28 04:53:56.655162 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:56.655106    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-20ba8c-6dc48cf4c8-qw7nc" podUID="4342a471-1d30-4a2f-8a62-ed51288f8385" containerName="payload-processing" containerID="cri-o://4a07d02a97fc6e22ed18e8a7ee04682b95335523bbf5e717b9b3a44afe9e6dc8" gracePeriod=30
Jul 28 04:53:56.665941 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:56.665913    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-20ba8c-f5b55757b-w6nkt"]
Jul 28 04:53:56.666291 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:56.666261    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-20ba8c-f5b55757b-w6nkt" podUID="dbdb2eb2-865e-4bdf-8bd5-d851edd6c10f" containerName="payload-pre-processing" containerID="cri-o://07353ef95c329cb08719652aaf766ce43679461735fbd12214e4830ecf8bcdd8" gracePeriod=30
Jul 28 04:53:56.888785 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:56.888753    2566 generic.go:356] "Generic (PLEG): container finished" podID="dbdb2eb2-865e-4bdf-8bd5-d851edd6c10f" containerID="07353ef95c329cb08719652aaf766ce43679461735fbd12214e4830ecf8bcdd8" exitCode=0
Jul 28 04:53:56.888957 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:56.888823    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-20ba8c-f5b55757b-w6nkt" event={"ID":"dbdb2eb2-865e-4bdf-8bd5-d851edd6c10f","Type":"ContainerDied","Data":"07353ef95c329cb08719652aaf766ce43679461735fbd12214e4830ecf8bcdd8"}
Jul 28 04:53:56.890866 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:56.890797    2566 generic.go:356] "Generic (PLEG): container finished" podID="4342a471-1d30-4a2f-8a62-ed51288f8385" containerID="4a07d02a97fc6e22ed18e8a7ee04682b95335523bbf5e717b9b3a44afe9e6dc8" exitCode=0
Jul 28 04:53:56.890992 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:56.890976    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-20ba8c-6dc48cf4c8-qw7nc" event={"ID":"4342a471-1d30-4a2f-8a62-ed51288f8385","Type":"ContainerDied","Data":"4a07d02a97fc6e22ed18e8a7ee04682b95335523bbf5e717b9b3a44afe9e6dc8"}
Jul 28 04:53:56.961900 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:56.961873    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-20ba8c-6dc48cf4c8-qw7nc"
Jul 28 04:53:56.975547 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:56.975523    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-20ba8c-f5b55757b-w6nkt"
Jul 28 04:53:57.060509 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:57.060478    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-czkvq\" (UniqueName: \"kubernetes.io/projected/dbdb2eb2-865e-4bdf-8bd5-d851edd6c10f-kube-api-access-czkvq\") pod \"dbdb2eb2-865e-4bdf-8bd5-d851edd6c10f\" (UID: \"dbdb2eb2-865e-4bdf-8bd5-d851edd6c10f\") "
Jul 28 04:53:57.060643 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:57.060529    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbdb2eb2-865e-4bdf-8bd5-d851edd6c10f-plugins-config-volume\") pod \"dbdb2eb2-865e-4bdf-8bd5-d851edd6c10f\" (UID: \"dbdb2eb2-865e-4bdf-8bd5-d851edd6c10f\") "
Jul 28 04:53:57.060643 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:57.060589    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4342a471-1d30-4a2f-8a62-ed51288f8385-plugins-config-volume\") pod \"4342a471-1d30-4a2f-8a62-ed51288f8385\" (UID: \"4342a471-1d30-4a2f-8a62-ed51288f8385\") "
Jul 28 04:53:57.060643 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:57.060630    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9t6lk\" (UniqueName: \"kubernetes.io/projected/4342a471-1d30-4a2f-8a62-ed51288f8385-kube-api-access-9t6lk\") pod \"4342a471-1d30-4a2f-8a62-ed51288f8385\" (UID: \"4342a471-1d30-4a2f-8a62-ed51288f8385\") "
Jul 28 04:53:57.060996 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:57.060972    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbdb2eb2-865e-4bdf-8bd5-d851edd6c10f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dbdb2eb2-865e-4bdf-8bd5-d851edd6c10f" (UID: "dbdb2eb2-865e-4bdf-8bd5-d851edd6c10f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 04:53:57.061078 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:57.061024    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4342a471-1d30-4a2f-8a62-ed51288f8385-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4342a471-1d30-4a2f-8a62-ed51288f8385" (UID: "4342a471-1d30-4a2f-8a62-ed51288f8385"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 04:53:57.062667 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:57.062646    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4342a471-1d30-4a2f-8a62-ed51288f8385-kube-api-access-9t6lk" (OuterVolumeSpecName: "kube-api-access-9t6lk") pod "4342a471-1d30-4a2f-8a62-ed51288f8385" (UID: "4342a471-1d30-4a2f-8a62-ed51288f8385"). InnerVolumeSpecName "kube-api-access-9t6lk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:53:57.062986 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:57.062968    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbdb2eb2-865e-4bdf-8bd5-d851edd6c10f-kube-api-access-czkvq" (OuterVolumeSpecName: "kube-api-access-czkvq") pod "dbdb2eb2-865e-4bdf-8bd5-d851edd6c10f" (UID: "dbdb2eb2-865e-4bdf-8bd5-d851edd6c10f"). InnerVolumeSpecName "kube-api-access-czkvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:53:57.068659 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:57.068639    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-20ba8c-74d7cbfdd7-czd8x"
Jul 28 04:53:57.162065 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:57.161980    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cd86c82-7f61-4828-b7c3-1be9fac77fc4-maas-api-tls\") pod \"3cd86c82-7f61-4828-b7c3-1be9fac77fc4\" (UID: \"3cd86c82-7f61-4828-b7c3-1be9fac77fc4\") "
Jul 28 04:53:57.162065 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:57.162033    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftvww\" (UniqueName: \"kubernetes.io/projected/3cd86c82-7f61-4828-b7c3-1be9fac77fc4-kube-api-access-ftvww\") pod \"3cd86c82-7f61-4828-b7c3-1be9fac77fc4\" (UID: \"3cd86c82-7f61-4828-b7c3-1be9fac77fc4\") "
Jul 28 04:53:57.162229 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:57.162142    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cd86c82-7f61-4828-b7c3-1be9fac77fc4-openshift-service-ca\") pod \"3cd86c82-7f61-4828-b7c3-1be9fac77fc4\" (UID: \"3cd86c82-7f61-4828-b7c3-1be9fac77fc4\") "
Jul 28 04:53:57.162480 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:57.162459    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t6lk\" (UniqueName: \"kubernetes.io/projected/4342a471-1d30-4a2f-8a62-ed51288f8385-kube-api-access-9t6lk\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:53:57.162480 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:57.162467    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cd86c82-7f61-4828-b7c3-1be9fac77fc4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3cd86c82-7f61-4828-b7c3-1be9fac77fc4" (UID: "3cd86c82-7f61-4828-b7c3-1be9fac77fc4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 04:53:57.162638 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:57.162477    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czkvq\" (UniqueName: \"kubernetes.io/projected/dbdb2eb2-865e-4bdf-8bd5-d851edd6c10f-kube-api-access-czkvq\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:53:57.162638 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:57.162515    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbdb2eb2-865e-4bdf-8bd5-d851edd6c10f-plugins-config-volume\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:53:57.162638 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:57.162532    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4342a471-1d30-4a2f-8a62-ed51288f8385-plugins-config-volume\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:53:57.164105 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:57.164086    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cd86c82-7f61-4828-b7c3-1be9fac77fc4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3cd86c82-7f61-4828-b7c3-1be9fac77fc4" (UID: "3cd86c82-7f61-4828-b7c3-1be9fac77fc4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 04:53:57.164167 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:57.164129    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cd86c82-7f61-4828-b7c3-1be9fac77fc4-kube-api-access-ftvww" (OuterVolumeSpecName: "kube-api-access-ftvww") pod "3cd86c82-7f61-4828-b7c3-1be9fac77fc4" (UID: "3cd86c82-7f61-4828-b7c3-1be9fac77fc4"). InnerVolumeSpecName "kube-api-access-ftvww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:53:57.263195 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:57.263165    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cd86c82-7f61-4828-b7c3-1be9fac77fc4-openshift-service-ca\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:53:57.263195 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:57.263194    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3cd86c82-7f61-4828-b7c3-1be9fac77fc4-maas-api-tls\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:53:57.263195 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:57.263203    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftvww\" (UniqueName: \"kubernetes.io/projected/3cd86c82-7f61-4828-b7c3-1be9fac77fc4-kube-api-access-ftvww\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:53:57.897809 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:57.897777    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-20ba8c-f5b55757b-w6nkt"
Jul 28 04:53:57.897809 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:57.897784    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-20ba8c-f5b55757b-w6nkt" event={"ID":"dbdb2eb2-865e-4bdf-8bd5-d851edd6c10f","Type":"ContainerDied","Data":"ea82d4dbff8495f0a883cc54f2a41f3c76d4db8962d692e448315e47097b0f99"}
Jul 28 04:53:57.898344 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:57.897874    2566 scope.go:117] "RemoveContainer" containerID="07353ef95c329cb08719652aaf766ce43679461735fbd12214e4830ecf8bcdd8"
Jul 28 04:53:57.899319 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:57.899296    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-20ba8c-6dc48cf4c8-qw7nc"
Jul 28 04:53:57.899445 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:57.899321    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-20ba8c-6dc48cf4c8-qw7nc" event={"ID":"4342a471-1d30-4a2f-8a62-ed51288f8385","Type":"ContainerDied","Data":"2fce3def5cad6507b5dbc81925e5a689e7be4d7be1de840bbb24712b59162465"}
Jul 28 04:53:57.900853 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:57.900817    2566 generic.go:356] "Generic (PLEG): container finished" podID="3cd86c82-7f61-4828-b7c3-1be9fac77fc4" containerID="35a593f3b6a66611ac34625eaab26847f78b6560e1a77f62472a0680a19aba65" exitCode=0
Jul 28 04:53:57.900958 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:57.900897    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-20ba8c-74d7cbfdd7-czd8x" event={"ID":"3cd86c82-7f61-4828-b7c3-1be9fac77fc4","Type":"ContainerDied","Data":"35a593f3b6a66611ac34625eaab26847f78b6560e1a77f62472a0680a19aba65"}
Jul 28 04:53:57.900958 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:57.900950    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-20ba8c-74d7cbfdd7-czd8x"
Jul 28 04:53:57.901056 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:57.900963    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-20ba8c-74d7cbfdd7-czd8x" event={"ID":"3cd86c82-7f61-4828-b7c3-1be9fac77fc4","Type":"ContainerDied","Data":"cb8dade7571b35e1d82a6601d270077d4c606bb7472fe99bdf446d5f9b57e99c"}
Jul 28 04:53:57.908108 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:57.907878    2566 scope.go:117] "RemoveContainer" containerID="4a07d02a97fc6e22ed18e8a7ee04682b95335523bbf5e717b9b3a44afe9e6dc8"
Jul 28 04:53:57.916612 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:57.916595    2566 scope.go:117] "RemoveContainer" containerID="35a593f3b6a66611ac34625eaab26847f78b6560e1a77f62472a0680a19aba65"
Jul 28 04:53:57.920966 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:57.920942    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-20ba8c-f5b55757b-w6nkt"]
Jul 28 04:53:57.924376 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:57.924327    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-20ba8c-f5b55757b-w6nkt"]
Jul 28 04:53:57.924792 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:57.924775    2566 scope.go:117] "RemoveContainer" containerID="35a593f3b6a66611ac34625eaab26847f78b6560e1a77f62472a0680a19aba65"
Jul 28 04:53:57.925084 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:53:57.925063    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35a593f3b6a66611ac34625eaab26847f78b6560e1a77f62472a0680a19aba65\": container with ID starting with 35a593f3b6a66611ac34625eaab26847f78b6560e1a77f62472a0680a19aba65 not found: ID does not exist" containerID="35a593f3b6a66611ac34625eaab26847f78b6560e1a77f62472a0680a19aba65"
Jul 28 04:53:57.925191 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:57.925089    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35a593f3b6a66611ac34625eaab26847f78b6560e1a77f62472a0680a19aba65"} err="failed to get container status \"35a593f3b6a66611ac34625eaab26847f78b6560e1a77f62472a0680a19aba65\": rpc error: code = NotFound desc = could not find container \"35a593f3b6a66611ac34625eaab26847f78b6560e1a77f62472a0680a19aba65\": container with ID starting with 35a593f3b6a66611ac34625eaab26847f78b6560e1a77f62472a0680a19aba65 not found: ID does not exist"
Jul 28 04:53:57.934614 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:57.934593    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-20ba8c-6dc48cf4c8-qw7nc"]
Jul 28 04:53:57.937824 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:57.937802    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-20ba8c-6dc48cf4c8-qw7nc"]
Jul 28 04:53:57.948180 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:57.948148    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-20ba8c-74d7cbfdd7-czd8x"]
Jul 28 04:53:57.951616 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:57.951591    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-20ba8c-74d7cbfdd7-czd8x"]
Jul 28 04:53:58.088384 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:58.088351    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cd86c82-7f61-4828-b7c3-1be9fac77fc4" path="/var/lib/kubelet/pods/3cd86c82-7f61-4828-b7c3-1be9fac77fc4/volumes"
Jul 28 04:53:58.088713 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:58.088699    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4342a471-1d30-4a2f-8a62-ed51288f8385" path="/var/lib/kubelet/pods/4342a471-1d30-4a2f-8a62-ed51288f8385/volumes"
Jul 28 04:53:58.089056 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:53:58.089044    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbdb2eb2-865e-4bdf-8bd5-d851edd6c10f" path="/var/lib/kubelet/pods/dbdb2eb2-865e-4bdf-8bd5-d851edd6c10f/volumes"
Jul 28 04:54:01.038101 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:01.038059    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-1b1e7a/test-model-1b1e7a-kserve-b4ff58867-bg5m9" podUID="30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d" containerName="main" containerID="cri-o://3ed80a23a04048dbba7d80a8d5e9b77b40000bbe0e57a7fe9adc1893f949bdfb" gracePeriod=45
Jul 28 04:54:01.302192 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:01.302128    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1b1e7a/test-model-1b1e7a-kserve-b4ff58867-bg5m9"
Jul 28 04:54:01.397464 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:01.397416    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d-tmp-dir\") pod \"30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d\" (UID: \"30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d\") "
Jul 28 04:54:01.397644 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:01.397502    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d-model-cache\") pod \"30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d\" (UID: \"30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d\") "
Jul 28 04:54:01.397644 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:01.397539    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d-tls-certs\") pod \"30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d\" (UID: \"30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d\") "
Jul 28 04:54:01.397644 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:01.397604    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lq9wk\" (UniqueName: \"kubernetes.io/projected/30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d-kube-api-access-lq9wk\") pod \"30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d\" (UID: \"30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d\") "
Jul 28 04:54:01.397644 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:01.397635    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d-home\") pod \"30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d\" (UID: \"30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d\") "
Jul 28 04:54:01.397883 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:01.397665    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d" (UID: "30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:54:01.397883 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:01.397681    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d-dshm\") pod \"30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d\" (UID: \"30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d\") "
Jul 28 04:54:01.397883 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:01.397774    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d-model-cache" (OuterVolumeSpecName: "model-cache") pod "30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d" (UID: "30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:54:01.398045 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:01.397957    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d-home" (OuterVolumeSpecName: "home") pod "30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d" (UID: "30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:54:01.398161 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:01.398139    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d-tmp-dir\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:54:01.398161 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:01.398162    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d-model-cache\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:54:01.398296 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:01.398170    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d-home\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:54:01.399719 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:01.399676    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d-dshm" (OuterVolumeSpecName: "dshm") pod "30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d" (UID: "30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:54:01.399830 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:01.399751    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d" (UID: "30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 04:54:01.399830 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:01.399759    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d-kube-api-access-lq9wk" (OuterVolumeSpecName: "kube-api-access-lq9wk") pod "30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d" (UID: "30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d"). InnerVolumeSpecName "kube-api-access-lq9wk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:54:01.498946 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:01.498912    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq9wk\" (UniqueName: \"kubernetes.io/projected/30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d-kube-api-access-lq9wk\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:54:01.498946 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:01.498942    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d-dshm\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:54:01.499125 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:01.498957    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d-tls-certs\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:54:01.924325 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:01.924288    2566 generic.go:356] "Generic (PLEG): container finished" podID="30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d" containerID="3ed80a23a04048dbba7d80a8d5e9b77b40000bbe0e57a7fe9adc1893f949bdfb" exitCode=0
Jul 28 04:54:01.924528 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:01.924337    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-1b1e7a/test-model-1b1e7a-kserve-b4ff58867-bg5m9"
Jul 28 04:54:01.924528 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:01.924379    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1b1e7a/test-model-1b1e7a-kserve-b4ff58867-bg5m9" event={"ID":"30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d","Type":"ContainerDied","Data":"3ed80a23a04048dbba7d80a8d5e9b77b40000bbe0e57a7fe9adc1893f949bdfb"}
Jul 28 04:54:01.924528 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:01.924432    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-1b1e7a/test-model-1b1e7a-kserve-b4ff58867-bg5m9" event={"ID":"30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d","Type":"ContainerDied","Data":"d6ce48ce2d2b3f11dab52cc5bd998faefe969d034de30c07ede73c991280f195"}
Jul 28 04:54:01.924528 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:01.924455    2566 scope.go:117] "RemoveContainer" containerID="3ed80a23a04048dbba7d80a8d5e9b77b40000bbe0e57a7fe9adc1893f949bdfb"
Jul 28 04:54:01.933440 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:01.933421    2566 scope.go:117] "RemoveContainer" containerID="3ed80a23a04048dbba7d80a8d5e9b77b40000bbe0e57a7fe9adc1893f949bdfb"
Jul 28 04:54:01.933717 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:54:01.933699    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ed80a23a04048dbba7d80a8d5e9b77b40000bbe0e57a7fe9adc1893f949bdfb\": container with ID starting with 3ed80a23a04048dbba7d80a8d5e9b77b40000bbe0e57a7fe9adc1893f949bdfb not found: ID does not exist" containerID="3ed80a23a04048dbba7d80a8d5e9b77b40000bbe0e57a7fe9adc1893f949bdfb"
Jul 28 04:54:01.933780 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:01.933726    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ed80a23a04048dbba7d80a8d5e9b77b40000bbe0e57a7fe9adc1893f949bdfb"} err="failed to get container status \"3ed80a23a04048dbba7d80a8d5e9b77b40000bbe0e57a7fe9adc1893f949bdfb\": rpc error: code = NotFound desc = could not find container \"3ed80a23a04048dbba7d80a8d5e9b77b40000bbe0e57a7fe9adc1893f949bdfb\": container with ID starting with 3ed80a23a04048dbba7d80a8d5e9b77b40000bbe0e57a7fe9adc1893f949bdfb not found: ID does not exist"
Jul 28 04:54:01.944560 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:01.944538    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-1b1e7a/test-model-1b1e7a-kserve-b4ff58867-bg5m9"]
Jul 28 04:54:01.948392 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:01.948371    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-1b1e7a/test-model-1b1e7a-kserve-b4ff58867-bg5m9"]
Jul 28 04:54:02.088414 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:02.088379    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d" path="/var/lib/kubelet/pods/30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d/volumes"
Jul 28 04:54:07.597304 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:07.597265    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588"]
Jul 28 04:54:07.597746 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:07.597489    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588" podUID="2dd74d22-d2b4-47ec-acac-5d60279c7c59" containerName="istio-proxy" containerID="cri-o://7d745e49486875a349818d281d277ac8671ead2e4f17a42840dcd982ef073193" gracePeriod=30
Jul 28 04:54:12.852825 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:12.852800    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588"
Jul 28 04:54:12.908091 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:12.908008    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd74d22-d2b4-47ec-acac-5d60279c7c59-credential-socket\") pod \"2dd74d22-d2b4-47ec-acac-5d60279c7c59\" (UID: \"2dd74d22-d2b4-47ec-acac-5d60279c7c59\") "
Jul 28 04:54:12.908091 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:12.908050    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dd74d22-d2b4-47ec-acac-5d60279c7c59-istio-data\") pod \"2dd74d22-d2b4-47ec-acac-5d60279c7c59\" (UID: \"2dd74d22-d2b4-47ec-acac-5d60279c7c59\") "
Jul 28 04:54:12.908091 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:12.908072    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v797q\" (UniqueName: \"kubernetes.io/projected/2dd74d22-d2b4-47ec-acac-5d60279c7c59-kube-api-access-v797q\") pod \"2dd74d22-d2b4-47ec-acac-5d60279c7c59\" (UID: \"2dd74d22-d2b4-47ec-acac-5d60279c7c59\") "
Jul 28 04:54:12.908347 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:12.908104    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dd74d22-d2b4-47ec-acac-5d60279c7c59-istiod-ca-cert\") pod \"2dd74d22-d2b4-47ec-acac-5d60279c7c59\" (UID: \"2dd74d22-d2b4-47ec-acac-5d60279c7c59\") "
Jul 28 04:54:12.908347 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:12.908137    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd74d22-d2b4-47ec-acac-5d60279c7c59-workload-socket\") pod \"2dd74d22-d2b4-47ec-acac-5d60279c7c59\" (UID: \"2dd74d22-d2b4-47ec-acac-5d60279c7c59\") "
Jul 28 04:54:12.908347 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:12.908169    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dd74d22-d2b4-47ec-acac-5d60279c7c59-istio-token\") pod \"2dd74d22-d2b4-47ec-acac-5d60279c7c59\" (UID: \"2dd74d22-d2b4-47ec-acac-5d60279c7c59\") "
Jul 28 04:54:12.908347 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:12.908274    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dd74d22-d2b4-47ec-acac-5d60279c7c59-istio-podinfo\") pod \"2dd74d22-d2b4-47ec-acac-5d60279c7c59\" (UID: \"2dd74d22-d2b4-47ec-acac-5d60279c7c59\") "
Jul 28 04:54:12.908347 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:12.908308    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dd74d22-d2b4-47ec-acac-5d60279c7c59-workload-certs\") pod \"2dd74d22-d2b4-47ec-acac-5d60279c7c59\" (UID: \"2dd74d22-d2b4-47ec-acac-5d60279c7c59\") "
Jul 28 04:54:12.908347 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:12.908347    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dd74d22-d2b4-47ec-acac-5d60279c7c59-istio-envoy\") pod \"2dd74d22-d2b4-47ec-acac-5d60279c7c59\" (UID: \"2dd74d22-d2b4-47ec-acac-5d60279c7c59\") "
Jul 28 04:54:12.908738 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:12.908301    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd74d22-d2b4-47ec-acac-5d60279c7c59-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2dd74d22-d2b4-47ec-acac-5d60279c7c59" (UID: "2dd74d22-d2b4-47ec-acac-5d60279c7c59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:54:12.908738 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:12.908498    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dd74d22-d2b4-47ec-acac-5d60279c7c59-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2dd74d22-d2b4-47ec-acac-5d60279c7c59" (UID: "2dd74d22-d2b4-47ec-acac-5d60279c7c59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 04:54:12.908738 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:12.908490    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd74d22-d2b4-47ec-acac-5d60279c7c59-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2dd74d22-d2b4-47ec-acac-5d60279c7c59" (UID: "2dd74d22-d2b4-47ec-acac-5d60279c7c59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:54:12.908931 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:12.908745    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd74d22-d2b4-47ec-acac-5d60279c7c59-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2dd74d22-d2b4-47ec-acac-5d60279c7c59" (UID: "2dd74d22-d2b4-47ec-acac-5d60279c7c59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:54:12.908931 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:12.908861    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dd74d22-d2b4-47ec-acac-5d60279c7c59-workload-certs\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:54:12.908931 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:12.908880    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd74d22-d2b4-47ec-acac-5d60279c7c59-credential-socket\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:54:12.908931 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:12.908895    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dd74d22-d2b4-47ec-acac-5d60279c7c59-istiod-ca-cert\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:54:12.908931 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:12.908907    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd74d22-d2b4-47ec-acac-5d60279c7c59-workload-socket\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:54:12.909121 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:12.908934    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd74d22-d2b4-47ec-acac-5d60279c7c59-istio-data" (OuterVolumeSpecName: "istio-data") pod "2dd74d22-d2b4-47ec-acac-5d60279c7c59" (UID: "2dd74d22-d2b4-47ec-acac-5d60279c7c59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:54:12.910333 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:12.910308    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dd74d22-d2b4-47ec-acac-5d60279c7c59-kube-api-access-v797q" (OuterVolumeSpecName: "kube-api-access-v797q") pod "2dd74d22-d2b4-47ec-acac-5d60279c7c59" (UID: "2dd74d22-d2b4-47ec-acac-5d60279c7c59"). InnerVolumeSpecName "kube-api-access-v797q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:54:12.910575 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:12.910558    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2dd74d22-d2b4-47ec-acac-5d60279c7c59-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2dd74d22-d2b4-47ec-acac-5d60279c7c59" (UID: "2dd74d22-d2b4-47ec-acac-5d60279c7c59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 04:54:12.910944 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:12.910918    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd74d22-d2b4-47ec-acac-5d60279c7c59-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2dd74d22-d2b4-47ec-acac-5d60279c7c59" (UID: "2dd74d22-d2b4-47ec-acac-5d60279c7c59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:54:12.910944 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:12.910924    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dd74d22-d2b4-47ec-acac-5d60279c7c59-istio-token" (OuterVolumeSpecName: "istio-token") pod "2dd74d22-d2b4-47ec-acac-5d60279c7c59" (UID: "2dd74d22-d2b4-47ec-acac-5d60279c7c59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:54:12.980181 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:12.980145    2566 generic.go:356] "Generic (PLEG): container finished" podID="2dd74d22-d2b4-47ec-acac-5d60279c7c59" containerID="7d745e49486875a349818d281d277ac8671ead2e4f17a42840dcd982ef073193" exitCode=0
Jul 28 04:54:12.980353 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:12.980190    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588"
Jul 28 04:54:12.980353 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:12.980225    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588" event={"ID":"2dd74d22-d2b4-47ec-acac-5d60279c7c59","Type":"ContainerDied","Data":"7d745e49486875a349818d281d277ac8671ead2e4f17a42840dcd982ef073193"}
Jul 28 04:54:12.980353 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:12.980280    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588" event={"ID":"2dd74d22-d2b4-47ec-acac-5d60279c7c59","Type":"ContainerDied","Data":"ea227a02c3e7df85d8973ba84764f69c4cf437b6158499a1e36a975c92679a4e"}
Jul 28 04:54:12.980353 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:12.980306    2566 scope.go:117] "RemoveContainer" containerID="7d745e49486875a349818d281d277ac8671ead2e4f17a42840dcd982ef073193"
Jul 28 04:54:12.989604 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:12.989584    2566 scope.go:117] "RemoveContainer" containerID="d30beb77beb43961d5a4ae6c025b99b2702bd69e2730ba444a28ea756c13197e"
Jul 28 04:54:12.998231 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:12.998211    2566 scope.go:117] "RemoveContainer" containerID="7d745e49486875a349818d281d277ac8671ead2e4f17a42840dcd982ef073193"
Jul 28 04:54:12.998524 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:54:12.998503    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d745e49486875a349818d281d277ac8671ead2e4f17a42840dcd982ef073193\": container with ID starting with 7d745e49486875a349818d281d277ac8671ead2e4f17a42840dcd982ef073193 not found: ID does not exist" containerID="7d745e49486875a349818d281d277ac8671ead2e4f17a42840dcd982ef073193"
Jul 28 04:54:12.998578 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:12.998535    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d745e49486875a349818d281d277ac8671ead2e4f17a42840dcd982ef073193"} err="failed to get container status \"7d745e49486875a349818d281d277ac8671ead2e4f17a42840dcd982ef073193\": rpc error: code = NotFound desc = could not find container \"7d745e49486875a349818d281d277ac8671ead2e4f17a42840dcd982ef073193\": container with ID starting with 7d745e49486875a349818d281d277ac8671ead2e4f17a42840dcd982ef073193 not found: ID does not exist"
Jul 28 04:54:12.998578 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:12.998555    2566 scope.go:117] "RemoveContainer" containerID="d30beb77beb43961d5a4ae6c025b99b2702bd69e2730ba444a28ea756c13197e"
Jul 28 04:54:12.998784 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:54:12.998766    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d30beb77beb43961d5a4ae6c025b99b2702bd69e2730ba444a28ea756c13197e\": container with ID starting with d30beb77beb43961d5a4ae6c025b99b2702bd69e2730ba444a28ea756c13197e not found: ID does not exist" containerID="d30beb77beb43961d5a4ae6c025b99b2702bd69e2730ba444a28ea756c13197e"
Jul 28 04:54:12.998863 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:12.998795    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d30beb77beb43961d5a4ae6c025b99b2702bd69e2730ba444a28ea756c13197e"} err="failed to get container status \"d30beb77beb43961d5a4ae6c025b99b2702bd69e2730ba444a28ea756c13197e\": rpc error: code = NotFound desc = could not find container \"d30beb77beb43961d5a4ae6c025b99b2702bd69e2730ba444a28ea756c13197e\": container with ID starting with d30beb77beb43961d5a4ae6c025b99b2702bd69e2730ba444a28ea756c13197e not found: ID does not exist"
Jul 28 04:54:13.002941 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:13.002911    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588"]
Jul 28 04:54:13.004864 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:13.004830    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-20ba8c-openshift-default-68c89854-kx588"]
Jul 28 04:54:13.009504 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:13.009486    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dd74d22-d2b4-47ec-acac-5d60279c7c59-istio-podinfo\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:54:13.009572 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:13.009505    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dd74d22-d2b4-47ec-acac-5d60279c7c59-istio-envoy\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:54:13.009572 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:13.009516    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dd74d22-d2b4-47ec-acac-5d60279c7c59-istio-data\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:54:13.009572 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:13.009524    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v797q\" (UniqueName: \"kubernetes.io/projected/2dd74d22-d2b4-47ec-acac-5d60279c7c59-kube-api-access-v797q\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:54:13.009572 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:13.009533    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dd74d22-d2b4-47ec-acac-5d60279c7c59-istio-token\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:54:14.089055 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:14.089020    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dd74d22-d2b4-47ec-acac-5d60279c7c59" path="/var/lib/kubelet/pods/2dd74d22-d2b4-47ec-acac-5d60279c7c59/volumes"
Jul 28 04:54:18.429245 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:18.429212    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1b1e7a-78679fcd47-whqrx"]
Jul 28 04:54:18.429729 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:18.429473    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1b1e7a-78679fcd47-whqrx" podUID="e7a32da6-b18e-4f7a-9fcd-60822e49a2f2" containerName="maas-api" containerID="cri-o://02b13fa66abf7922e1188372ea3dc66e6f813627846344dc315cda1ce7d915b5" gracePeriod=30
Jul 28 04:54:18.477420 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:18.476647    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1b1e7a-b555b7f7d-85gfb"]
Jul 28 04:54:18.477420 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:18.476909    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-1b1e7a-b555b7f7d-85gfb" podUID="65845d6f-9e4f-4150-85a8-bf22f2c6f47d" containerName="payload-processing" containerID="cri-o://7a113745eb195f969f5ecadfc037a76e978bba5422add518e0b895ac30069a5d" gracePeriod=30
Jul 28 04:54:18.749434 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:18.749411    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1b1e7a-78679fcd47-whqrx"
Jul 28 04:54:18.752074 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:18.752054    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1b1e7a-b555b7f7d-85gfb"
Jul 28 04:54:18.865745 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:18.865717    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65845d6f-9e4f-4150-85a8-bf22f2c6f47d-plugins-config-volume\") pod \"65845d6f-9e4f-4150-85a8-bf22f2c6f47d\" (UID: \"65845d6f-9e4f-4150-85a8-bf22f2c6f47d\") "
Jul 28 04:54:18.865930 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:18.865776    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4q62\" (UniqueName: \"kubernetes.io/projected/65845d6f-9e4f-4150-85a8-bf22f2c6f47d-kube-api-access-c4q62\") pod \"65845d6f-9e4f-4150-85a8-bf22f2c6f47d\" (UID: \"65845d6f-9e4f-4150-85a8-bf22f2c6f47d\") "
Jul 28 04:54:18.865930 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:18.865807    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7a32da6-b18e-4f7a-9fcd-60822e49a2f2-maas-api-tls\") pod \"e7a32da6-b18e-4f7a-9fcd-60822e49a2f2\" (UID: \"e7a32da6-b18e-4f7a-9fcd-60822e49a2f2\") "
Jul 28 04:54:18.865930 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:18.865860    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7c8pl\" (UniqueName: \"kubernetes.io/projected/e7a32da6-b18e-4f7a-9fcd-60822e49a2f2-kube-api-access-7c8pl\") pod \"e7a32da6-b18e-4f7a-9fcd-60822e49a2f2\" (UID: \"e7a32da6-b18e-4f7a-9fcd-60822e49a2f2\") "
Jul 28 04:54:18.866060 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:18.865933    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7a32da6-b18e-4f7a-9fcd-60822e49a2f2-openshift-service-ca\") pod \"e7a32da6-b18e-4f7a-9fcd-60822e49a2f2\" (UID: \"e7a32da6-b18e-4f7a-9fcd-60822e49a2f2\") "
Jul 28 04:54:18.866333 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:18.866164    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65845d6f-9e4f-4150-85a8-bf22f2c6f47d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "65845d6f-9e4f-4150-85a8-bf22f2c6f47d" (UID: "65845d6f-9e4f-4150-85a8-bf22f2c6f47d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 04:54:18.866448 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:18.866333    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7a32da6-b18e-4f7a-9fcd-60822e49a2f2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e7a32da6-b18e-4f7a-9fcd-60822e49a2f2" (UID: "e7a32da6-b18e-4f7a-9fcd-60822e49a2f2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 04:54:18.867997 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:18.867974    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7a32da6-b18e-4f7a-9fcd-60822e49a2f2-kube-api-access-7c8pl" (OuterVolumeSpecName: "kube-api-access-7c8pl") pod "e7a32da6-b18e-4f7a-9fcd-60822e49a2f2" (UID: "e7a32da6-b18e-4f7a-9fcd-60822e49a2f2"). InnerVolumeSpecName "kube-api-access-7c8pl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:54:18.868081 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:18.868005    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7a32da6-b18e-4f7a-9fcd-60822e49a2f2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e7a32da6-b18e-4f7a-9fcd-60822e49a2f2" (UID: "e7a32da6-b18e-4f7a-9fcd-60822e49a2f2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 04:54:18.868119 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:18.868072    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65845d6f-9e4f-4150-85a8-bf22f2c6f47d-kube-api-access-c4q62" (OuterVolumeSpecName: "kube-api-access-c4q62") pod "65845d6f-9e4f-4150-85a8-bf22f2c6f47d" (UID: "65845d6f-9e4f-4150-85a8-bf22f2c6f47d"). InnerVolumeSpecName "kube-api-access-c4q62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:54:18.966945 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:18.966913    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e7a32da6-b18e-4f7a-9fcd-60822e49a2f2-maas-api-tls\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:54:18.966945 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:18.966941    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c8pl\" (UniqueName: \"kubernetes.io/projected/e7a32da6-b18e-4f7a-9fcd-60822e49a2f2-kube-api-access-7c8pl\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:54:18.966945 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:18.966952    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7a32da6-b18e-4f7a-9fcd-60822e49a2f2-openshift-service-ca\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:54:18.967178 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:18.966962    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65845d6f-9e4f-4150-85a8-bf22f2c6f47d-plugins-config-volume\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:54:18.967178 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:18.966972    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4q62\" (UniqueName: \"kubernetes.io/projected/65845d6f-9e4f-4150-85a8-bf22f2c6f47d-kube-api-access-c4q62\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:54:19.013507 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:19.013422    2566 generic.go:356] "Generic (PLEG): container finished" podID="65845d6f-9e4f-4150-85a8-bf22f2c6f47d" containerID="7a113745eb195f969f5ecadfc037a76e978bba5422add518e0b895ac30069a5d" exitCode=0
Jul 28 04:54:19.013507 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:19.013469    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1b1e7a-b555b7f7d-85gfb"
Jul 28 04:54:19.013698 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:19.013514    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1b1e7a-b555b7f7d-85gfb" event={"ID":"65845d6f-9e4f-4150-85a8-bf22f2c6f47d","Type":"ContainerDied","Data":"7a113745eb195f969f5ecadfc037a76e978bba5422add518e0b895ac30069a5d"}
Jul 28 04:54:19.013698 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:19.013569    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1b1e7a-b555b7f7d-85gfb" event={"ID":"65845d6f-9e4f-4150-85a8-bf22f2c6f47d","Type":"ContainerDied","Data":"d80437528976fe97897e64dd37e84178e55ab9ccebad5d1d2969683ac70d00f2"}
Jul 28 04:54:19.013698 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:19.013600    2566 scope.go:117] "RemoveContainer" containerID="7a113745eb195f969f5ecadfc037a76e978bba5422add518e0b895ac30069a5d"
Jul 28 04:54:19.015042 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:19.014963    2566 generic.go:356] "Generic (PLEG): container finished" podID="e7a32da6-b18e-4f7a-9fcd-60822e49a2f2" containerID="02b13fa66abf7922e1188372ea3dc66e6f813627846344dc315cda1ce7d915b5" exitCode=0
Jul 28 04:54:19.015042 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:19.015005    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1b1e7a-78679fcd47-whqrx"
Jul 28 04:54:19.015042 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:19.015013    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1b1e7a-78679fcd47-whqrx" event={"ID":"e7a32da6-b18e-4f7a-9fcd-60822e49a2f2","Type":"ContainerDied","Data":"02b13fa66abf7922e1188372ea3dc66e6f813627846344dc315cda1ce7d915b5"}
Jul 28 04:54:19.015272 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:19.015067    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1b1e7a-78679fcd47-whqrx" event={"ID":"e7a32da6-b18e-4f7a-9fcd-60822e49a2f2","Type":"ContainerDied","Data":"09a4a7894211585d6e9290ec876ba55363936fd15a7a3812e3de2507ed1dc115"}
Jul 28 04:54:19.023425 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:19.023385    2566 scope.go:117] "RemoveContainer" containerID="7a113745eb195f969f5ecadfc037a76e978bba5422add518e0b895ac30069a5d"
Jul 28 04:54:19.023722 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:54:19.023699    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a113745eb195f969f5ecadfc037a76e978bba5422add518e0b895ac30069a5d\": container with ID starting with 7a113745eb195f969f5ecadfc037a76e978bba5422add518e0b895ac30069a5d not found: ID does not exist" containerID="7a113745eb195f969f5ecadfc037a76e978bba5422add518e0b895ac30069a5d"
Jul 28 04:54:19.023798 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:19.023734    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a113745eb195f969f5ecadfc037a76e978bba5422add518e0b895ac30069a5d"} err="failed to get container status \"7a113745eb195f969f5ecadfc037a76e978bba5422add518e0b895ac30069a5d\": rpc error: code = NotFound desc = could not find container \"7a113745eb195f969f5ecadfc037a76e978bba5422add518e0b895ac30069a5d\": container with ID starting with 7a113745eb195f969f5ecadfc037a76e978bba5422add518e0b895ac30069a5d not found: ID does not exist"
Jul 28 04:54:19.023798 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:19.023753    2566 scope.go:117] "RemoveContainer" containerID="02b13fa66abf7922e1188372ea3dc66e6f813627846344dc315cda1ce7d915b5"
Jul 28 04:54:19.031783 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:19.031766    2566 scope.go:117] "RemoveContainer" containerID="02b13fa66abf7922e1188372ea3dc66e6f813627846344dc315cda1ce7d915b5"
Jul 28 04:54:19.032049 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:54:19.032031    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02b13fa66abf7922e1188372ea3dc66e6f813627846344dc315cda1ce7d915b5\": container with ID starting with 02b13fa66abf7922e1188372ea3dc66e6f813627846344dc315cda1ce7d915b5 not found: ID does not exist" containerID="02b13fa66abf7922e1188372ea3dc66e6f813627846344dc315cda1ce7d915b5"
Jul 28 04:54:19.032094 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:19.032056    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02b13fa66abf7922e1188372ea3dc66e6f813627846344dc315cda1ce7d915b5"} err="failed to get container status \"02b13fa66abf7922e1188372ea3dc66e6f813627846344dc315cda1ce7d915b5\": rpc error: code = NotFound desc = could not find container \"02b13fa66abf7922e1188372ea3dc66e6f813627846344dc315cda1ce7d915b5\": container with ID starting with 02b13fa66abf7922e1188372ea3dc66e6f813627846344dc315cda1ce7d915b5 not found: ID does not exist"
Jul 28 04:54:19.039833 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:19.039718    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1b1e7a-b555b7f7d-85gfb"]
Jul 28 04:54:19.042127 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:19.042106    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1b1e7a-b555b7f7d-85gfb"]
Jul 28 04:54:19.051884 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:19.051862    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1b1e7a-78679fcd47-whqrx"]
Jul 28 04:54:19.055256 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:19.055235    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1b1e7a-78679fcd47-whqrx"]
Jul 28 04:54:20.092685 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:20.092648    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65845d6f-9e4f-4150-85a8-bf22f2c6f47d" path="/var/lib/kubelet/pods/65845d6f-9e4f-4150-85a8-bf22f2c6f47d/volumes"
Jul 28 04:54:20.093104 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:20.093089    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7a32da6-b18e-4f7a-9fcd-60822e49a2f2" path="/var/lib/kubelet/pods/e7a32da6-b18e-4f7a-9fcd-60822e49a2f2/volumes"
Jul 28 04:54:30.487355 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.487319    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl"]
Jul 28 04:54:30.487940 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.487919    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cd86c82-7f61-4828-b7c3-1be9fac77fc4" containerName="maas-api"
Jul 28 04:54:30.487998 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.487943    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cd86c82-7f61-4828-b7c3-1be9fac77fc4" containerName="maas-api"
Jul 28 04:54:30.487998 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.487955    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7a32da6-b18e-4f7a-9fcd-60822e49a2f2" containerName="maas-api"
Jul 28 04:54:30.487998 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.487964    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7a32da6-b18e-4f7a-9fcd-60822e49a2f2" containerName="maas-api"
Jul 28 04:54:30.487998 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.487975    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dd74d22-d2b4-47ec-acac-5d60279c7c59" containerName="istio-proxy"
Jul 28 04:54:30.487998 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.487985    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dd74d22-d2b4-47ec-acac-5d60279c7c59" containerName="istio-proxy"
Jul 28 04:54:30.488144 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.488015    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbdb2eb2-865e-4bdf-8bd5-d851edd6c10f" containerName="payload-pre-processing"
Jul 28 04:54:30.488144 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.488024    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbdb2eb2-865e-4bdf-8bd5-d851edd6c10f" containerName="payload-pre-processing"
Jul 28 04:54:30.488144 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.488034    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4342a471-1d30-4a2f-8a62-ed51288f8385" containerName="payload-processing"
Jul 28 04:54:30.488144 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.488042    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4342a471-1d30-4a2f-8a62-ed51288f8385" containerName="payload-processing"
Jul 28 04:54:30.488144 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.488062    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d" containerName="main"
Jul 28 04:54:30.488144 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.488070    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d" containerName="main"
Jul 28 04:54:30.488144 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.488083    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65845d6f-9e4f-4150-85a8-bf22f2c6f47d" containerName="payload-processing"
Jul 28 04:54:30.488144 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.488091    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="65845d6f-9e4f-4150-85a8-bf22f2c6f47d" containerName="payload-processing"
Jul 28 04:54:30.488377 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.488184    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="30c5c4d6-fcb2-4fa1-9e7e-24d1b0eaf40d" containerName="main"
Jul 28 04:54:30.488377 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.488199    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbdb2eb2-865e-4bdf-8bd5-d851edd6c10f" containerName="payload-pre-processing"
Jul 28 04:54:30.488377 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.488209    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dd74d22-d2b4-47ec-acac-5d60279c7c59" containerName="istio-proxy"
Jul 28 04:54:30.488377 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.488218    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cd86c82-7f61-4828-b7c3-1be9fac77fc4" containerName="maas-api"
Jul 28 04:54:30.488377 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.488229    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dd74d22-d2b4-47ec-acac-5d60279c7c59" containerName="istio-proxy"
Jul 28 04:54:30.488377 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.488239    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7a32da6-b18e-4f7a-9fcd-60822e49a2f2" containerName="maas-api"
Jul 28 04:54:30.488377 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.488249    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="65845d6f-9e4f-4150-85a8-bf22f2c6f47d" containerName="payload-processing"
Jul 28 04:54:30.488377 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.488259    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4342a471-1d30-4a2f-8a62-ed51288f8385" containerName="payload-processing"
Jul 28 04:54:30.491557 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.491537    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl"
Jul 28 04:54:30.494497 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.494476    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 28 04:54:30.494616 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.494495    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f397e0-openshift-default-dockercfg-gbt9w\""
Jul 28 04:54:30.501952 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.501929    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl"]
Jul 28 04:54:30.673446 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.673407    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4eba9e9-5195-4bd9-a764-d756c44c1c15-istiod-ca-cert\") pod \"e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl\" (UID: \"e4eba9e9-5195-4bd9-a764-d756c44c1c15\") " pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl"
Jul 28 04:54:30.673627 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.673461    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nz48\" (UniqueName: \"kubernetes.io/projected/e4eba9e9-5195-4bd9-a764-d756c44c1c15-kube-api-access-6nz48\") pod \"e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl\" (UID: \"e4eba9e9-5195-4bd9-a764-d756c44c1c15\") " pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl"
Jul 28 04:54:30.673627 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.673506    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4eba9e9-5195-4bd9-a764-d756c44c1c15-istio-podinfo\") pod \"e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl\" (UID: \"e4eba9e9-5195-4bd9-a764-d756c44c1c15\") " pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl"
Jul 28 04:54:30.673627 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.673585    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4eba9e9-5195-4bd9-a764-d756c44c1c15-istio-token\") pod \"e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl\" (UID: \"e4eba9e9-5195-4bd9-a764-d756c44c1c15\") " pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl"
Jul 28 04:54:30.673834 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.673623    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4eba9e9-5195-4bd9-a764-d756c44c1c15-workload-socket\") pod \"e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl\" (UID: \"e4eba9e9-5195-4bd9-a764-d756c44c1c15\") " pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl"
Jul 28 04:54:30.673834 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.673671    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4eba9e9-5195-4bd9-a764-d756c44c1c15-credential-socket\") pod \"e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl\" (UID: \"e4eba9e9-5195-4bd9-a764-d756c44c1c15\") " pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl"
Jul 28 04:54:30.673834 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.673701    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4eba9e9-5195-4bd9-a764-d756c44c1c15-workload-certs\") pod \"e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl\" (UID: \"e4eba9e9-5195-4bd9-a764-d756c44c1c15\") " pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl"
Jul 28 04:54:30.673834 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.673742    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4eba9e9-5195-4bd9-a764-d756c44c1c15-istio-envoy\") pod \"e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl\" (UID: \"e4eba9e9-5195-4bd9-a764-d756c44c1c15\") " pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl"
Jul 28 04:54:30.674090 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.673863    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4eba9e9-5195-4bd9-a764-d756c44c1c15-istio-data\") pod \"e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl\" (UID: \"e4eba9e9-5195-4bd9-a764-d756c44c1c15\") " pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl"
Jul 28 04:54:30.774680 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.774646    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4eba9e9-5195-4bd9-a764-d756c44c1c15-istiod-ca-cert\") pod \"e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl\" (UID: \"e4eba9e9-5195-4bd9-a764-d756c44c1c15\") " pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl"
Jul 28 04:54:30.774899 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.774686    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nz48\" (UniqueName: \"kubernetes.io/projected/e4eba9e9-5195-4bd9-a764-d756c44c1c15-kube-api-access-6nz48\") pod \"e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl\" (UID: \"e4eba9e9-5195-4bd9-a764-d756c44c1c15\") " pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl"
Jul 28 04:54:30.774899 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.774722    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4eba9e9-5195-4bd9-a764-d756c44c1c15-istio-podinfo\") pod \"e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl\" (UID: \"e4eba9e9-5195-4bd9-a764-d756c44c1c15\") " pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl"
Jul 28 04:54:30.774899 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.774787    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4eba9e9-5195-4bd9-a764-d756c44c1c15-istio-token\") pod \"e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl\" (UID: \"e4eba9e9-5195-4bd9-a764-d756c44c1c15\") " pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl"
Jul 28 04:54:30.774899 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.774818    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4eba9e9-5195-4bd9-a764-d756c44c1c15-workload-socket\") pod \"e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl\" (UID: \"e4eba9e9-5195-4bd9-a764-d756c44c1c15\") " pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl"
Jul 28 04:54:30.775133 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.774997    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4eba9e9-5195-4bd9-a764-d756c44c1c15-credential-socket\") pod \"e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl\" (UID: \"e4eba9e9-5195-4bd9-a764-d756c44c1c15\") " pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl"
Jul 28 04:54:30.775133 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.775042    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4eba9e9-5195-4bd9-a764-d756c44c1c15-workload-certs\") pod \"e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl\" (UID: \"e4eba9e9-5195-4bd9-a764-d756c44c1c15\") " pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl"
Jul 28 04:54:30.775133 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.775090    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4eba9e9-5195-4bd9-a764-d756c44c1c15-istio-envoy\") pod \"e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl\" (UID: \"e4eba9e9-5195-4bd9-a764-d756c44c1c15\") " pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl"
Jul 28 04:54:30.775285 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.775144    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4eba9e9-5195-4bd9-a764-d756c44c1c15-istio-data\") pod \"e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl\" (UID: \"e4eba9e9-5195-4bd9-a764-d756c44c1c15\") " pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl"
Jul 28 04:54:30.775285 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.775199    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4eba9e9-5195-4bd9-a764-d756c44c1c15-workload-socket\") pod \"e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl\" (UID: \"e4eba9e9-5195-4bd9-a764-d756c44c1c15\") " pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl"
Jul 28 04:54:30.775384 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.775348    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4eba9e9-5195-4bd9-a764-d756c44c1c15-credential-socket\") pod \"e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl\" (UID: \"e4eba9e9-5195-4bd9-a764-d756c44c1c15\") " pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl"
Jul 28 04:54:30.775441 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.775401    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4eba9e9-5195-4bd9-a764-d756c44c1c15-workload-certs\") pod \"e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl\" (UID: \"e4eba9e9-5195-4bd9-a764-d756c44c1c15\") " pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl"
Jul 28 04:54:30.775499 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.775466    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4eba9e9-5195-4bd9-a764-d756c44c1c15-istio-data\") pod \"e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl\" (UID: \"e4eba9e9-5195-4bd9-a764-d756c44c1c15\") " pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl"
Jul 28 04:54:30.775499 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.775484    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4eba9e9-5195-4bd9-a764-d756c44c1c15-istiod-ca-cert\") pod \"e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl\" (UID: \"e4eba9e9-5195-4bd9-a764-d756c44c1c15\") " pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl"
Jul 28 04:54:30.777266 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.777242    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4eba9e9-5195-4bd9-a764-d756c44c1c15-istio-envoy\") pod \"e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl\" (UID: \"e4eba9e9-5195-4bd9-a764-d756c44c1c15\") " pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl"
Jul 28 04:54:30.777365 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.777302    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4eba9e9-5195-4bd9-a764-d756c44c1c15-istio-podinfo\") pod \"e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl\" (UID: \"e4eba9e9-5195-4bd9-a764-d756c44c1c15\") " pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl"
Jul 28 04:54:30.782682 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.782656    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4eba9e9-5195-4bd9-a764-d756c44c1c15-istio-token\") pod \"e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl\" (UID: \"e4eba9e9-5195-4bd9-a764-d756c44c1c15\") " pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl"
Jul 28 04:54:30.782780 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.782763    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nz48\" (UniqueName: \"kubernetes.io/projected/e4eba9e9-5195-4bd9-a764-d756c44c1c15-kube-api-access-6nz48\") pod \"e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl\" (UID: \"e4eba9e9-5195-4bd9-a764-d756c44c1c15\") " pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl"
Jul 28 04:54:30.802355 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.802331    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl"
Jul 28 04:54:30.943491 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.943463    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl"]
Jul 28 04:54:30.945231 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:54:30.945194    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4eba9e9_5195_4bd9_a764_d756c44c1c15.slice/crio-e5282d20b27ec527558edc57f90f7df804744412213397e80f1f633d74d758a1 WatchSource:0}: Error finding container e5282d20b27ec527558edc57f90f7df804744412213397e80f1f633d74d758a1: Status 404 returned error can't find the container with id e5282d20b27ec527558edc57f90f7df804744412213397e80f1f633d74d758a1
Jul 28 04:54:30.947563 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.947523    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:54:30.947637 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.947616    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:54:30.947680 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:30.947659    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:54:31.076676 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:31.076565    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl" event={"ID":"e4eba9e9-5195-4bd9-a764-d756c44c1c15","Type":"ContainerStarted","Data":"a63ba5204c0d21a9b3724beb728d446809819901e459aecc71861a4e2de20dbc"}
Jul 28 04:54:31.076676 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:31.076649    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl" event={"ID":"e4eba9e9-5195-4bd9-a764-d756c44c1c15","Type":"ContainerStarted","Data":"e5282d20b27ec527558edc57f90f7df804744412213397e80f1f633d74d758a1"}
Jul 28 04:54:31.095755 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:31.095702    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl" podStartSLOduration=1.095686218 podStartE2EDuration="1.095686218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:54:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:54:31.094135365 +0000 UTC m=+2571.484311386" watchObservedRunningTime="2026-07-28 04:54:31.095686218 +0000 UTC m=+2571.485862284"
Jul 28 04:54:31.802711 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:31.802671    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl"
Jul 28 04:54:32.808744 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:32.808716    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl"
Jul 28 04:54:33.086869 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:33.086780    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl"
Jul 28 04:54:33.087726 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:33.087706    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl"
Jul 28 04:54:33.281979 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:33.281942    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f397e0-755589554-c8twz"]
Jul 28 04:54:33.282520 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:33.282504    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dd74d22-d2b4-47ec-acac-5d60279c7c59" containerName="istio-proxy"
Jul 28 04:54:33.282561 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:33.282525    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dd74d22-d2b4-47ec-acac-5d60279c7c59" containerName="istio-proxy"
Jul 28 04:54:33.286210 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:33.286186    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f397e0-755589554-c8twz"
Jul 28 04:54:33.288900 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:33.288871    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-f397e0-serving-cert\""
Jul 28 04:54:33.289022 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:33.289001    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gfn77\""
Jul 28 04:54:33.289022 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:33.289015    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 04:54:33.289271 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:33.289257    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 04:54:33.295463 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:33.295436    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f397e0-755589554-c8twz"]
Jul 28 04:54:33.400637 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:33.400544    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e615277-b304-43cc-8057-40bee75edcc6-maas-api-tls\") pod \"maas-api-e2e-api-a-f397e0-755589554-c8twz\" (UID: \"9e615277-b304-43cc-8057-40bee75edcc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f397e0-755589554-c8twz"
Jul 28 04:54:33.400637 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:33.400623    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txwkg\" (UniqueName: \"kubernetes.io/projected/9e615277-b304-43cc-8057-40bee75edcc6-kube-api-access-txwkg\") pod \"maas-api-e2e-api-a-f397e0-755589554-c8twz\" (UID: \"9e615277-b304-43cc-8057-40bee75edcc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f397e0-755589554-c8twz"
Jul 28 04:54:33.400877 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:33.400721    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e615277-b304-43cc-8057-40bee75edcc6-openshift-service-ca\") pod \"maas-api-e2e-api-a-f397e0-755589554-c8twz\" (UID: \"9e615277-b304-43cc-8057-40bee75edcc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f397e0-755589554-c8twz"
Jul 28 04:54:33.502008 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:33.501978    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e615277-b304-43cc-8057-40bee75edcc6-maas-api-tls\") pod \"maas-api-e2e-api-a-f397e0-755589554-c8twz\" (UID: \"9e615277-b304-43cc-8057-40bee75edcc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f397e0-755589554-c8twz"
Jul 28 04:54:33.502148 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:33.502029    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txwkg\" (UniqueName: \"kubernetes.io/projected/9e615277-b304-43cc-8057-40bee75edcc6-kube-api-access-txwkg\") pod \"maas-api-e2e-api-a-f397e0-755589554-c8twz\" (UID: \"9e615277-b304-43cc-8057-40bee75edcc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f397e0-755589554-c8twz"
Jul 28 04:54:33.502148 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:33.502095    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e615277-b304-43cc-8057-40bee75edcc6-openshift-service-ca\") pod \"maas-api-e2e-api-a-f397e0-755589554-c8twz\" (UID: \"9e615277-b304-43cc-8057-40bee75edcc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f397e0-755589554-c8twz"
Jul 28 04:54:33.502148 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:54:33.502128    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-f397e0-serving-cert: secret "maas-api-e2e-api-a-f397e0-serving-cert" not found
Jul 28 04:54:33.502259 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:54:33.502194    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e615277-b304-43cc-8057-40bee75edcc6-maas-api-tls podName:9e615277-b304-43cc-8057-40bee75edcc6 nodeName:}" failed. No retries permitted until 2026-07-28 04:54:34.002175606 +0000 UTC m=+2574.392351604 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9e615277-b304-43cc-8057-40bee75edcc6-maas-api-tls") pod "maas-api-e2e-api-a-f397e0-755589554-c8twz" (UID: "9e615277-b304-43cc-8057-40bee75edcc6") : secret "maas-api-e2e-api-a-f397e0-serving-cert" not found
Jul 28 04:54:33.502729 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:33.502712    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e615277-b304-43cc-8057-40bee75edcc6-openshift-service-ca\") pod \"maas-api-e2e-api-a-f397e0-755589554-c8twz\" (UID: \"9e615277-b304-43cc-8057-40bee75edcc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f397e0-755589554-c8twz"
Jul 28 04:54:33.511720 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:33.511695    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txwkg\" (UniqueName: \"kubernetes.io/projected/9e615277-b304-43cc-8057-40bee75edcc6-kube-api-access-txwkg\") pod \"maas-api-e2e-api-a-f397e0-755589554-c8twz\" (UID: \"9e615277-b304-43cc-8057-40bee75edcc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f397e0-755589554-c8twz"
Jul 28 04:54:33.884445 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:33.884410    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f397e0-b6bbfc748-f2dv6"]
Jul 28 04:54:33.889725 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:33.889693    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f397e0-b6bbfc748-f2dv6"
Jul 28 04:54:33.893069 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:33.893046    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-f397e0-dockercfg-t8mnz\""
Jul 28 04:54:33.893215 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:33.893071    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-f397e0\""
Jul 28 04:54:33.899966 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:33.899939    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f397e0-b6bbfc748-f2dv6"]
Jul 28 04:54:33.929828 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:33.929801    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f397e0-b897c75dd-z98ng"]
Jul 28 04:54:33.933375 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:33.933357    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f397e0-b897c75dd-z98ng"
Jul 28 04:54:33.941544 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:33.941518    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f397e0-b897c75dd-z98ng"]
Jul 28 04:54:34.007055 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:34.007022    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r42px\" (UniqueName: \"kubernetes.io/projected/a54ddc02-8c8e-4419-9918-c9da251cff0f-kube-api-access-r42px\") pod \"payload-processing-e2e-api-a-f397e0-b897c75dd-z98ng\" (UID: \"a54ddc02-8c8e-4419-9918-c9da251cff0f\") " pod="openshift-ingress/payload-processing-e2e-api-a-f397e0-b897c75dd-z98ng"
Jul 28 04:54:34.007208 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:34.007063    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpmj5\" (UniqueName: \"kubernetes.io/projected/6851d96f-883c-40b0-992d-0db4e2a10fa5-kube-api-access-jpmj5\") pod \"payload-pre-processing-e2e-api-a-f397e0-b6bbfc748-f2dv6\" (UID: \"6851d96f-883c-40b0-992d-0db4e2a10fa5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f397e0-b6bbfc748-f2dv6"
Jul 28 04:54:34.007208 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:34.007095    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e615277-b304-43cc-8057-40bee75edcc6-maas-api-tls\") pod \"maas-api-e2e-api-a-f397e0-755589554-c8twz\" (UID: \"9e615277-b304-43cc-8057-40bee75edcc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f397e0-755589554-c8twz"
Jul 28 04:54:34.007208 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:34.007119    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a54ddc02-8c8e-4419-9918-c9da251cff0f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f397e0-b897c75dd-z98ng\" (UID: \"a54ddc02-8c8e-4419-9918-c9da251cff0f\") " pod="openshift-ingress/payload-processing-e2e-api-a-f397e0-b897c75dd-z98ng"
Jul 28 04:54:34.007208 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:34.007138    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6851d96f-883c-40b0-992d-0db4e2a10fa5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f397e0-b6bbfc748-f2dv6\" (UID: \"6851d96f-883c-40b0-992d-0db4e2a10fa5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f397e0-b6bbfc748-f2dv6"
Jul 28 04:54:34.009600 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:34.009576    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e615277-b304-43cc-8057-40bee75edcc6-maas-api-tls\") pod \"maas-api-e2e-api-a-f397e0-755589554-c8twz\" (UID: \"9e615277-b304-43cc-8057-40bee75edcc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f397e0-755589554-c8twz"
Jul 28 04:54:34.108292 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:34.108263    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r42px\" (UniqueName: \"kubernetes.io/projected/a54ddc02-8c8e-4419-9918-c9da251cff0f-kube-api-access-r42px\") pod \"payload-processing-e2e-api-a-f397e0-b897c75dd-z98ng\" (UID: \"a54ddc02-8c8e-4419-9918-c9da251cff0f\") " pod="openshift-ingress/payload-processing-e2e-api-a-f397e0-b897c75dd-z98ng"
Jul 28 04:54:34.108442 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:34.108309    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpmj5\" (UniqueName: \"kubernetes.io/projected/6851d96f-883c-40b0-992d-0db4e2a10fa5-kube-api-access-jpmj5\") pod \"payload-pre-processing-e2e-api-a-f397e0-b6bbfc748-f2dv6\" (UID: \"6851d96f-883c-40b0-992d-0db4e2a10fa5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f397e0-b6bbfc748-f2dv6"
Jul 28 04:54:34.108442 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:34.108338    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a54ddc02-8c8e-4419-9918-c9da251cff0f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f397e0-b897c75dd-z98ng\" (UID: \"a54ddc02-8c8e-4419-9918-c9da251cff0f\") " pod="openshift-ingress/payload-processing-e2e-api-a-f397e0-b897c75dd-z98ng"
Jul 28 04:54:34.108442 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:34.108367    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6851d96f-883c-40b0-992d-0db4e2a10fa5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f397e0-b6bbfc748-f2dv6\" (UID: \"6851d96f-883c-40b0-992d-0db4e2a10fa5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f397e0-b6bbfc748-f2dv6"
Jul 28 04:54:34.109094 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:34.109071    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a54ddc02-8c8e-4419-9918-c9da251cff0f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f397e0-b897c75dd-z98ng\" (UID: \"a54ddc02-8c8e-4419-9918-c9da251cff0f\") " pod="openshift-ingress/payload-processing-e2e-api-a-f397e0-b897c75dd-z98ng"
Jul 28 04:54:34.109094 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:34.109080    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6851d96f-883c-40b0-992d-0db4e2a10fa5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f397e0-b6bbfc748-f2dv6\" (UID: \"6851d96f-883c-40b0-992d-0db4e2a10fa5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f397e0-b6bbfc748-f2dv6"
Jul 28 04:54:34.117071 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:34.117045    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r42px\" (UniqueName: \"kubernetes.io/projected/a54ddc02-8c8e-4419-9918-c9da251cff0f-kube-api-access-r42px\") pod \"payload-processing-e2e-api-a-f397e0-b897c75dd-z98ng\" (UID: \"a54ddc02-8c8e-4419-9918-c9da251cff0f\") " pod="openshift-ingress/payload-processing-e2e-api-a-f397e0-b897c75dd-z98ng"
Jul 28 04:54:34.117232 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:34.117216    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpmj5\" (UniqueName: \"kubernetes.io/projected/6851d96f-883c-40b0-992d-0db4e2a10fa5-kube-api-access-jpmj5\") pod \"payload-pre-processing-e2e-api-a-f397e0-b6bbfc748-f2dv6\" (UID: \"6851d96f-883c-40b0-992d-0db4e2a10fa5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f397e0-b6bbfc748-f2dv6"
Jul 28 04:54:34.199063 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:34.198990    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f397e0-755589554-c8twz"
Jul 28 04:54:34.202525 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:34.202503    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f397e0-b6bbfc748-f2dv6"
Jul 28 04:54:34.246394 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:34.246129    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f397e0-b897c75dd-z98ng"
Jul 28 04:54:34.364021 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:34.363935    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f397e0-755589554-c8twz"]
Jul 28 04:54:34.371578 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:54:34.371496    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e615277_b304_43cc_8057_40bee75edcc6.slice/crio-50921493c576144ba17ce360c09bd66e92074994d917ffef75eb653511348d45 WatchSource:0}: Error finding container 50921493c576144ba17ce360c09bd66e92074994d917ffef75eb653511348d45: Status 404 returned error can't find the container with id 50921493c576144ba17ce360c09bd66e92074994d917ffef75eb653511348d45
Jul 28 04:54:34.373288 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:34.373216    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 04:54:34.387468 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:34.387435    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f397e0-b6bbfc748-f2dv6"]
Jul 28 04:54:34.389589 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:54:34.389563    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6851d96f_883c_40b0_992d_0db4e2a10fa5.slice/crio-efdc2895facd895c7e2c36a12758aa68bc4422465dc1bd09c994f55b7dbfd8d8 WatchSource:0}: Error finding container efdc2895facd895c7e2c36a12758aa68bc4422465dc1bd09c994f55b7dbfd8d8: Status 404 returned error can't find the container with id efdc2895facd895c7e2c36a12758aa68bc4422465dc1bd09c994f55b7dbfd8d8
Jul 28 04:54:34.419241 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:34.419219    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f397e0-b897c75dd-z98ng"]
Jul 28 04:54:34.424991 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:54:34.424966    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda54ddc02_8c8e_4419_9918_c9da251cff0f.slice/crio-1a90216395e693b5055ca27ec5c7a4ba6f4cc1b58b312ce2d8bfb4c0421c440c WatchSource:0}: Error finding container 1a90216395e693b5055ca27ec5c7a4ba6f4cc1b58b312ce2d8bfb4c0421c440c: Status 404 returned error can't find the container with id 1a90216395e693b5055ca27ec5c7a4ba6f4cc1b58b312ce2d8bfb4c0421c440c
Jul 28 04:54:35.098240 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:35.098202    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f397e0-b6bbfc748-f2dv6" event={"ID":"6851d96f-883c-40b0-992d-0db4e2a10fa5","Type":"ContainerStarted","Data":"1a1d5dc1c184f3a93bd7dfaab67a949a89da11af28ef92f4bf453a99b242a8c3"}
Jul 28 04:54:35.098605 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:35.098256    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f397e0-b6bbfc748-f2dv6" event={"ID":"6851d96f-883c-40b0-992d-0db4e2a10fa5","Type":"ContainerStarted","Data":"efdc2895facd895c7e2c36a12758aa68bc4422465dc1bd09c994f55b7dbfd8d8"}
Jul 28 04:54:35.098605 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:35.098304    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f397e0-b6bbfc748-f2dv6"
Jul 28 04:54:35.099811 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:35.099789    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f397e0-b897c75dd-z98ng" event={"ID":"a54ddc02-8c8e-4419-9918-c9da251cff0f","Type":"ContainerStarted","Data":"865bfe29928e6d58788c9b1aee384ed6bdeb1521deedc1ca218a16bd9415c45d"}
Jul 28 04:54:35.099927 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:35.099828    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f397e0-b897c75dd-z98ng" event={"ID":"a54ddc02-8c8e-4419-9918-c9da251cff0f","Type":"ContainerStarted","Data":"1a90216395e693b5055ca27ec5c7a4ba6f4cc1b58b312ce2d8bfb4c0421c440c"}
Jul 28 04:54:35.099927 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:35.099877    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-f397e0-b897c75dd-z98ng"
Jul 28 04:54:35.101297 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:35.101277    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f397e0-755589554-c8twz" event={"ID":"9e615277-b304-43cc-8057-40bee75edcc6","Type":"ContainerStarted","Data":"98f3768968d6c4e06db0a12266667cd99f2cca8cd8bfae4abc72a4b868947fa0"}
Jul 28 04:54:35.101388 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:35.101321    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f397e0-755589554-c8twz" event={"ID":"9e615277-b304-43cc-8057-40bee75edcc6","Type":"ContainerStarted","Data":"50921493c576144ba17ce360c09bd66e92074994d917ffef75eb653511348d45"}
Jul 28 04:54:35.101530 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:35.101513    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f397e0-755589554-c8twz"
Jul 28 04:54:35.114701 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:35.114631    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f397e0-b6bbfc748-f2dv6" podStartSLOduration=2.114614698 podStartE2EDuration="2.114614698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:54:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:54:35.113163232 +0000 UTC m=+2575.503339253" watchObservedRunningTime="2026-07-28 04:54:35.114614698 +0000 UTC m=+2575.504790722"
Jul 28 04:54:35.131682 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:35.131637    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f397e0-755589554-c8twz" podStartSLOduration=1.552765971 podStartE2EDuration="2.131624177s" totalImagesPullingTime="578.858206ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:54:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:54:35.130517208 +0000 UTC m=+2575.520693230" watchObservedRunningTime="2026-07-28 04:54:35.131624177 +0000 UTC m=+2575.521800234"
Jul 28 04:54:35.145986 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:35.145943    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-f397e0-b897c75dd-z98ng" podStartSLOduration=2.145930403 podStartE2EDuration="2.145930403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:54:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:54:35.144508618 +0000 UTC m=+2575.534684651" watchObservedRunningTime="2026-07-28 04:54:35.145930403 +0000 UTC m=+2575.536106423"
Jul 28 04:54:38.442865 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:38.441943    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c"]
Jul 28 04:54:38.449886 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:38.449828    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c"
Jul 28 04:54:38.452700 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:38.452681    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-1b2859-openshift-default-dockercfg-77dp4\""
Jul 28 04:54:38.459338 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:38.459158    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c"]
Jul 28 04:54:38.551716 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:38.551670    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc23c275-9d40-4d1c-aa91-405814c44687-workload-socket\") pod \"e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c\" (UID: \"bc23c275-9d40-4d1c-aa91-405814c44687\") " pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c"
Jul 28 04:54:38.551940 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:38.551734    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc23c275-9d40-4d1c-aa91-405814c44687-workload-certs\") pod \"e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c\" (UID: \"bc23c275-9d40-4d1c-aa91-405814c44687\") " pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c"
Jul 28 04:54:38.551940 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:38.551823    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc23c275-9d40-4d1c-aa91-405814c44687-istio-podinfo\") pod \"e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c\" (UID: \"bc23c275-9d40-4d1c-aa91-405814c44687\") " pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c"
Jul 28 04:54:38.551940 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:38.551878    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc23c275-9d40-4d1c-aa91-405814c44687-istio-envoy\") pod \"e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c\" (UID: \"bc23c275-9d40-4d1c-aa91-405814c44687\") " pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c"
Jul 28 04:54:38.551940 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:38.551903    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc23c275-9d40-4d1c-aa91-405814c44687-istio-data\") pod \"e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c\" (UID: \"bc23c275-9d40-4d1c-aa91-405814c44687\") " pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c"
Jul 28 04:54:38.552104 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:38.551998    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc23c275-9d40-4d1c-aa91-405814c44687-credential-socket\") pod \"e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c\" (UID: \"bc23c275-9d40-4d1c-aa91-405814c44687\") " pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c"
Jul 28 04:54:38.552104 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:38.552028    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc23c275-9d40-4d1c-aa91-405814c44687-istiod-ca-cert\") pod \"e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c\" (UID: \"bc23c275-9d40-4d1c-aa91-405814c44687\") " pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c"
Jul 28 04:54:38.552184 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:38.552104    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc23c275-9d40-4d1c-aa91-405814c44687-istio-token\") pod \"e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c\" (UID: \"bc23c275-9d40-4d1c-aa91-405814c44687\") " pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c"
Jul 28 04:54:38.552184 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:38.552120    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg4lt\" (UniqueName: \"kubernetes.io/projected/bc23c275-9d40-4d1c-aa91-405814c44687-kube-api-access-qg4lt\") pod \"e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c\" (UID: \"bc23c275-9d40-4d1c-aa91-405814c44687\") " pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c"
Jul 28 04:54:38.652644 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:38.652603    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc23c275-9d40-4d1c-aa91-405814c44687-istio-podinfo\") pod \"e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c\" (UID: \"bc23c275-9d40-4d1c-aa91-405814c44687\") " pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c"
Jul 28 04:54:38.652821 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:38.652656    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc23c275-9d40-4d1c-aa91-405814c44687-istio-envoy\") pod \"e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c\" (UID: \"bc23c275-9d40-4d1c-aa91-405814c44687\") " pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c"
Jul 28 04:54:38.652821 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:38.652681    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc23c275-9d40-4d1c-aa91-405814c44687-istio-data\") pod \"e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c\" (UID: \"bc23c275-9d40-4d1c-aa91-405814c44687\") " pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c"
Jul 28 04:54:38.652821 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:38.652734    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc23c275-9d40-4d1c-aa91-405814c44687-credential-socket\") pod \"e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c\" (UID: \"bc23c275-9d40-4d1c-aa91-405814c44687\") " pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c"
Jul 28 04:54:38.652821 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:38.652755    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc23c275-9d40-4d1c-aa91-405814c44687-istiod-ca-cert\") pod \"e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c\" (UID: \"bc23c275-9d40-4d1c-aa91-405814c44687\") " pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c"
Jul 28 04:54:38.652821 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:38.652809    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc23c275-9d40-4d1c-aa91-405814c44687-istio-token\") pod \"e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c\" (UID: \"bc23c275-9d40-4d1c-aa91-405814c44687\") " pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c"
Jul 28 04:54:38.653132 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:38.652854    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg4lt\" (UniqueName: \"kubernetes.io/projected/bc23c275-9d40-4d1c-aa91-405814c44687-kube-api-access-qg4lt\") pod \"e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c\" (UID: \"bc23c275-9d40-4d1c-aa91-405814c44687\") " pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c"
Jul 28 04:54:38.653132 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:38.652902    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc23c275-9d40-4d1c-aa91-405814c44687-workload-socket\") pod \"e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c\" (UID: \"bc23c275-9d40-4d1c-aa91-405814c44687\") " pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c"
Jul 28 04:54:38.653132 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:38.652926    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc23c275-9d40-4d1c-aa91-405814c44687-workload-certs\") pod \"e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c\" (UID: \"bc23c275-9d40-4d1c-aa91-405814c44687\") " pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c"
Jul 28 04:54:38.653950 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:38.653409    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc23c275-9d40-4d1c-aa91-405814c44687-workload-certs\") pod \"e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c\" (UID: \"bc23c275-9d40-4d1c-aa91-405814c44687\") " pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c"
Jul 28 04:54:38.653950 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:38.653592    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc23c275-9d40-4d1c-aa91-405814c44687-credential-socket\") pod \"e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c\" (UID: \"bc23c275-9d40-4d1c-aa91-405814c44687\") " pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c"
Jul 28 04:54:38.653950 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:38.653665    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc23c275-9d40-4d1c-aa91-405814c44687-workload-socket\") pod \"e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c\" (UID: \"bc23c275-9d40-4d1c-aa91-405814c44687\") " pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c"
Jul 28 04:54:38.653950 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:38.653814    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc23c275-9d40-4d1c-aa91-405814c44687-istio-data\") pod \"e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c\" (UID: \"bc23c275-9d40-4d1c-aa91-405814c44687\") " pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c"
Jul 28 04:54:38.653950 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:38.653915    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc23c275-9d40-4d1c-aa91-405814c44687-istiod-ca-cert\") pod \"e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c\" (UID: \"bc23c275-9d40-4d1c-aa91-405814c44687\") " pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c"
Jul 28 04:54:38.655865 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:38.655263    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc23c275-9d40-4d1c-aa91-405814c44687-istio-envoy\") pod \"e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c\" (UID: \"bc23c275-9d40-4d1c-aa91-405814c44687\") " pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c"
Jul 28 04:54:38.655865 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:38.655624    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc23c275-9d40-4d1c-aa91-405814c44687-istio-podinfo\") pod \"e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c\" (UID: \"bc23c275-9d40-4d1c-aa91-405814c44687\") " pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c"
Jul 28 04:54:38.661600 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:38.661571    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc23c275-9d40-4d1c-aa91-405814c44687-istio-token\") pod \"e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c\" (UID: \"bc23c275-9d40-4d1c-aa91-405814c44687\") " pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c"
Jul 28 04:54:38.661807 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:38.661792    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg4lt\" (UniqueName: \"kubernetes.io/projected/bc23c275-9d40-4d1c-aa91-405814c44687-kube-api-access-qg4lt\") pod \"e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c\" (UID: \"bc23c275-9d40-4d1c-aa91-405814c44687\") " pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c"
Jul 28 04:54:38.764514 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:38.764422    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c"
Jul 28 04:54:38.960914 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:38.960878    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c"]
Jul 28 04:54:38.964035 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:54:38.964009    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc23c275_9d40_4d1c_aa91_405814c44687.slice/crio-ec3d05812ded14a4019685d971c6561d51720972258c74f08ee6ff2f7da249e3 WatchSource:0}: Error finding container ec3d05812ded14a4019685d971c6561d51720972258c74f08ee6ff2f7da249e3: Status 404 returned error can't find the container with id ec3d05812ded14a4019685d971c6561d51720972258c74f08ee6ff2f7da249e3
Jul 28 04:54:38.966703 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:38.966668    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:54:38.966793 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:38.966742    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:54:38.966793 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:38.966776    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:54:39.136862 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:39.135943    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c" event={"ID":"bc23c275-9d40-4d1c-aa91-405814c44687","Type":"ContainerStarted","Data":"eb8a9d1113cc393fc4586575a332663398d0c4dd5ef20a0c40571699867259b9"}
Jul 28 04:54:39.136862 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:39.136011    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c" event={"ID":"bc23c275-9d40-4d1c-aa91-405814c44687","Type":"ContainerStarted","Data":"ec3d05812ded14a4019685d971c6561d51720972258c74f08ee6ff2f7da249e3"}
Jul 28 04:54:39.157055 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:39.156998    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c" podStartSLOduration=1.156979157 podStartE2EDuration="1.156979157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:54:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:54:39.155539136 +0000 UTC m=+2579.545715158" watchObservedRunningTime="2026-07-28 04:54:39.156979157 +0000 UTC m=+2579.547155183"
Jul 28 04:54:39.764659 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:39.764616    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c"
Jul 28 04:54:40.770932 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:40.770901    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c"
Jul 28 04:54:41.116502 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:41.116415    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f397e0-755589554-c8twz"
Jul 28 04:54:41.148525 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:41.148495    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c"
Jul 28 04:54:41.149712 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:41.149691    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c"
Jul 28 04:54:44.165070 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:44.165042    2566 generic.go:356] "Generic (PLEG): container finished" podID="e4eba9e9-5195-4bd9-a764-d756c44c1c15" containerID="a63ba5204c0d21a9b3724beb728d446809819901e459aecc71861a4e2de20dbc" exitCode=0
Jul 28 04:54:44.165459 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:44.165111    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl" event={"ID":"e4eba9e9-5195-4bd9-a764-d756c44c1c15","Type":"ContainerDied","Data":"a63ba5204c0d21a9b3724beb728d446809819901e459aecc71861a4e2de20dbc"}
Jul 28 04:54:44.165529 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:44.165511    2566 scope.go:117] "RemoveContainer" containerID="a63ba5204c0d21a9b3724beb728d446809819901e459aecc71861a4e2de20dbc"
Jul 28 04:54:44.166630 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:44.166602    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:54:44.166694 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:44.166683    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:54:44.166734 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:44.166712    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:54:45.171786 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:45.171747    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl" event={"ID":"e4eba9e9-5195-4bd9-a764-d756c44c1c15","Type":"ContainerStarted","Data":"6f041ce73607e6aa3ab6e2c959ca208d15479dccae8efdc046a36e0d6547591e"}
Jul 28 04:54:45.803559 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:45.803523    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl"
Jul 28 04:54:45.804693 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:45.804664    2566 patch_prober.go:28] interesting pod/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Jul 28 04:54:45.804819 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:45.804717    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl" podUID="e4eba9e9-5195-4bd9-a764-d756c44c1c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Jul 28 04:54:46.108645 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:46.108540    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f397e0-b6bbfc748-f2dv6" podUID="6851d96f-883c-40b0-992d-0db4e2a10fa5" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.79:9004: connect: connection refused"
Jul 28 04:54:46.109104 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:46.109084    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-f397e0-b897c75dd-z98ng"
Jul 28 04:54:46.803057 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:46.803021    2566 patch_prober.go:28] interesting pod/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Jul 28 04:54:46.803464 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:46.803086    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl" podUID="e4eba9e9-5195-4bd9-a764-d756c44c1c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Jul 28 04:54:47.803749 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:47.803712    2566 patch_prober.go:28] interesting pod/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Jul 28 04:54:47.804145 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:47.803765    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl" podUID="e4eba9e9-5195-4bd9-a764-d756c44c1c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Jul 28 04:54:48.088707 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:48.088623    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl"
Jul 28 04:54:48.803410 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:48.803374    2566 patch_prober.go:28] interesting pod/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body=
Jul 28 04:54:48.803590 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:48.803428    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl" podUID="e4eba9e9-5195-4bd9-a764-d756c44c1c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused"
Jul 28 04:54:49.806640 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:49.806608    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl"
Jul 28 04:54:49.807609 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:49.807589    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl"
Jul 28 04:54:54.203044 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:54.203005    2566 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f397e0-b6bbfc748-f2dv6" podUID="6851d96f-883c-40b0-992d-0db4e2a10fa5" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.79:9004: connect: connection refused"
Jul 28 04:54:56.107917 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:56.107886    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f397e0-b6bbfc748-f2dv6"
Jul 28 04:54:56.769498 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:56.768121    2566 patch_prober.go:28] interesting pod/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": read tcp 10.132.0.2:39762->10.132.0.81:15021: read: connection reset by peer" start-of-body=
Jul 28 04:54:56.769498 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:56.768199    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c" podUID="bc23c275-9d40-4d1c-aa91-405814c44687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": read tcp 10.132.0.2:39762->10.132.0.81:15021: read: connection reset by peer"
Jul 28 04:54:57.250938 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:57.250830    2566 generic.go:356] "Generic (PLEG): container finished" podID="bc23c275-9d40-4d1c-aa91-405814c44687" containerID="eb8a9d1113cc393fc4586575a332663398d0c4dd5ef20a0c40571699867259b9" exitCode=0
Jul 28 04:54:57.250938 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:57.250916    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c" event={"ID":"bc23c275-9d40-4d1c-aa91-405814c44687","Type":"ContainerDied","Data":"eb8a9d1113cc393fc4586575a332663398d0c4dd5ef20a0c40571699867259b9"}
Jul 28 04:54:57.251380 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:57.251366    2566 scope.go:117] "RemoveContainer" containerID="eb8a9d1113cc393fc4586575a332663398d0c4dd5ef20a0c40571699867259b9"
Jul 28 04:54:57.252695 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:57.252662    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:54:57.252780 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:57.252756    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:54:57.252825 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:57.252798    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:54:58.257636 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:58.257602    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c" event={"ID":"bc23c275-9d40-4d1c-aa91-405814c44687","Type":"ContainerStarted","Data":"00c578a6d142d4190bfe1d1419308f0b9f2004ea3fb9c85889e1d297fbf4f306"}
Jul 28 04:54:58.765108 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:58.765066    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c"
Jul 28 04:54:58.765362 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:58.765340    2566 patch_prober.go:28] interesting pod/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Jul 28 04:54:58.765407 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:58.765388    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c" podUID="bc23c275-9d40-4d1c-aa91-405814c44687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Jul 28 04:54:59.765389 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:59.765355    2566 patch_prober.go:28] interesting pod/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Jul 28 04:54:59.765784 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:54:59.765420    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c" podUID="bc23c275-9d40-4d1c-aa91-405814c44687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Jul 28 04:55:00.765028 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:00.764987    2566 patch_prober.go:28] interesting pod/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Jul 28 04:55:00.765224 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:00.765057    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c" podUID="bc23c275-9d40-4d1c-aa91-405814c44687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Jul 28 04:55:01.765660 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:01.765620    2566 patch_prober.go:28] interesting pod/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body=
Jul 28 04:55:01.766185 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:01.765702    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c" podUID="bc23c275-9d40-4d1c-aa91-405814c44687" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused"
Jul 28 04:55:02.768537 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:02.768502    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c"
Jul 28 04:55:02.768967 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:02.768869    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c"
Jul 28 04:55:02.769450 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:02.769434    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c"
Jul 28 04:55:09.037479 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:09.037443    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f397e0-755589554-c8twz"]
Jul 28 04:55:09.037989 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:09.037722    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f397e0-755589554-c8twz" podUID="9e615277-b304-43cc-8057-40bee75edcc6" containerName="maas-api" containerID="cri-o://98f3768968d6c4e06db0a12266667cd99f2cca8cd8bfae4abc72a4b868947fa0" gracePeriod=30
Jul 28 04:55:09.084286 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:09.084259    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f397e0-b897c75dd-z98ng"]
Jul 28 04:55:09.084574 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:09.084434    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-f397e0-b897c75dd-z98ng" podUID="a54ddc02-8c8e-4419-9918-c9da251cff0f" containerName="payload-processing" containerID="cri-o://865bfe29928e6d58788c9b1aee384ed6bdeb1521deedc1ca218a16bd9415c45d" gracePeriod=30
Jul 28 04:55:09.091864 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:09.091827    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f397e0-b6bbfc748-f2dv6"]
Jul 28 04:55:09.092055 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:09.092035    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f397e0-b6bbfc748-f2dv6" podUID="6851d96f-883c-40b0-992d-0db4e2a10fa5" containerName="payload-pre-processing" containerID="cri-o://1a1d5dc1c184f3a93bd7dfaab67a949a89da11af28ef92f4bf453a99b242a8c3" gracePeriod=30
Jul 28 04:55:09.320135 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:09.320097    2566 generic.go:356] "Generic (PLEG): container finished" podID="6851d96f-883c-40b0-992d-0db4e2a10fa5" containerID="1a1d5dc1c184f3a93bd7dfaab67a949a89da11af28ef92f4bf453a99b242a8c3" exitCode=0
Jul 28 04:55:09.320291 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:09.320139    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f397e0-b6bbfc748-f2dv6" event={"ID":"6851d96f-883c-40b0-992d-0db4e2a10fa5","Type":"ContainerDied","Data":"1a1d5dc1c184f3a93bd7dfaab67a949a89da11af28ef92f4bf453a99b242a8c3"}
Jul 28 04:55:09.323545 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:09.322324    2566 generic.go:356] "Generic (PLEG): container finished" podID="a54ddc02-8c8e-4419-9918-c9da251cff0f" containerID="865bfe29928e6d58788c9b1aee384ed6bdeb1521deedc1ca218a16bd9415c45d" exitCode=0
Jul 28 04:55:09.323545 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:09.322375    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f397e0-b897c75dd-z98ng" event={"ID":"a54ddc02-8c8e-4419-9918-c9da251cff0f","Type":"ContainerDied","Data":"865bfe29928e6d58788c9b1aee384ed6bdeb1521deedc1ca218a16bd9415c45d"}
Jul 28 04:55:09.324307 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:09.324253    2566 generic.go:356] "Generic (PLEG): container finished" podID="9e615277-b304-43cc-8057-40bee75edcc6" containerID="98f3768968d6c4e06db0a12266667cd99f2cca8cd8bfae4abc72a4b868947fa0" exitCode=0
Jul 28 04:55:09.324418 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:09.324304    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f397e0-755589554-c8twz" event={"ID":"9e615277-b304-43cc-8057-40bee75edcc6","Type":"ContainerDied","Data":"98f3768968d6c4e06db0a12266667cd99f2cca8cd8bfae4abc72a4b868947fa0"}
Jul 28 04:55:09.361377 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:09.361321    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f397e0-755589554-c8twz"
Jul 28 04:55:09.373527 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:09.373501    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e615277-b304-43cc-8057-40bee75edcc6-openshift-service-ca\") pod \"9e615277-b304-43cc-8057-40bee75edcc6\" (UID: \"9e615277-b304-43cc-8057-40bee75edcc6\") "
Jul 28 04:55:09.373649 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:09.373538    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txwkg\" (UniqueName: \"kubernetes.io/projected/9e615277-b304-43cc-8057-40bee75edcc6-kube-api-access-txwkg\") pod \"9e615277-b304-43cc-8057-40bee75edcc6\" (UID: \"9e615277-b304-43cc-8057-40bee75edcc6\") "
Jul 28 04:55:09.373649 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:09.373592    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e615277-b304-43cc-8057-40bee75edcc6-maas-api-tls\") pod \"9e615277-b304-43cc-8057-40bee75edcc6\" (UID: \"9e615277-b304-43cc-8057-40bee75edcc6\") "
Jul 28 04:55:09.373919 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:09.373894    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e615277-b304-43cc-8057-40bee75edcc6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9e615277-b304-43cc-8057-40bee75edcc6" (UID: "9e615277-b304-43cc-8057-40bee75edcc6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 04:55:09.375602 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:09.375577    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e615277-b304-43cc-8057-40bee75edcc6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9e615277-b304-43cc-8057-40bee75edcc6" (UID: "9e615277-b304-43cc-8057-40bee75edcc6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 04:55:09.375693 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:09.375669    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e615277-b304-43cc-8057-40bee75edcc6-kube-api-access-txwkg" (OuterVolumeSpecName: "kube-api-access-txwkg") pod "9e615277-b304-43cc-8057-40bee75edcc6" (UID: "9e615277-b304-43cc-8057-40bee75edcc6"). InnerVolumeSpecName "kube-api-access-txwkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:55:09.389649 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:09.389622    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f397e0-b897c75dd-z98ng"
Jul 28 04:55:09.413613 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:09.413593    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f397e0-b6bbfc748-f2dv6"
Jul 28 04:55:09.475175 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:09.475143    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r42px\" (UniqueName: \"kubernetes.io/projected/a54ddc02-8c8e-4419-9918-c9da251cff0f-kube-api-access-r42px\") pod \"a54ddc02-8c8e-4419-9918-c9da251cff0f\" (UID: \"a54ddc02-8c8e-4419-9918-c9da251cff0f\") "
Jul 28 04:55:09.475324 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:09.475208    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpmj5\" (UniqueName: \"kubernetes.io/projected/6851d96f-883c-40b0-992d-0db4e2a10fa5-kube-api-access-jpmj5\") pod \"6851d96f-883c-40b0-992d-0db4e2a10fa5\" (UID: \"6851d96f-883c-40b0-992d-0db4e2a10fa5\") "
Jul 28 04:55:09.475324 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:09.475236    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6851d96f-883c-40b0-992d-0db4e2a10fa5-plugins-config-volume\") pod \"6851d96f-883c-40b0-992d-0db4e2a10fa5\" (UID: \"6851d96f-883c-40b0-992d-0db4e2a10fa5\") "
Jul 28 04:55:09.475324 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:09.475278    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a54ddc02-8c8e-4419-9918-c9da251cff0f-plugins-config-volume\") pod \"a54ddc02-8c8e-4419-9918-c9da251cff0f\" (UID: \"a54ddc02-8c8e-4419-9918-c9da251cff0f\") "
Jul 28 04:55:09.475600 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:09.475584    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e615277-b304-43cc-8057-40bee75edcc6-openshift-service-ca\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:55:09.475678 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:09.475605    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txwkg\" (UniqueName: \"kubernetes.io/projected/9e615277-b304-43cc-8057-40bee75edcc6-kube-api-access-txwkg\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:55:09.475678 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:09.475619    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e615277-b304-43cc-8057-40bee75edcc6-maas-api-tls\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:55:09.475756 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:09.475671    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a54ddc02-8c8e-4419-9918-c9da251cff0f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a54ddc02-8c8e-4419-9918-c9da251cff0f" (UID: "a54ddc02-8c8e-4419-9918-c9da251cff0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 04:55:09.475756 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:09.475691    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6851d96f-883c-40b0-992d-0db4e2a10fa5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6851d96f-883c-40b0-992d-0db4e2a10fa5" (UID: "6851d96f-883c-40b0-992d-0db4e2a10fa5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 04:55:09.477726 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:09.477704    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a54ddc02-8c8e-4419-9918-c9da251cff0f-kube-api-access-r42px" (OuterVolumeSpecName: "kube-api-access-r42px") pod "a54ddc02-8c8e-4419-9918-c9da251cff0f" (UID: "a54ddc02-8c8e-4419-9918-c9da251cff0f"). InnerVolumeSpecName "kube-api-access-r42px". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:55:09.477794 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:09.477731    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6851d96f-883c-40b0-992d-0db4e2a10fa5-kube-api-access-jpmj5" (OuterVolumeSpecName: "kube-api-access-jpmj5") pod "6851d96f-883c-40b0-992d-0db4e2a10fa5" (UID: "6851d96f-883c-40b0-992d-0db4e2a10fa5"). InnerVolumeSpecName "kube-api-access-jpmj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:55:09.576212 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:09.576140    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r42px\" (UniqueName: \"kubernetes.io/projected/a54ddc02-8c8e-4419-9918-c9da251cff0f-kube-api-access-r42px\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:55:09.576212 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:09.576169    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpmj5\" (UniqueName: \"kubernetes.io/projected/6851d96f-883c-40b0-992d-0db4e2a10fa5-kube-api-access-jpmj5\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:55:09.576212 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:09.576179    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6851d96f-883c-40b0-992d-0db4e2a10fa5-plugins-config-volume\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:55:09.576212 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:09.576187    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a54ddc02-8c8e-4419-9918-c9da251cff0f-plugins-config-volume\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:55:10.330905 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:10.330819    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f397e0-b6bbfc748-f2dv6"
Jul 28 04:55:10.330905 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:10.330830    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f397e0-b6bbfc748-f2dv6" event={"ID":"6851d96f-883c-40b0-992d-0db4e2a10fa5","Type":"ContainerDied","Data":"efdc2895facd895c7e2c36a12758aa68bc4422465dc1bd09c994f55b7dbfd8d8"}
Jul 28 04:55:10.331437 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:10.330908    2566 scope.go:117] "RemoveContainer" containerID="1a1d5dc1c184f3a93bd7dfaab67a949a89da11af28ef92f4bf453a99b242a8c3"
Jul 28 04:55:10.332475 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:10.332429    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f397e0-b897c75dd-z98ng" event={"ID":"a54ddc02-8c8e-4419-9918-c9da251cff0f","Type":"ContainerDied","Data":"1a90216395e693b5055ca27ec5c7a4ba6f4cc1b58b312ce2d8bfb4c0421c440c"}
Jul 28 04:55:10.332585 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:10.332439    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f397e0-b897c75dd-z98ng"
Jul 28 04:55:10.333991 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:10.333963    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f397e0-755589554-c8twz"
Jul 28 04:55:10.333991 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:10.333987    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f397e0-755589554-c8twz" event={"ID":"9e615277-b304-43cc-8057-40bee75edcc6","Type":"ContainerDied","Data":"50921493c576144ba17ce360c09bd66e92074994d917ffef75eb653511348d45"}
Jul 28 04:55:10.346641 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:10.346623    2566 scope.go:117] "RemoveContainer" containerID="865bfe29928e6d58788c9b1aee384ed6bdeb1521deedc1ca218a16bd9415c45d"
Jul 28 04:55:10.354233 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:10.354218    2566 scope.go:117] "RemoveContainer" containerID="98f3768968d6c4e06db0a12266667cd99f2cca8cd8bfae4abc72a4b868947fa0"
Jul 28 04:55:10.358642 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:10.358619    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f397e0-755589554-c8twz"]
Jul 28 04:55:10.361559 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:10.361537    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f397e0-755589554-c8twz"]
Jul 28 04:55:10.370596 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:10.370575    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f397e0-b897c75dd-z98ng"]
Jul 28 04:55:10.372864 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:10.372830    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f397e0-b897c75dd-z98ng"]
Jul 28 04:55:10.382316 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:10.382297    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f397e0-b6bbfc748-f2dv6"]
Jul 28 04:55:10.384442 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:10.384422    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f397e0-b6bbfc748-f2dv6"]
Jul 28 04:55:12.088002 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:12.087968    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6851d96f-883c-40b0-992d-0db4e2a10fa5" path="/var/lib/kubelet/pods/6851d96f-883c-40b0-992d-0db4e2a10fa5/volumes"
Jul 28 04:55:12.088390 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:12.088377    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e615277-b304-43cc-8057-40bee75edcc6" path="/var/lib/kubelet/pods/9e615277-b304-43cc-8057-40bee75edcc6/volumes"
Jul 28 04:55:12.088735 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:12.088720    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a54ddc02-8c8e-4419-9918-c9da251cff0f" path="/var/lib/kubelet/pods/a54ddc02-8c8e-4419-9918-c9da251cff0f/volumes"
Jul 28 04:55:20.426531 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:20.426494    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl"]
Jul 28 04:55:20.427378 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:20.427333    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl" podUID="e4eba9e9-5195-4bd9-a764-d756c44c1c15" containerName="istio-proxy" containerID="cri-o://6f041ce73607e6aa3ab6e2c959ca208d15479dccae8efdc046a36e0d6547591e" gracePeriod=30
Jul 28 04:55:25.680208 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:25.680183    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl"
Jul 28 04:55:25.722389 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:25.722312    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4eba9e9-5195-4bd9-a764-d756c44c1c15-workload-socket\") pod \"e4eba9e9-5195-4bd9-a764-d756c44c1c15\" (UID: \"e4eba9e9-5195-4bd9-a764-d756c44c1c15\") "
Jul 28 04:55:25.722536 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:25.722390    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4eba9e9-5195-4bd9-a764-d756c44c1c15-istio-envoy\") pod \"e4eba9e9-5195-4bd9-a764-d756c44c1c15\" (UID: \"e4eba9e9-5195-4bd9-a764-d756c44c1c15\") "
Jul 28 04:55:25.722536 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:25.722421    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4eba9e9-5195-4bd9-a764-d756c44c1c15-istio-token\") pod \"e4eba9e9-5195-4bd9-a764-d756c44c1c15\" (UID: \"e4eba9e9-5195-4bd9-a764-d756c44c1c15\") "
Jul 28 04:55:25.722536 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:25.722443    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4eba9e9-5195-4bd9-a764-d756c44c1c15-istio-data\") pod \"e4eba9e9-5195-4bd9-a764-d756c44c1c15\" (UID: \"e4eba9e9-5195-4bd9-a764-d756c44c1c15\") "
Jul 28 04:55:25.722536 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:25.722522    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nz48\" (UniqueName: \"kubernetes.io/projected/e4eba9e9-5195-4bd9-a764-d756c44c1c15-kube-api-access-6nz48\") pod \"e4eba9e9-5195-4bd9-a764-d756c44c1c15\" (UID: \"e4eba9e9-5195-4bd9-a764-d756c44c1c15\") "
Jul 28 04:55:25.722757 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:25.722572    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4eba9e9-5195-4bd9-a764-d756c44c1c15-workload-certs\") pod \"e4eba9e9-5195-4bd9-a764-d756c44c1c15\" (UID: \"e4eba9e9-5195-4bd9-a764-d756c44c1c15\") "
Jul 28 04:55:25.722757 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:25.722591    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4eba9e9-5195-4bd9-a764-d756c44c1c15-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e4eba9e9-5195-4bd9-a764-d756c44c1c15" (UID: "e4eba9e9-5195-4bd9-a764-d756c44c1c15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:55:25.722757 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:25.722608    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4eba9e9-5195-4bd9-a764-d756c44c1c15-istio-podinfo\") pod \"e4eba9e9-5195-4bd9-a764-d756c44c1c15\" (UID: \"e4eba9e9-5195-4bd9-a764-d756c44c1c15\") "
Jul 28 04:55:25.722757 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:25.722642    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4eba9e9-5195-4bd9-a764-d756c44c1c15-istiod-ca-cert\") pod \"e4eba9e9-5195-4bd9-a764-d756c44c1c15\" (UID: \"e4eba9e9-5195-4bd9-a764-d756c44c1c15\") "
Jul 28 04:55:25.722757 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:25.722711    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4eba9e9-5195-4bd9-a764-d756c44c1c15-credential-socket\") pod \"e4eba9e9-5195-4bd9-a764-d756c44c1c15\" (UID: \"e4eba9e9-5195-4bd9-a764-d756c44c1c15\") "
Jul 28 04:55:25.723739 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:25.723084    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4eba9e9-5195-4bd9-a764-d756c44c1c15-workload-socket\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:55:25.723739 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:25.723316    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4eba9e9-5195-4bd9-a764-d756c44c1c15-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e4eba9e9-5195-4bd9-a764-d756c44c1c15" (UID: "e4eba9e9-5195-4bd9-a764-d756c44c1c15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:55:25.723739 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:25.723399    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4eba9e9-5195-4bd9-a764-d756c44c1c15-istio-data" (OuterVolumeSpecName: "istio-data") pod "e4eba9e9-5195-4bd9-a764-d756c44c1c15" (UID: "e4eba9e9-5195-4bd9-a764-d756c44c1c15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:55:25.723739 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:25.723618    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4eba9e9-5195-4bd9-a764-d756c44c1c15-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e4eba9e9-5195-4bd9-a764-d756c44c1c15" (UID: "e4eba9e9-5195-4bd9-a764-d756c44c1c15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:55:25.724057 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:25.723789    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4eba9e9-5195-4bd9-a764-d756c44c1c15-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e4eba9e9-5195-4bd9-a764-d756c44c1c15" (UID: "e4eba9e9-5195-4bd9-a764-d756c44c1c15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 04:55:25.724998 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:25.724977    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4eba9e9-5195-4bd9-a764-d756c44c1c15-kube-api-access-6nz48" (OuterVolumeSpecName: "kube-api-access-6nz48") pod "e4eba9e9-5195-4bd9-a764-d756c44c1c15" (UID: "e4eba9e9-5195-4bd9-a764-d756c44c1c15"). InnerVolumeSpecName "kube-api-access-6nz48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:55:25.725130 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:25.725105    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4eba9e9-5195-4bd9-a764-d756c44c1c15-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e4eba9e9-5195-4bd9-a764-d756c44c1c15" (UID: "e4eba9e9-5195-4bd9-a764-d756c44c1c15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 04:55:25.725194 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:25.725154    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4eba9e9-5195-4bd9-a764-d756c44c1c15-istio-token" (OuterVolumeSpecName: "istio-token") pod "e4eba9e9-5195-4bd9-a764-d756c44c1c15" (UID: "e4eba9e9-5195-4bd9-a764-d756c44c1c15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:55:25.725596 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:25.725580    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4eba9e9-5195-4bd9-a764-d756c44c1c15-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e4eba9e9-5195-4bd9-a764-d756c44c1c15" (UID: "e4eba9e9-5195-4bd9-a764-d756c44c1c15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:55:25.824514 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:25.824481    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nz48\" (UniqueName: \"kubernetes.io/projected/e4eba9e9-5195-4bd9-a764-d756c44c1c15-kube-api-access-6nz48\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:55:25.824514 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:25.824512    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4eba9e9-5195-4bd9-a764-d756c44c1c15-workload-certs\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:55:25.824746 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:25.824526    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4eba9e9-5195-4bd9-a764-d756c44c1c15-istio-podinfo\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:55:25.824746 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:25.824538    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4eba9e9-5195-4bd9-a764-d756c44c1c15-istiod-ca-cert\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:55:25.824746 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:25.824550    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4eba9e9-5195-4bd9-a764-d756c44c1c15-credential-socket\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:55:25.824746 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:25.824561    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4eba9e9-5195-4bd9-a764-d756c44c1c15-istio-envoy\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:55:25.824746 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:25.824571    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4eba9e9-5195-4bd9-a764-d756c44c1c15-istio-token\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:55:25.824746 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:25.824581    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4eba9e9-5195-4bd9-a764-d756c44c1c15-istio-data\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:55:26.415305 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:26.415217    2566 generic.go:356] "Generic (PLEG): container finished" podID="e4eba9e9-5195-4bd9-a764-d756c44c1c15" containerID="6f041ce73607e6aa3ab6e2c959ca208d15479dccae8efdc046a36e0d6547591e" exitCode=0
Jul 28 04:55:26.415305 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:26.415264    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl"
Jul 28 04:55:26.415499 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:26.415309    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl" event={"ID":"e4eba9e9-5195-4bd9-a764-d756c44c1c15","Type":"ContainerDied","Data":"6f041ce73607e6aa3ab6e2c959ca208d15479dccae8efdc046a36e0d6547591e"}
Jul 28 04:55:26.415499 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:26.415362    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl" event={"ID":"e4eba9e9-5195-4bd9-a764-d756c44c1c15","Type":"ContainerDied","Data":"e5282d20b27ec527558edc57f90f7df804744412213397e80f1f633d74d758a1"}
Jul 28 04:55:26.415499 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:26.415384    2566 scope.go:117] "RemoveContainer" containerID="6f041ce73607e6aa3ab6e2c959ca208d15479dccae8efdc046a36e0d6547591e"
Jul 28 04:55:26.424009 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:26.423995    2566 scope.go:117] "RemoveContainer" containerID="a63ba5204c0d21a9b3724beb728d446809819901e459aecc71861a4e2de20dbc"
Jul 28 04:55:26.432013 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:26.431992    2566 scope.go:117] "RemoveContainer" containerID="6f041ce73607e6aa3ab6e2c959ca208d15479dccae8efdc046a36e0d6547591e"
Jul 28 04:55:26.432279 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:55:26.432262    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f041ce73607e6aa3ab6e2c959ca208d15479dccae8efdc046a36e0d6547591e\": container with ID starting with 6f041ce73607e6aa3ab6e2c959ca208d15479dccae8efdc046a36e0d6547591e not found: ID does not exist" containerID="6f041ce73607e6aa3ab6e2c959ca208d15479dccae8efdc046a36e0d6547591e"
Jul 28 04:55:26.432350 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:26.432287    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f041ce73607e6aa3ab6e2c959ca208d15479dccae8efdc046a36e0d6547591e"} err="failed to get container status \"6f041ce73607e6aa3ab6e2c959ca208d15479dccae8efdc046a36e0d6547591e\": rpc error: code = NotFound desc = could not find container \"6f041ce73607e6aa3ab6e2c959ca208d15479dccae8efdc046a36e0d6547591e\": container with ID starting with 6f041ce73607e6aa3ab6e2c959ca208d15479dccae8efdc046a36e0d6547591e not found: ID does not exist"
Jul 28 04:55:26.432350 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:26.432303    2566 scope.go:117] "RemoveContainer" containerID="a63ba5204c0d21a9b3724beb728d446809819901e459aecc71861a4e2de20dbc"
Jul 28 04:55:26.432548 ip-10-0-129-187 kubenswrapper[2566]: E0728 04:55:26.432529    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a63ba5204c0d21a9b3724beb728d446809819901e459aecc71861a4e2de20dbc\": container with ID starting with a63ba5204c0d21a9b3724beb728d446809819901e459aecc71861a4e2de20dbc not found: ID does not exist" containerID="a63ba5204c0d21a9b3724beb728d446809819901e459aecc71861a4e2de20dbc"
Jul 28 04:55:26.432612 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:26.432557    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a63ba5204c0d21a9b3724beb728d446809819901e459aecc71861a4e2de20dbc"} err="failed to get container status \"a63ba5204c0d21a9b3724beb728d446809819901e459aecc71861a4e2de20dbc\": rpc error: code = NotFound desc = could not find container \"a63ba5204c0d21a9b3724beb728d446809819901e459aecc71861a4e2de20dbc\": container with ID starting with a63ba5204c0d21a9b3724beb728d446809819901e459aecc71861a4e2de20dbc not found: ID does not exist"
Jul 28 04:55:26.433692 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:26.433669    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl"]
Jul 28 04:55:26.436877 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:26.436857    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f397e0-openshift-default-76785b9f88-9g6zl"]
Jul 28 04:55:28.096204 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:28.096169    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4eba9e9-5195-4bd9-a764-d756c44c1c15" path="/var/lib/kubelet/pods/e4eba9e9-5195-4bd9-a764-d756c44c1c15/volumes"
Jul 28 04:55:47.358633 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:47.358601    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c"]
Jul 28 04:55:47.358998 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:47.358853    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c" podUID="bc23c275-9d40-4d1c-aa91-405814c44687" containerName="istio-proxy" containerID="cri-o://00c578a6d142d4190bfe1d1419308f0b9f2004ea3fb9c85889e1d297fbf4f306" gracePeriod=30
Jul 28 04:55:47.771065 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:47.771034    2566 patch_prober.go:28] interesting pod/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 28 04:55:47.771233 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:47.771102    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c" podUID="bc23c275-9d40-4d1c-aa91-405814c44687" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 04:55:48.242439 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.242352    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz"]
Jul 28 04:55:48.242929 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.242913    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4eba9e9-5195-4bd9-a764-d756c44c1c15" containerName="istio-proxy"
Jul 28 04:55:48.242981 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.242932    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4eba9e9-5195-4bd9-a764-d756c44c1c15" containerName="istio-proxy"
Jul 28 04:55:48.242981 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.242962    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4eba9e9-5195-4bd9-a764-d756c44c1c15" containerName="istio-proxy"
Jul 28 04:55:48.242981 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.242971    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4eba9e9-5195-4bd9-a764-d756c44c1c15" containerName="istio-proxy"
Jul 28 04:55:48.243072 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.242988    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e615277-b304-43cc-8057-40bee75edcc6" containerName="maas-api"
Jul 28 04:55:48.243072 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.242997    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e615277-b304-43cc-8057-40bee75edcc6" containerName="maas-api"
Jul 28 04:55:48.243072 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.243009    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6851d96f-883c-40b0-992d-0db4e2a10fa5" containerName="payload-pre-processing"
Jul 28 04:55:48.243072 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.243017    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6851d96f-883c-40b0-992d-0db4e2a10fa5" containerName="payload-pre-processing"
Jul 28 04:55:48.243072 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.243031    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a54ddc02-8c8e-4419-9918-c9da251cff0f" containerName="payload-processing"
Jul 28 04:55:48.243072 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.243040    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a54ddc02-8c8e-4419-9918-c9da251cff0f" containerName="payload-processing"
Jul 28 04:55:48.243254 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.243121    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e615277-b304-43cc-8057-40bee75edcc6" containerName="maas-api"
Jul 28 04:55:48.243254 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.243134    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4eba9e9-5195-4bd9-a764-d756c44c1c15" containerName="istio-proxy"
Jul 28 04:55:48.243254 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.243143    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6851d96f-883c-40b0-992d-0db4e2a10fa5" containerName="payload-pre-processing"
Jul 28 04:55:48.243254 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.243156    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a54ddc02-8c8e-4419-9918-c9da251cff0f" containerName="payload-processing"
Jul 28 04:55:48.246293 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.246272    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz"
Jul 28 04:55:48.248938 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.248918    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-1badbb-openshift-default-dockercfg-p6cgd\""
Jul 28 04:55:48.255888 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.255861    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz"]
Jul 28 04:55:48.433715 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.433682    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abc8eda5-4cd4-4570-b474-c525a74eb3c6-istio-data\") pod \"e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz\" (UID: \"abc8eda5-4cd4-4570-b474-c525a74eb3c6\") " pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz"
Jul 28 04:55:48.434180 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.433735    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc8eda5-4cd4-4570-b474-c525a74eb3c6-workload-socket\") pod \"e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz\" (UID: \"abc8eda5-4cd4-4570-b474-c525a74eb3c6\") " pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz"
Jul 28 04:55:48.434180 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.433764    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h9hp\" (UniqueName: \"kubernetes.io/projected/abc8eda5-4cd4-4570-b474-c525a74eb3c6-kube-api-access-9h9hp\") pod \"e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz\" (UID: \"abc8eda5-4cd4-4570-b474-c525a74eb3c6\") " pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz"
Jul 28 04:55:48.434180 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.433780    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abc8eda5-4cd4-4570-b474-c525a74eb3c6-istio-envoy\") pod \"e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz\" (UID: \"abc8eda5-4cd4-4570-b474-c525a74eb3c6\") " pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz"
Jul 28 04:55:48.434180 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.433805    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abc8eda5-4cd4-4570-b474-c525a74eb3c6-istio-token\") pod \"e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz\" (UID: \"abc8eda5-4cd4-4570-b474-c525a74eb3c6\") " pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz"
Jul 28 04:55:48.434180 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.433821    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc8eda5-4cd4-4570-b474-c525a74eb3c6-credential-socket\") pod \"e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz\" (UID: \"abc8eda5-4cd4-4570-b474-c525a74eb3c6\") " pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz"
Jul 28 04:55:48.434180 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.433898    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abc8eda5-4cd4-4570-b474-c525a74eb3c6-workload-certs\") pod \"e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz\" (UID: \"abc8eda5-4cd4-4570-b474-c525a74eb3c6\") " pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz"
Jul 28 04:55:48.434180 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.433914    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abc8eda5-4cd4-4570-b474-c525a74eb3c6-istiod-ca-cert\") pod \"e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz\" (UID: \"abc8eda5-4cd4-4570-b474-c525a74eb3c6\") " pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz"
Jul 28 04:55:48.434180 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.433946    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abc8eda5-4cd4-4570-b474-c525a74eb3c6-istio-podinfo\") pod \"e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz\" (UID: \"abc8eda5-4cd4-4570-b474-c525a74eb3c6\") " pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz"
Jul 28 04:55:48.534476 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.534447    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abc8eda5-4cd4-4570-b474-c525a74eb3c6-workload-certs\") pod \"e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz\" (UID: \"abc8eda5-4cd4-4570-b474-c525a74eb3c6\") " pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz"
Jul 28 04:55:48.534476 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.534480    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abc8eda5-4cd4-4570-b474-c525a74eb3c6-istiod-ca-cert\") pod \"e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz\" (UID: \"abc8eda5-4cd4-4570-b474-c525a74eb3c6\") " pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz"
Jul 28 04:55:48.534735 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.534515    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abc8eda5-4cd4-4570-b474-c525a74eb3c6-istio-podinfo\") pod \"e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz\" (UID: \"abc8eda5-4cd4-4570-b474-c525a74eb3c6\") " pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz"
Jul 28 04:55:48.534735 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.534567    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abc8eda5-4cd4-4570-b474-c525a74eb3c6-istio-data\") pod \"e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz\" (UID: \"abc8eda5-4cd4-4570-b474-c525a74eb3c6\") " pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz"
Jul 28 04:55:48.534735 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.534612    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc8eda5-4cd4-4570-b474-c525a74eb3c6-workload-socket\") pod \"e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz\" (UID: \"abc8eda5-4cd4-4570-b474-c525a74eb3c6\") " pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz"
Jul 28 04:55:48.534735 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.534638    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h9hp\" (UniqueName: \"kubernetes.io/projected/abc8eda5-4cd4-4570-b474-c525a74eb3c6-kube-api-access-9h9hp\") pod \"e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz\" (UID: \"abc8eda5-4cd4-4570-b474-c525a74eb3c6\") " pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz"
Jul 28 04:55:48.534735 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.534664    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abc8eda5-4cd4-4570-b474-c525a74eb3c6-istio-envoy\") pod \"e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz\" (UID: \"abc8eda5-4cd4-4570-b474-c525a74eb3c6\") " pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz"
Jul 28 04:55:48.534735 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.534705    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abc8eda5-4cd4-4570-b474-c525a74eb3c6-istio-token\") pod \"e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz\" (UID: \"abc8eda5-4cd4-4570-b474-c525a74eb3c6\") " pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz"
Jul 28 04:55:48.534735 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.534726    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc8eda5-4cd4-4570-b474-c525a74eb3c6-credential-socket\") pod \"e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz\" (UID: \"abc8eda5-4cd4-4570-b474-c525a74eb3c6\") " pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz"
Jul 28 04:55:48.535126 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.534926    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abc8eda5-4cd4-4570-b474-c525a74eb3c6-workload-certs\") pod \"e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz\" (UID: \"abc8eda5-4cd4-4570-b474-c525a74eb3c6\") " pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz"
Jul 28 04:55:48.535126 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.535060    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abc8eda5-4cd4-4570-b474-c525a74eb3c6-istio-data\") pod \"e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz\" (UID: \"abc8eda5-4cd4-4570-b474-c525a74eb3c6\") " pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz"
Jul 28 04:55:48.535310 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.535287    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc8eda5-4cd4-4570-b474-c525a74eb3c6-credential-socket\") pod \"e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz\" (UID: \"abc8eda5-4cd4-4570-b474-c525a74eb3c6\") " pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz"
Jul 28 04:55:48.535354 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.535289    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abc8eda5-4cd4-4570-b474-c525a74eb3c6-istiod-ca-cert\") pod \"e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz\" (UID: \"abc8eda5-4cd4-4570-b474-c525a74eb3c6\") " pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz"
Jul 28 04:55:48.535354 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.535292    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc8eda5-4cd4-4570-b474-c525a74eb3c6-workload-socket\") pod \"e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz\" (UID: \"abc8eda5-4cd4-4570-b474-c525a74eb3c6\") " pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz"
Jul 28 04:55:48.537119 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.537100    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abc8eda5-4cd4-4570-b474-c525a74eb3c6-istio-podinfo\") pod \"e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz\" (UID: \"abc8eda5-4cd4-4570-b474-c525a74eb3c6\") " pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz"
Jul 28 04:55:48.537240 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.537147    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abc8eda5-4cd4-4570-b474-c525a74eb3c6-istio-envoy\") pod \"e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz\" (UID: \"abc8eda5-4cd4-4570-b474-c525a74eb3c6\") " pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz"
Jul 28 04:55:48.542281 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.542259    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h9hp\" (UniqueName: \"kubernetes.io/projected/abc8eda5-4cd4-4570-b474-c525a74eb3c6-kube-api-access-9h9hp\") pod \"e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz\" (UID: \"abc8eda5-4cd4-4570-b474-c525a74eb3c6\") " pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz"
Jul 28 04:55:48.542281 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.542275    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abc8eda5-4cd4-4570-b474-c525a74eb3c6-istio-token\") pod \"e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz\" (UID: \"abc8eda5-4cd4-4570-b474-c525a74eb3c6\") " pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz"
Jul 28 04:55:48.558290 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.558271    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz"
Jul 28 04:55:48.690704 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.690677    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz"]
Jul 28 04:55:48.692829 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:55:48.692804    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabc8eda5_4cd4_4570_b474_c525a74eb3c6.slice/crio-e3c22d9efe46ae3d1c25b0651b2982ec5ac09130f952342321454b811620904c WatchSource:0}: Error finding container e3c22d9efe46ae3d1c25b0651b2982ec5ac09130f952342321454b811620904c: Status 404 returned error can't find the container with id e3c22d9efe46ae3d1c25b0651b2982ec5ac09130f952342321454b811620904c
Jul 28 04:55:48.695615 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.695583    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:55:48.695697 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.695653    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:55:48.695735 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:48.695701    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:55:49.531377 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:49.531337    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz" event={"ID":"abc8eda5-4cd4-4570-b474-c525a74eb3c6","Type":"ContainerStarted","Data":"fb4f2119ef0b5ff6db30de8277e23a49fe72bdd60d16dc12deeb8688ecadf4dd"}
Jul 28 04:55:49.531761 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:49.531397    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz" event={"ID":"abc8eda5-4cd4-4570-b474-c525a74eb3c6","Type":"ContainerStarted","Data":"e3c22d9efe46ae3d1c25b0651b2982ec5ac09130f952342321454b811620904c"}
Jul 28 04:55:49.552447 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:49.552409    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz" podStartSLOduration=1.552394595 podStartE2EDuration="1.552394595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:55:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:55:49.551072906 +0000 UTC m=+2649.941248927" watchObservedRunningTime="2026-07-28 04:55:49.552394595 +0000 UTC m=+2649.942570617"
Jul 28 04:55:49.559350 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:49.559326    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz"
Jul 28 04:55:49.563940 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:49.563921    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz"
Jul 28 04:55:50.536136 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:50.536098    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz"
Jul 28 04:55:50.537152 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:50.537134    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz"
Jul 28 04:55:51.758944 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:51.758903    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-1badbb-5fd5858c-sjvg9"]
Jul 28 04:55:51.759564 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:51.759544    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4eba9e9-5195-4bd9-a764-d756c44c1c15" containerName="istio-proxy"
Jul 28 04:55:51.762268 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:51.762248    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1badbb-5fd5858c-sjvg9"
Jul 28 04:55:51.764746 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:51.764724    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gfn77\""
Jul 28 04:55:51.764882 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:51.764863    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-1badbb-serving-cert\""
Jul 28 04:55:51.765896 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:51.765878    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 28 04:55:51.765999 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:51.765979    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 28 04:55:51.771025 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:51.770986    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-1badbb-5fd5858c-sjvg9"]
Jul 28 04:55:51.868572 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:51.868535    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9bb777e-25c1-4454-b326-96ef0a1941d4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-1badbb-5fd5858c-sjvg9\" (UID: \"c9bb777e-25c1-4454-b326-96ef0a1941d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1badbb-5fd5858c-sjvg9"
Jul 28 04:55:51.868772 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:51.868577    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grt7q\" (UniqueName: \"kubernetes.io/projected/c9bb777e-25c1-4454-b326-96ef0a1941d4-kube-api-access-grt7q\") pod \"maas-api-e2e-shared-a-1badbb-5fd5858c-sjvg9\" (UID: \"c9bb777e-25c1-4454-b326-96ef0a1941d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1badbb-5fd5858c-sjvg9"
Jul 28 04:55:51.868772 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:51.868645    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9bb777e-25c1-4454-b326-96ef0a1941d4-maas-api-tls\") pod \"maas-api-e2e-shared-a-1badbb-5fd5858c-sjvg9\" (UID: \"c9bb777e-25c1-4454-b326-96ef0a1941d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1badbb-5fd5858c-sjvg9"
Jul 28 04:55:51.969694 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:51.969657    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9bb777e-25c1-4454-b326-96ef0a1941d4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-1badbb-5fd5858c-sjvg9\" (UID: \"c9bb777e-25c1-4454-b326-96ef0a1941d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1badbb-5fd5858c-sjvg9"
Jul 28 04:55:51.969694 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:51.969699    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grt7q\" (UniqueName: \"kubernetes.io/projected/c9bb777e-25c1-4454-b326-96ef0a1941d4-kube-api-access-grt7q\") pod \"maas-api-e2e-shared-a-1badbb-5fd5858c-sjvg9\" (UID: \"c9bb777e-25c1-4454-b326-96ef0a1941d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1badbb-5fd5858c-sjvg9"
Jul 28 04:55:51.970005 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:51.969819    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9bb777e-25c1-4454-b326-96ef0a1941d4-maas-api-tls\") pod \"maas-api-e2e-shared-a-1badbb-5fd5858c-sjvg9\" (UID: \"c9bb777e-25c1-4454-b326-96ef0a1941d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1badbb-5fd5858c-sjvg9"
Jul 28 04:55:51.970400 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:51.970373    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9bb777e-25c1-4454-b326-96ef0a1941d4-openshift-service-ca\") pod \"maas-api-e2e-shared-a-1badbb-5fd5858c-sjvg9\" (UID: \"c9bb777e-25c1-4454-b326-96ef0a1941d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1badbb-5fd5858c-sjvg9"
Jul 28 04:55:51.979857 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:51.978326    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grt7q\" (UniqueName: \"kubernetes.io/projected/c9bb777e-25c1-4454-b326-96ef0a1941d4-kube-api-access-grt7q\") pod \"maas-api-e2e-shared-a-1badbb-5fd5858c-sjvg9\" (UID: \"c9bb777e-25c1-4454-b326-96ef0a1941d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1badbb-5fd5858c-sjvg9"
Jul 28 04:55:51.980224 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:51.980201    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9bb777e-25c1-4454-b326-96ef0a1941d4-maas-api-tls\") pod \"maas-api-e2e-shared-a-1badbb-5fd5858c-sjvg9\" (UID: \"c9bb777e-25c1-4454-b326-96ef0a1941d4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1badbb-5fd5858c-sjvg9"
Jul 28 04:55:52.075203 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.075160    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1badbb-5fd5858c-sjvg9"
Jul 28 04:55:52.220917 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.220895    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-1badbb-5fd5858c-sjvg9"]
Jul 28 04:55:52.223105 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:55:52.223061    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9bb777e_25c1_4454_b326_96ef0a1941d4.slice/crio-05478c885b5065902d692ad781ce03f0bca84619546c114e8a5008a7484f5bb1 WatchSource:0}: Error finding container 05478c885b5065902d692ad781ce03f0bca84619546c114e8a5008a7484f5bb1: Status 404 returned error can't find the container with id 05478c885b5065902d692ad781ce03f0bca84619546c114e8a5008a7484f5bb1
Jul 28 04:55:52.428997 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.428962    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-1badbb-595f44df4d-mm2k8"]
Jul 28 04:55:52.432496 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.432477    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1badbb-595f44df4d-mm2k8"
Jul 28 04:55:52.435111 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.435089    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-1badbb-dockercfg-lld4p\""
Jul 28 04:55:52.435271 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.435253    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-1badbb\""
Jul 28 04:55:52.442893 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.442870    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-1badbb-595f44df4d-mm2k8"]
Jul 28 04:55:52.447939 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.447919    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-1badbb-7ff76ff76b-mklzc"]
Jul 28 04:55:52.451265 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.451250    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-1badbb-7ff76ff76b-mklzc"
Jul 28 04:55:52.463688 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.463666    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-1badbb-7ff76ff76b-mklzc"]
Jul 28 04:55:52.474852 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.474818    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd82k\" (UniqueName: \"kubernetes.io/projected/907caf08-9b81-40eb-937f-777750801bfe-kube-api-access-vd82k\") pod \"payload-processing-e2e-shared-a-1badbb-7ff76ff76b-mklzc\" (UID: \"907caf08-9b81-40eb-937f-777750801bfe\") " pod="openshift-ingress/payload-processing-e2e-shared-a-1badbb-7ff76ff76b-mklzc"
Jul 28 04:55:52.474993 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.474952    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/907caf08-9b81-40eb-937f-777750801bfe-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-1badbb-7ff76ff76b-mklzc\" (UID: \"907caf08-9b81-40eb-937f-777750801bfe\") " pod="openshift-ingress/payload-processing-e2e-shared-a-1badbb-7ff76ff76b-mklzc"
Jul 28 04:55:52.474993 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.474985    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2ddf640-7e9d-4af8-8702-9d50954a2fe8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-1badbb-595f44df4d-mm2k8\" (UID: \"f2ddf640-7e9d-4af8-8702-9d50954a2fe8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1badbb-595f44df4d-mm2k8"
Jul 28 04:55:52.475092 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.475053    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pk25\" (UniqueName: \"kubernetes.io/projected/f2ddf640-7e9d-4af8-8702-9d50954a2fe8-kube-api-access-8pk25\") pod \"payload-pre-processing-e2e-shared-a-1badbb-595f44df4d-mm2k8\" (UID: \"f2ddf640-7e9d-4af8-8702-9d50954a2fe8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1badbb-595f44df4d-mm2k8"
Jul 28 04:55:52.549492 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.549430    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1badbb-5fd5858c-sjvg9" event={"ID":"c9bb777e-25c1-4454-b326-96ef0a1941d4","Type":"ContainerStarted","Data":"05478c885b5065902d692ad781ce03f0bca84619546c114e8a5008a7484f5bb1"}
Jul 28 04:55:52.553049 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.553015    2566 generic.go:356] "Generic (PLEG): container finished" podID="bc23c275-9d40-4d1c-aa91-405814c44687" containerID="00c578a6d142d4190bfe1d1419308f0b9f2004ea3fb9c85889e1d297fbf4f306" exitCode=0
Jul 28 04:55:52.553216 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.553076    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c" event={"ID":"bc23c275-9d40-4d1c-aa91-405814c44687","Type":"ContainerDied","Data":"00c578a6d142d4190bfe1d1419308f0b9f2004ea3fb9c85889e1d297fbf4f306"}
Jul 28 04:55:52.553216 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.553156    2566 scope.go:117] "RemoveContainer" containerID="eb8a9d1113cc393fc4586575a332663398d0c4dd5ef20a0c40571699867259b9"
Jul 28 04:55:52.576308 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.576270    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd82k\" (UniqueName: \"kubernetes.io/projected/907caf08-9b81-40eb-937f-777750801bfe-kube-api-access-vd82k\") pod \"payload-processing-e2e-shared-a-1badbb-7ff76ff76b-mklzc\" (UID: \"907caf08-9b81-40eb-937f-777750801bfe\") " pod="openshift-ingress/payload-processing-e2e-shared-a-1badbb-7ff76ff76b-mklzc"
Jul 28 04:55:52.576456 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.576397    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/907caf08-9b81-40eb-937f-777750801bfe-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-1badbb-7ff76ff76b-mklzc\" (UID: \"907caf08-9b81-40eb-937f-777750801bfe\") " pod="openshift-ingress/payload-processing-e2e-shared-a-1badbb-7ff76ff76b-mklzc"
Jul 28 04:55:52.576456 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.576431    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2ddf640-7e9d-4af8-8702-9d50954a2fe8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-1badbb-595f44df4d-mm2k8\" (UID: \"f2ddf640-7e9d-4af8-8702-9d50954a2fe8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1badbb-595f44df4d-mm2k8"
Jul 28 04:55:52.576574 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.576469    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pk25\" (UniqueName: \"kubernetes.io/projected/f2ddf640-7e9d-4af8-8702-9d50954a2fe8-kube-api-access-8pk25\") pod \"payload-pre-processing-e2e-shared-a-1badbb-595f44df4d-mm2k8\" (UID: \"f2ddf640-7e9d-4af8-8702-9d50954a2fe8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1badbb-595f44df4d-mm2k8"
Jul 28 04:55:52.577241 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.577219    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/907caf08-9b81-40eb-937f-777750801bfe-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-1badbb-7ff76ff76b-mklzc\" (UID: \"907caf08-9b81-40eb-937f-777750801bfe\") " pod="openshift-ingress/payload-processing-e2e-shared-a-1badbb-7ff76ff76b-mklzc"
Jul 28 04:55:52.577359 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.577319    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2ddf640-7e9d-4af8-8702-9d50954a2fe8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-1badbb-595f44df4d-mm2k8\" (UID: \"f2ddf640-7e9d-4af8-8702-9d50954a2fe8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1badbb-595f44df4d-mm2k8"
Jul 28 04:55:52.584480 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.584448    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd82k\" (UniqueName: \"kubernetes.io/projected/907caf08-9b81-40eb-937f-777750801bfe-kube-api-access-vd82k\") pod \"payload-processing-e2e-shared-a-1badbb-7ff76ff76b-mklzc\" (UID: \"907caf08-9b81-40eb-937f-777750801bfe\") " pod="openshift-ingress/payload-processing-e2e-shared-a-1badbb-7ff76ff76b-mklzc"
Jul 28 04:55:52.584940 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.584915    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pk25\" (UniqueName: \"kubernetes.io/projected/f2ddf640-7e9d-4af8-8702-9d50954a2fe8-kube-api-access-8pk25\") pod \"payload-pre-processing-e2e-shared-a-1badbb-595f44df4d-mm2k8\" (UID: \"f2ddf640-7e9d-4af8-8702-9d50954a2fe8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1badbb-595f44df4d-mm2k8"
Jul 28 04:55:52.610434 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.610410    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c"
Jul 28 04:55:52.677872 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.677826    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc23c275-9d40-4d1c-aa91-405814c44687-credential-socket\") pod \"bc23c275-9d40-4d1c-aa91-405814c44687\" (UID: \"bc23c275-9d40-4d1c-aa91-405814c44687\") "
Jul 28 04:55:52.677997 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.677899    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc23c275-9d40-4d1c-aa91-405814c44687-istio-podinfo\") pod \"bc23c275-9d40-4d1c-aa91-405814c44687\" (UID: \"bc23c275-9d40-4d1c-aa91-405814c44687\") "
Jul 28 04:55:52.677997 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.677934    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc23c275-9d40-4d1c-aa91-405814c44687-istiod-ca-cert\") pod \"bc23c275-9d40-4d1c-aa91-405814c44687\" (UID: \"bc23c275-9d40-4d1c-aa91-405814c44687\") "
Jul 28 04:55:52.677997 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.677963    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc23c275-9d40-4d1c-aa91-405814c44687-workload-certs\") pod \"bc23c275-9d40-4d1c-aa91-405814c44687\" (UID: \"bc23c275-9d40-4d1c-aa91-405814c44687\") "
Jul 28 04:55:52.678237 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.678027    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc23c275-9d40-4d1c-aa91-405814c44687-istio-token\") pod \"bc23c275-9d40-4d1c-aa91-405814c44687\" (UID: \"bc23c275-9d40-4d1c-aa91-405814c44687\") "
Jul 28 04:55:52.678237 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.678065    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc23c275-9d40-4d1c-aa91-405814c44687-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bc23c275-9d40-4d1c-aa91-405814c44687" (UID: "bc23c275-9d40-4d1c-aa91-405814c44687"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:55:52.678237 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.678077    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc23c275-9d40-4d1c-aa91-405814c44687-workload-socket\") pod \"bc23c275-9d40-4d1c-aa91-405814c44687\" (UID: \"bc23c275-9d40-4d1c-aa91-405814c44687\") "
Jul 28 04:55:52.678237 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.678134    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc23c275-9d40-4d1c-aa91-405814c44687-istio-data\") pod \"bc23c275-9d40-4d1c-aa91-405814c44687\" (UID: \"bc23c275-9d40-4d1c-aa91-405814c44687\") "
Jul 28 04:55:52.678237 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.678161    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qg4lt\" (UniqueName: \"kubernetes.io/projected/bc23c275-9d40-4d1c-aa91-405814c44687-kube-api-access-qg4lt\") pod \"bc23c275-9d40-4d1c-aa91-405814c44687\" (UID: \"bc23c275-9d40-4d1c-aa91-405814c44687\") "
Jul 28 04:55:52.678237 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.678219    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc23c275-9d40-4d1c-aa91-405814c44687-istio-envoy\") pod \"bc23c275-9d40-4d1c-aa91-405814c44687\" (UID: \"bc23c275-9d40-4d1c-aa91-405814c44687\") "
Jul 28 04:55:52.678561 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.678259    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc23c275-9d40-4d1c-aa91-405814c44687-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bc23c275-9d40-4d1c-aa91-405814c44687" (UID: "bc23c275-9d40-4d1c-aa91-405814c44687"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:55:52.678561 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.678434    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc23c275-9d40-4d1c-aa91-405814c44687-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bc23c275-9d40-4d1c-aa91-405814c44687" (UID: "bc23c275-9d40-4d1c-aa91-405814c44687"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:55:52.678674 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.678630    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc23c275-9d40-4d1c-aa91-405814c44687-credential-socket\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:55:52.678674 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.678650    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc23c275-9d40-4d1c-aa91-405814c44687-workload-certs\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:55:52.678674 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.678664    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc23c275-9d40-4d1c-aa91-405814c44687-workload-socket\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:55:52.679023 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.678988    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc23c275-9d40-4d1c-aa91-405814c44687-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bc23c275-9d40-4d1c-aa91-405814c44687" (UID: "bc23c275-9d40-4d1c-aa91-405814c44687"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 04:55:52.679624 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.679599    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc23c275-9d40-4d1c-aa91-405814c44687-istio-data" (OuterVolumeSpecName: "istio-data") pod "bc23c275-9d40-4d1c-aa91-405814c44687" (UID: "bc23c275-9d40-4d1c-aa91-405814c44687"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:55:52.680609 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.680571    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc23c275-9d40-4d1c-aa91-405814c44687-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bc23c275-9d40-4d1c-aa91-405814c44687" (UID: "bc23c275-9d40-4d1c-aa91-405814c44687"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 04:55:52.680796 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.680774    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc23c275-9d40-4d1c-aa91-405814c44687-kube-api-access-qg4lt" (OuterVolumeSpecName: "kube-api-access-qg4lt") pod "bc23c275-9d40-4d1c-aa91-405814c44687" (UID: "bc23c275-9d40-4d1c-aa91-405814c44687"). InnerVolumeSpecName "kube-api-access-qg4lt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:55:52.681140 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.681122    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc23c275-9d40-4d1c-aa91-405814c44687-istio-token" (OuterVolumeSpecName: "istio-token") pod "bc23c275-9d40-4d1c-aa91-405814c44687" (UID: "bc23c275-9d40-4d1c-aa91-405814c44687"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 04:55:52.681140 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.681127    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc23c275-9d40-4d1c-aa91-405814c44687-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bc23c275-9d40-4d1c-aa91-405814c44687" (UID: "bc23c275-9d40-4d1c-aa91-405814c44687"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 04:55:52.744696 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.744663    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1badbb-595f44df4d-mm2k8"
Jul 28 04:55:52.761789 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.761760    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-1badbb-7ff76ff76b-mklzc"
Jul 28 04:55:52.779410 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.779381    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc23c275-9d40-4d1c-aa91-405814c44687-istio-podinfo\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:55:52.779530 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.779413    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc23c275-9d40-4d1c-aa91-405814c44687-istiod-ca-cert\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:55:52.779530 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.779426    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc23c275-9d40-4d1c-aa91-405814c44687-istio-token\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:55:52.779530 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.779434    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc23c275-9d40-4d1c-aa91-405814c44687-istio-data\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:55:52.779530 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.779442    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg4lt\" (UniqueName: \"kubernetes.io/projected/bc23c275-9d40-4d1c-aa91-405814c44687-kube-api-access-qg4lt\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:55:52.779530 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.779451    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc23c275-9d40-4d1c-aa91-405814c44687-istio-envoy\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 04:55:52.892100 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.891996    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-1badbb-595f44df4d-mm2k8"]
Jul 28 04:55:52.942999 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:52.942978    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-1badbb-7ff76ff76b-mklzc"]
Jul 28 04:55:52.944999 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:55:52.944975    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod907caf08_9b81_40eb_937f_777750801bfe.slice/crio-9e07166e7b0505e49c2ddddb741b5330aa2e881f5ad09efcbfc02bbb2990dd85 WatchSource:0}: Error finding container 9e07166e7b0505e49c2ddddb741b5330aa2e881f5ad09efcbfc02bbb2990dd85: Status 404 returned error can't find the container with id 9e07166e7b0505e49c2ddddb741b5330aa2e881f5ad09efcbfc02bbb2990dd85
Jul 28 04:55:53.558980 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:53.558944    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1badbb-595f44df4d-mm2k8" event={"ID":"f2ddf640-7e9d-4af8-8702-9d50954a2fe8","Type":"ContainerStarted","Data":"d05aa4f6e7689160c429d49dcc3a892ee89484ca70d75997ad803cdec5c1cef1"}
Jul 28 04:55:53.559178 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:53.559003    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1badbb-595f44df4d-mm2k8"
Jul 28 04:55:53.559178 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:53.559028    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1badbb-595f44df4d-mm2k8" event={"ID":"f2ddf640-7e9d-4af8-8702-9d50954a2fe8","Type":"ContainerStarted","Data":"60b60028695ac923744032bc57daf4a5766ea98ae402ce6a78e2c9cc2e7eb371"}
Jul 28 04:55:53.560239 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:53.560208    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c"
Jul 28 04:55:53.560363 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:53.560243    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c" event={"ID":"bc23c275-9d40-4d1c-aa91-405814c44687","Type":"ContainerDied","Data":"ec3d05812ded14a4019685d971c6561d51720972258c74f08ee6ff2f7da249e3"}
Jul 28 04:55:53.560363 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:53.560300    2566 scope.go:117] "RemoveContainer" containerID="00c578a6d142d4190bfe1d1419308f0b9f2004ea3fb9c85889e1d297fbf4f306"
Jul 28 04:55:53.561699 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:53.561673    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-1badbb-7ff76ff76b-mklzc" event={"ID":"907caf08-9b81-40eb-937f-777750801bfe","Type":"ContainerStarted","Data":"ea3d5d30451c5155b32a3cbf0d0a5e0e806bef194116ca0b220917c43f616d93"}
Jul 28 04:55:53.561805 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:53.561720    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-1badbb-7ff76ff76b-mklzc" event={"ID":"907caf08-9b81-40eb-937f-777750801bfe","Type":"ContainerStarted","Data":"9e07166e7b0505e49c2ddddb741b5330aa2e881f5ad09efcbfc02bbb2990dd85"}
Jul 28 04:55:53.561805 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:53.561750    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-1badbb-7ff76ff76b-mklzc"
Jul 28 04:55:53.563508 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:53.563483    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1badbb-5fd5858c-sjvg9" event={"ID":"c9bb777e-25c1-4454-b326-96ef0a1941d4","Type":"ContainerStarted","Data":"8e2d67b88f3f110ad5d488a59aacff341c94ae4fe3bc8e402cc8b2d67658aa62"}
Jul 28 04:55:53.563627 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:53.563610    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1badbb-5fd5858c-sjvg9"
Jul 28 04:55:53.575869 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:53.575811    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1badbb-595f44df4d-mm2k8" podStartSLOduration=1.575798343 podStartE2EDuration="1.575798343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:55:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:55:53.573770014 +0000 UTC m=+2653.963946034" watchObservedRunningTime="2026-07-28 04:55:53.575798343 +0000 UTC m=+2653.965974363"
Jul 28 04:55:53.589850 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:53.589822    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c"]
Jul 28 04:55:53.593360 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:53.593335    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-1b2859-openshift-default-76859d9644-gcs2c"]
Jul 28 04:55:53.606973 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:53.606933    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1badbb-5fd5858c-sjvg9" podStartSLOduration=2.198554367 podStartE2EDuration="2.606920988s" totalImagesPullingTime="408.366621ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:55:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:55:53.605721344 +0000 UTC m=+2653.995897366" watchObservedRunningTime="2026-07-28 04:55:53.606920988 +0000 UTC m=+2653.997097002"
Jul 28 04:55:53.621930 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:53.621885    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-1badbb-7ff76ff76b-mklzc" podStartSLOduration=1.6218709009999999 podStartE2EDuration="1.621870901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:55:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:55:53.620778603 +0000 UTC m=+2654.010954626" watchObservedRunningTime="2026-07-28 04:55:53.621870901 +0000 UTC m=+2654.012046954"
Jul 28 04:55:54.088801 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:54.088769    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc23c275-9d40-4d1c-aa91-405814c44687" path="/var/lib/kubelet/pods/bc23c275-9d40-4d1c-aa91-405814c44687/volumes"
Jul 28 04:55:59.574782 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:55:59.574755    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1badbb-5fd5858c-sjvg9"
Jul 28 04:56:00.168066 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:00.168024    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-92c557-858bb566bf-t5xks"]
Jul 28 04:56:00.168520 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:00.168501    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc23c275-9d40-4d1c-aa91-405814c44687" containerName="istio-proxy"
Jul 28 04:56:00.168646 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:00.168525    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc23c275-9d40-4d1c-aa91-405814c44687" containerName="istio-proxy"
Jul 28 04:56:00.168646 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:00.168642    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc23c275-9d40-4d1c-aa91-405814c44687" containerName="istio-proxy"
Jul 28 04:56:00.168775 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:00.168659    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc23c275-9d40-4d1c-aa91-405814c44687" containerName="istio-proxy"
Jul 28 04:56:00.173489 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:00.173468    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-92c557-858bb566bf-t5xks"
Jul 28 04:56:00.176149 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:00.176122    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-92c557-serving-cert\""
Jul 28 04:56:00.184678 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:00.184654    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-92c557-858bb566bf-t5xks"]
Jul 28 04:56:00.254681 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:00.254643    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aea644ae-603c-4f83-aded-079a83d3441f-maas-api-tls\") pod \"maas-api-e2e-shared-b-92c557-858bb566bf-t5xks\" (UID: \"aea644ae-603c-4f83-aded-079a83d3441f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-92c557-858bb566bf-t5xks"
Jul 28 04:56:00.254885 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:00.254787    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n65g2\" (UniqueName: \"kubernetes.io/projected/aea644ae-603c-4f83-aded-079a83d3441f-kube-api-access-n65g2\") pod \"maas-api-e2e-shared-b-92c557-858bb566bf-t5xks\" (UID: \"aea644ae-603c-4f83-aded-079a83d3441f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-92c557-858bb566bf-t5xks"
Jul 28 04:56:00.254885 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:00.254872    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aea644ae-603c-4f83-aded-079a83d3441f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-92c557-858bb566bf-t5xks\" (UID: \"aea644ae-603c-4f83-aded-079a83d3441f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-92c557-858bb566bf-t5xks"
Jul 28 04:56:00.355365 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:00.355333    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n65g2\" (UniqueName: \"kubernetes.io/projected/aea644ae-603c-4f83-aded-079a83d3441f-kube-api-access-n65g2\") pod \"maas-api-e2e-shared-b-92c557-858bb566bf-t5xks\" (UID: \"aea644ae-603c-4f83-aded-079a83d3441f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-92c557-858bb566bf-t5xks"
Jul 28 04:56:00.355547 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:00.355391    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aea644ae-603c-4f83-aded-079a83d3441f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-92c557-858bb566bf-t5xks\" (UID: \"aea644ae-603c-4f83-aded-079a83d3441f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-92c557-858bb566bf-t5xks"
Jul 28 04:56:00.355547 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:00.355440    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aea644ae-603c-4f83-aded-079a83d3441f-maas-api-tls\") pod \"maas-api-e2e-shared-b-92c557-858bb566bf-t5xks\" (UID: \"aea644ae-603c-4f83-aded-079a83d3441f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-92c557-858bb566bf-t5xks"
Jul 28 04:56:00.356234 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:00.356206    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aea644ae-603c-4f83-aded-079a83d3441f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-92c557-858bb566bf-t5xks\" (UID: \"aea644ae-603c-4f83-aded-079a83d3441f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-92c557-858bb566bf-t5xks"
Jul 28 04:56:00.357920 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:00.357891    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aea644ae-603c-4f83-aded-079a83d3441f-maas-api-tls\") pod \"maas-api-e2e-shared-b-92c557-858bb566bf-t5xks\" (UID: \"aea644ae-603c-4f83-aded-079a83d3441f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-92c557-858bb566bf-t5xks"
Jul 28 04:56:00.363416 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:00.363394    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n65g2\" (UniqueName: \"kubernetes.io/projected/aea644ae-603c-4f83-aded-079a83d3441f-kube-api-access-n65g2\") pod \"maas-api-e2e-shared-b-92c557-858bb566bf-t5xks\" (UID: \"aea644ae-603c-4f83-aded-079a83d3441f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-92c557-858bb566bf-t5xks"
Jul 28 04:56:00.488708 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:00.488606    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-92c557-858bb566bf-t5xks"
Jul 28 04:56:00.676798 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:00.676772    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-92c557-858bb566bf-t5xks"]
Jul 28 04:56:00.679942 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:56:00.679916    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaea644ae_603c_4f83_aded_079a83d3441f.slice/crio-987f15e3c509408e2cd2152f74a83c8a84b90b9ee5707e1e5fbceaede7f159dd WatchSource:0}: Error finding container 987f15e3c509408e2cd2152f74a83c8a84b90b9ee5707e1e5fbceaede7f159dd: Status 404 returned error can't find the container with id 987f15e3c509408e2cd2152f74a83c8a84b90b9ee5707e1e5fbceaede7f159dd
Jul 28 04:56:00.993466 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:00.993432    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-92c557-6f4d7cf958-mxbgg"]
Jul 28 04:56:00.993936 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:00.993917    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc23c275-9d40-4d1c-aa91-405814c44687" containerName="istio-proxy"
Jul 28 04:56:00.994021 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:00.993939    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc23c275-9d40-4d1c-aa91-405814c44687" containerName="istio-proxy"
Jul 28 04:56:00.997184 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:00.997080    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-92c557-6f4d7cf958-mxbgg"
Jul 28 04:56:00.999542 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:00.999521    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-92c557\""
Jul 28 04:56:00.999646 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:00.999624    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-92c557-dockercfg-cjwxj\""
Jul 28 04:56:01.006745 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:01.006715    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-92c557-6f4d7cf958-mxbgg"]
Jul 28 04:56:01.062713 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:01.062687    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtsz6\" (UniqueName: \"kubernetes.io/projected/d17d74fd-cd53-4292-a641-f8ca86d4b0d5-kube-api-access-jtsz6\") pod \"payload-processing-e2e-shared-b-92c557-6f4d7cf958-mxbgg\" (UID: \"d17d74fd-cd53-4292-a641-f8ca86d4b0d5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-92c557-6f4d7cf958-mxbgg"
Jul 28 04:56:01.062831 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:01.062790    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d17d74fd-cd53-4292-a641-f8ca86d4b0d5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-92c557-6f4d7cf958-mxbgg\" (UID: \"d17d74fd-cd53-4292-a641-f8ca86d4b0d5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-92c557-6f4d7cf958-mxbgg"
Jul 28 04:56:01.163388 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:01.163348    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d17d74fd-cd53-4292-a641-f8ca86d4b0d5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-92c557-6f4d7cf958-mxbgg\" (UID: \"d17d74fd-cd53-4292-a641-f8ca86d4b0d5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-92c557-6f4d7cf958-mxbgg"
Jul 28 04:56:01.163552 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:01.163432    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtsz6\" (UniqueName: \"kubernetes.io/projected/d17d74fd-cd53-4292-a641-f8ca86d4b0d5-kube-api-access-jtsz6\") pod \"payload-processing-e2e-shared-b-92c557-6f4d7cf958-mxbgg\" (UID: \"d17d74fd-cd53-4292-a641-f8ca86d4b0d5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-92c557-6f4d7cf958-mxbgg"
Jul 28 04:56:01.164161 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:01.164143    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d17d74fd-cd53-4292-a641-f8ca86d4b0d5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-92c557-6f4d7cf958-mxbgg\" (UID: \"d17d74fd-cd53-4292-a641-f8ca86d4b0d5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-92c557-6f4d7cf958-mxbgg"
Jul 28 04:56:01.172069 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:01.172039    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtsz6\" (UniqueName: \"kubernetes.io/projected/d17d74fd-cd53-4292-a641-f8ca86d4b0d5-kube-api-access-jtsz6\") pod \"payload-processing-e2e-shared-b-92c557-6f4d7cf958-mxbgg\" (UID: \"d17d74fd-cd53-4292-a641-f8ca86d4b0d5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-92c557-6f4d7cf958-mxbgg"
Jul 28 04:56:01.311045 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:01.311004    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-92c557-6f4d7cf958-mxbgg"
Jul 28 04:56:01.458170 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:01.458138    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-92c557-6f4d7cf958-mxbgg"]
Jul 28 04:56:01.459886 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:56:01.459857    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd17d74fd_cd53_4292_a641_f8ca86d4b0d5.slice/crio-c9aae3d95d3a91e0a9f756a38085ba3dcf40c859a7aabc6c87ce55e6bd6260e2 WatchSource:0}: Error finding container c9aae3d95d3a91e0a9f756a38085ba3dcf40c859a7aabc6c87ce55e6bd6260e2: Status 404 returned error can't find the container with id c9aae3d95d3a91e0a9f756a38085ba3dcf40c859a7aabc6c87ce55e6bd6260e2
Jul 28 04:56:01.606055 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:01.605945    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-92c557-858bb566bf-t5xks" event={"ID":"aea644ae-603c-4f83-aded-079a83d3441f","Type":"ContainerStarted","Data":"67726e08c494dc30b011db96a4e33f12a00aa29cf091f3565924b81b4e2a50fc"}
Jul 28 04:56:01.606225 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:01.606061    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-92c557-858bb566bf-t5xks" event={"ID":"aea644ae-603c-4f83-aded-079a83d3441f","Type":"ContainerStarted","Data":"987f15e3c509408e2cd2152f74a83c8a84b90b9ee5707e1e5fbceaede7f159dd"}
Jul 28 04:56:01.606225 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:01.606175    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-92c557-858bb566bf-t5xks"
Jul 28 04:56:01.608020 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:01.607939    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-92c557-6f4d7cf958-mxbgg" event={"ID":"d17d74fd-cd53-4292-a641-f8ca86d4b0d5","Type":"ContainerStarted","Data":"6e77dd842d7d08b72cc8101d8ca79379a047da7e326703e489a860ba21907613"}
Jul 28 04:56:01.608020 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:01.607989    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-92c557-6f4d7cf958-mxbgg" event={"ID":"d17d74fd-cd53-4292-a641-f8ca86d4b0d5","Type":"ContainerStarted","Data":"c9aae3d95d3a91e0a9f756a38085ba3dcf40c859a7aabc6c87ce55e6bd6260e2"}
Jul 28 04:56:01.608020 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:01.608017    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-92c557-6f4d7cf958-mxbgg"
Jul 28 04:56:01.622944 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:01.622885    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-92c557-858bb566bf-t5xks" podStartSLOduration=1.265530514 podStartE2EDuration="1.62286293s" totalImagesPullingTime="357.332416ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:56:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:56:01.620876487 +0000 UTC m=+2662.011052517" watchObservedRunningTime="2026-07-28 04:56:01.62286293 +0000 UTC m=+2662.013038952"
Jul 28 04:56:01.637507 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:01.637444    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-92c557-6f4d7cf958-mxbgg" podStartSLOduration=1.6374236290000002 podStartE2EDuration="1.637423629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:56:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:56:01.63461205 +0000 UTC m=+2662.024788073" watchObservedRunningTime="2026-07-28 04:56:01.637423629 +0000 UTC m=+2662.027599651"
Jul 28 04:56:04.572480 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:04.572445    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1badbb-595f44df4d-mm2k8"
Jul 28 04:56:04.572981 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:04.572541    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-1badbb-7ff76ff76b-mklzc"
Jul 28 04:56:04.628271 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:04.628238    2566 generic.go:356] "Generic (PLEG): container finished" podID="abc8eda5-4cd4-4570-b474-c525a74eb3c6" containerID="fb4f2119ef0b5ff6db30de8277e23a49fe72bdd60d16dc12deeb8688ecadf4dd" exitCode=0
Jul 28 04:56:04.628440 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:04.628321    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz" event={"ID":"abc8eda5-4cd4-4570-b474-c525a74eb3c6","Type":"ContainerDied","Data":"fb4f2119ef0b5ff6db30de8277e23a49fe72bdd60d16dc12deeb8688ecadf4dd"}
Jul 28 04:56:04.628778 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:04.628762    2566 scope.go:117] "RemoveContainer" containerID="fb4f2119ef0b5ff6db30de8277e23a49fe72bdd60d16dc12deeb8688ecadf4dd"
Jul 28 04:56:04.630041 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:04.630014    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:56:04.630125 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:04.630078    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:56:04.630125 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:04.630108    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 04:56:05.536854 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:05.536811    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz"
Jul 28 04:56:05.636545 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:05.636508    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz" event={"ID":"abc8eda5-4cd4-4570-b474-c525a74eb3c6","Type":"ContainerStarted","Data":"2b8afaa917f287b96c09e2645a0cd180f42c0781c27ceded912494bde94ccb4e"}
Jul 28 04:56:06.559087 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:06.559049    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz"
Jul 28 04:56:06.560335 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:06.560311    2566 patch_prober.go:28] interesting pod/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Jul 28 04:56:06.560391 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:06.560362    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz" podUID="abc8eda5-4cd4-4570-b474-c525a74eb3c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Jul 28 04:56:07.559042 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:07.559002    2566 patch_prober.go:28] interesting pod/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Jul 28 04:56:07.559432 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:07.559062    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz" podUID="abc8eda5-4cd4-4570-b474-c525a74eb3c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Jul 28 04:56:07.631157 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:07.631123    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-92c557-858bb566bf-t5xks"
Jul 28 04:56:08.559365 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:08.559323    2566 patch_prober.go:28] interesting pod/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Jul 28 04:56:08.559766 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:08.559391    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz" podUID="abc8eda5-4cd4-4570-b474-c525a74eb3c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Jul 28 04:56:09.562405 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:09.562375    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz"
Jul 28 04:56:09.562783 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:09.562646    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz"
Jul 28 04:56:09.563367 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:09.563351    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz"
Jul 28 04:56:12.615816 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:12.615787    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-92c557-6f4d7cf958-mxbgg"
Jul 28 04:56:12.891718 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:12.891635    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-1badbb/auth-test-model-1badbb-kserve-65b655568c-7dk52"]
Jul 28 04:56:12.895445 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:12.895427    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1badbb/auth-test-model-1badbb-kserve-65b655568c-7dk52"
Jul 28 04:56:12.898203 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:12.898180    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1badbb\"/\"auth-test-model-1badbb-kserve-self-signed-certs\""
Jul 28 04:56:12.898303 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:12.898204    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1badbb\"/\"kube-root-ca.crt\""
Jul 28 04:56:12.899357 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:12.899338    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1badbb\"/\"default-dockercfg-z2fqf\""
Jul 28 04:56:12.899463 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:12.899371    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1badbb\"/\"openshift-service-ca.crt\""
Jul 28 04:56:12.903383 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:12.903361    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1badbb/auth-test-model-1badbb-kserve-65b655568c-7dk52"]
Jul 28 04:56:12.984950 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:12.984916    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dmvh\" (UniqueName: \"kubernetes.io/projected/48f2ea38-5dae-48f5-9139-82f051c9b519-kube-api-access-7dmvh\") pod \"auth-test-model-1badbb-kserve-65b655568c-7dk52\" (UID: \"48f2ea38-5dae-48f5-9139-82f051c9b519\") " pod="ai-tenant-e2e-shared-a-1badbb/auth-test-model-1badbb-kserve-65b655568c-7dk52"
Jul 28 04:56:12.985133 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:12.984958    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48f2ea38-5dae-48f5-9139-82f051c9b519-model-cache\") pod \"auth-test-model-1badbb-kserve-65b655568c-7dk52\" (UID: \"48f2ea38-5dae-48f5-9139-82f051c9b519\") " pod="ai-tenant-e2e-shared-a-1badbb/auth-test-model-1badbb-kserve-65b655568c-7dk52"
Jul 28 04:56:12.985133 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:12.984989    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48f2ea38-5dae-48f5-9139-82f051c9b519-tmp-dir\") pod \"auth-test-model-1badbb-kserve-65b655568c-7dk52\" (UID: \"48f2ea38-5dae-48f5-9139-82f051c9b519\") " pod="ai-tenant-e2e-shared-a-1badbb/auth-test-model-1badbb-kserve-65b655568c-7dk52"
Jul 28 04:56:12.985133 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:12.985075    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48f2ea38-5dae-48f5-9139-82f051c9b519-home\") pod \"auth-test-model-1badbb-kserve-65b655568c-7dk52\" (UID: \"48f2ea38-5dae-48f5-9139-82f051c9b519\") " pod="ai-tenant-e2e-shared-a-1badbb/auth-test-model-1badbb-kserve-65b655568c-7dk52"
Jul 28 04:56:12.985133 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:12.985121    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48f2ea38-5dae-48f5-9139-82f051c9b519-dshm\") pod \"auth-test-model-1badbb-kserve-65b655568c-7dk52\" (UID: \"48f2ea38-5dae-48f5-9139-82f051c9b519\") " pod="ai-tenant-e2e-shared-a-1badbb/auth-test-model-1badbb-kserve-65b655568c-7dk52"
Jul 28 04:56:12.985333 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:12.985224    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48f2ea38-5dae-48f5-9139-82f051c9b519-tls-certs\") pod \"auth-test-model-1badbb-kserve-65b655568c-7dk52\" (UID: \"48f2ea38-5dae-48f5-9139-82f051c9b519\") " pod="ai-tenant-e2e-shared-a-1badbb/auth-test-model-1badbb-kserve-65b655568c-7dk52"
Jul 28 04:56:13.085772 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:13.085741    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dmvh\" (UniqueName: \"kubernetes.io/projected/48f2ea38-5dae-48f5-9139-82f051c9b519-kube-api-access-7dmvh\") pod \"auth-test-model-1badbb-kserve-65b655568c-7dk52\" (UID: \"48f2ea38-5dae-48f5-9139-82f051c9b519\") " pod="ai-tenant-e2e-shared-a-1badbb/auth-test-model-1badbb-kserve-65b655568c-7dk52"
Jul 28 04:56:13.085983 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:13.085778    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48f2ea38-5dae-48f5-9139-82f051c9b519-model-cache\") pod \"auth-test-model-1badbb-kserve-65b655568c-7dk52\" (UID: \"48f2ea38-5dae-48f5-9139-82f051c9b519\") " pod="ai-tenant-e2e-shared-a-1badbb/auth-test-model-1badbb-kserve-65b655568c-7dk52"
Jul 28 04:56:13.085983 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:13.085801    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48f2ea38-5dae-48f5-9139-82f051c9b519-tmp-dir\") pod \"auth-test-model-1badbb-kserve-65b655568c-7dk52\" (UID: \"48f2ea38-5dae-48f5-9139-82f051c9b519\") " pod="ai-tenant-e2e-shared-a-1badbb/auth-test-model-1badbb-kserve-65b655568c-7dk52"
Jul 28 04:56:13.085983 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:13.085856    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48f2ea38-5dae-48f5-9139-82f051c9b519-home\") pod \"auth-test-model-1badbb-kserve-65b655568c-7dk52\" (UID: \"48f2ea38-5dae-48f5-9139-82f051c9b519\") " pod="ai-tenant-e2e-shared-a-1badbb/auth-test-model-1badbb-kserve-65b655568c-7dk52"
Jul 28 04:56:13.085983 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:13.085904    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48f2ea38-5dae-48f5-9139-82f051c9b519-dshm\") pod \"auth-test-model-1badbb-kserve-65b655568c-7dk52\" (UID: \"48f2ea38-5dae-48f5-9139-82f051c9b519\") " pod="ai-tenant-e2e-shared-a-1badbb/auth-test-model-1badbb-kserve-65b655568c-7dk52"
Jul 28 04:56:13.085983 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:13.085945    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48f2ea38-5dae-48f5-9139-82f051c9b519-tls-certs\") pod \"auth-test-model-1badbb-kserve-65b655568c-7dk52\" (UID: \"48f2ea38-5dae-48f5-9139-82f051c9b519\") " pod="ai-tenant-e2e-shared-a-1badbb/auth-test-model-1badbb-kserve-65b655568c-7dk52"
Jul 28 04:56:13.086251 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:13.086230    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48f2ea38-5dae-48f5-9139-82f051c9b519-model-cache\") pod \"auth-test-model-1badbb-kserve-65b655568c-7dk52\" (UID: \"48f2ea38-5dae-48f5-9139-82f051c9b519\") " pod="ai-tenant-e2e-shared-a-1badbb/auth-test-model-1badbb-kserve-65b655568c-7dk52"
Jul 28 04:56:13.086304 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:13.086269    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48f2ea38-5dae-48f5-9139-82f051c9b519-tmp-dir\") pod \"auth-test-model-1badbb-kserve-65b655568c-7dk52\" (UID: \"48f2ea38-5dae-48f5-9139-82f051c9b519\") " pod="ai-tenant-e2e-shared-a-1badbb/auth-test-model-1badbb-kserve-65b655568c-7dk52"
Jul 28 04:56:13.086354 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:13.086293    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48f2ea38-5dae-48f5-9139-82f051c9b519-home\") pod \"auth-test-model-1badbb-kserve-65b655568c-7dk52\" (UID: \"48f2ea38-5dae-48f5-9139-82f051c9b519\") " pod="ai-tenant-e2e-shared-a-1badbb/auth-test-model-1badbb-kserve-65b655568c-7dk52"
Jul 28 04:56:13.088335 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:13.088308    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48f2ea38-5dae-48f5-9139-82f051c9b519-dshm\") pod \"auth-test-model-1badbb-kserve-65b655568c-7dk52\" (UID: \"48f2ea38-5dae-48f5-9139-82f051c9b519\") " pod="ai-tenant-e2e-shared-a-1badbb/auth-test-model-1badbb-kserve-65b655568c-7dk52"
Jul 28 04:56:13.088460 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:13.088440    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48f2ea38-5dae-48f5-9139-82f051c9b519-tls-certs\") pod \"auth-test-model-1badbb-kserve-65b655568c-7dk52\" (UID: \"48f2ea38-5dae-48f5-9139-82f051c9b519\") " pod="ai-tenant-e2e-shared-a-1badbb/auth-test-model-1badbb-kserve-65b655568c-7dk52"
Jul 28 04:56:13.093608 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:13.093586    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dmvh\" (UniqueName: \"kubernetes.io/projected/48f2ea38-5dae-48f5-9139-82f051c9b519-kube-api-access-7dmvh\") pod \"auth-test-model-1badbb-kserve-65b655568c-7dk52\" (UID: \"48f2ea38-5dae-48f5-9139-82f051c9b519\") " pod="ai-tenant-e2e-shared-a-1badbb/auth-test-model-1badbb-kserve-65b655568c-7dk52"
Jul 28 04:56:13.205732 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:13.205649    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1badbb/auth-test-model-1badbb-kserve-65b655568c-7dk52"
Jul 28 04:56:13.345086 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:13.345060    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1badbb/auth-test-model-1badbb-kserve-65b655568c-7dk52"]
Jul 28 04:56:13.347396 ip-10-0-129-187 kubenswrapper[2566]: W0728 04:56:13.347357    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48f2ea38_5dae_48f5_9139_82f051c9b519.slice/crio-a4e1a3dcf04c29bbae87a9bf75b5b83cecf9a480aec82197c0507a5b043df790 WatchSource:0}: Error finding container a4e1a3dcf04c29bbae87a9bf75b5b83cecf9a480aec82197c0507a5b043df790: Status 404 returned error can't find the container with id a4e1a3dcf04c29bbae87a9bf75b5b83cecf9a480aec82197c0507a5b043df790
Jul 28 04:56:13.680646 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:13.680612    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1badbb/auth-test-model-1badbb-kserve-65b655568c-7dk52" event={"ID":"48f2ea38-5dae-48f5-9139-82f051c9b519","Type":"ContainerStarted","Data":"becc1f51b4d38a70ccaa04aaea5d75469a447693b11601d4ad9643a735198c13"}
Jul 28 04:56:13.680646 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:13.680652    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1badbb/auth-test-model-1badbb-kserve-65b655568c-7dk52" event={"ID":"48f2ea38-5dae-48f5-9139-82f051c9b519","Type":"ContainerStarted","Data":"a4e1a3dcf04c29bbae87a9bf75b5b83cecf9a480aec82197c0507a5b043df790"}
Jul 28 04:56:13.700687 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:13.700641    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-1badbb/auth-test-model-1badbb-kserve-65b655568c-7dk52" podStartSLOduration=1.700625632 podStartE2EDuration="1.700625632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 04:56:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 04:56:13.698758802 +0000 UTC m=+2674.088934824" watchObservedRunningTime="2026-07-28 04:56:13.700625632 +0000 UTC m=+2674.090801652"
Jul 28 04:56:23.205991 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:23.205953    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-1badbb/auth-test-model-1badbb-kserve-65b655568c-7dk52"
Jul 28 04:56:23.206520 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:23.206013    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-1badbb/auth-test-model-1badbb-kserve-65b655568c-7dk52"
Jul 28 04:56:23.218687 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:23.218661    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-1badbb/auth-test-model-1badbb-kserve-65b655568c-7dk52"
Jul 28 04:56:23.750534 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:23.750500    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-1badbb/auth-test-model-1badbb-kserve-65b655568c-7dk52"
Jul 28 04:56:43.758468 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:43.758437    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4glpr_bfd2d3d8-f287-42d8-83c7-f82185c96735/console-operator/2.log"
Jul 28 04:56:43.764492 ip-10-0-129-187 kubenswrapper[2566]: I0728 04:56:43.764470    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4glpr_bfd2d3d8-f287-42d8-83c7-f82185c96735/console-operator/2.log"
Jul 28 05:01:06.938375 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:06.938343    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1badbb/auth-test-model-1badbb-kserve-65b655568c-7dk52"]
Jul 28 05:01:21.958393 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:21.958353    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-1badbb/auth-test-model-1badbb-kserve-65b655568c-7dk52" podUID="48f2ea38-5dae-48f5-9139-82f051c9b519" containerName="main" containerID="cri-o://becc1f51b4d38a70ccaa04aaea5d75469a447693b11601d4ad9643a735198c13" gracePeriod=45
Jul 28 05:01:22.217654 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:22.217583    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1badbb/auth-test-model-1badbb-kserve-65b655568c-7dk52"
Jul 28 05:01:22.219434 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:22.219409    2566 generic.go:356] "Generic (PLEG): container finished" podID="48f2ea38-5dae-48f5-9139-82f051c9b519" containerID="becc1f51b4d38a70ccaa04aaea5d75469a447693b11601d4ad9643a735198c13" exitCode=0
Jul 28 05:01:22.219553 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:22.219447    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1badbb/auth-test-model-1badbb-kserve-65b655568c-7dk52"
Jul 28 05:01:22.219553 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:22.219500    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1badbb/auth-test-model-1badbb-kserve-65b655568c-7dk52" event={"ID":"48f2ea38-5dae-48f5-9139-82f051c9b519","Type":"ContainerDied","Data":"becc1f51b4d38a70ccaa04aaea5d75469a447693b11601d4ad9643a735198c13"}
Jul 28 05:01:22.219674 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:22.219551    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1badbb/auth-test-model-1badbb-kserve-65b655568c-7dk52" event={"ID":"48f2ea38-5dae-48f5-9139-82f051c9b519","Type":"ContainerDied","Data":"a4e1a3dcf04c29bbae87a9bf75b5b83cecf9a480aec82197c0507a5b043df790"}
Jul 28 05:01:22.219674 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:22.219579    2566 scope.go:117] "RemoveContainer" containerID="becc1f51b4d38a70ccaa04aaea5d75469a447693b11601d4ad9643a735198c13"
Jul 28 05:01:22.228514 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:22.228495    2566 scope.go:117] "RemoveContainer" containerID="becc1f51b4d38a70ccaa04aaea5d75469a447693b11601d4ad9643a735198c13"
Jul 28 05:01:22.228776 ip-10-0-129-187 kubenswrapper[2566]: E0728 05:01:22.228757    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"becc1f51b4d38a70ccaa04aaea5d75469a447693b11601d4ad9643a735198c13\": container with ID starting with becc1f51b4d38a70ccaa04aaea5d75469a447693b11601d4ad9643a735198c13 not found: ID does not exist" containerID="becc1f51b4d38a70ccaa04aaea5d75469a447693b11601d4ad9643a735198c13"
Jul 28 05:01:22.228832 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:22.228784    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"becc1f51b4d38a70ccaa04aaea5d75469a447693b11601d4ad9643a735198c13"} err="failed to get container status \"becc1f51b4d38a70ccaa04aaea5d75469a447693b11601d4ad9643a735198c13\": rpc error: code = NotFound desc = could not find container \"becc1f51b4d38a70ccaa04aaea5d75469a447693b11601d4ad9643a735198c13\": container with ID starting with becc1f51b4d38a70ccaa04aaea5d75469a447693b11601d4ad9643a735198c13 not found: ID does not exist"
Jul 28 05:01:22.323922 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:22.323889    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7dmvh\" (UniqueName: \"kubernetes.io/projected/48f2ea38-5dae-48f5-9139-82f051c9b519-kube-api-access-7dmvh\") pod \"48f2ea38-5dae-48f5-9139-82f051c9b519\" (UID: \"48f2ea38-5dae-48f5-9139-82f051c9b519\") "
Jul 28 05:01:22.324100 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:22.323988    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48f2ea38-5dae-48f5-9139-82f051c9b519-model-cache\") pod \"48f2ea38-5dae-48f5-9139-82f051c9b519\" (UID: \"48f2ea38-5dae-48f5-9139-82f051c9b519\") "
Jul 28 05:01:22.324100 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:22.324028    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48f2ea38-5dae-48f5-9139-82f051c9b519-home\") pod \"48f2ea38-5dae-48f5-9139-82f051c9b519\" (UID: \"48f2ea38-5dae-48f5-9139-82f051c9b519\") "
Jul 28 05:01:22.324189 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:22.324111    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48f2ea38-5dae-48f5-9139-82f051c9b519-dshm\") pod \"48f2ea38-5dae-48f5-9139-82f051c9b519\" (UID: \"48f2ea38-5dae-48f5-9139-82f051c9b519\") "
Jul 28 05:01:22.324189 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:22.324150    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48f2ea38-5dae-48f5-9139-82f051c9b519-tls-certs\") pod \"48f2ea38-5dae-48f5-9139-82f051c9b519\" (UID: \"48f2ea38-5dae-48f5-9139-82f051c9b519\") "
Jul 28 05:01:22.324189 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:22.324184    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48f2ea38-5dae-48f5-9139-82f051c9b519-tmp-dir\") pod \"48f2ea38-5dae-48f5-9139-82f051c9b519\" (UID: \"48f2ea38-5dae-48f5-9139-82f051c9b519\") "
Jul 28 05:01:22.324324 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:22.324290    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48f2ea38-5dae-48f5-9139-82f051c9b519-model-cache" (OuterVolumeSpecName: "model-cache") pod "48f2ea38-5dae-48f5-9139-82f051c9b519" (UID: "48f2ea38-5dae-48f5-9139-82f051c9b519"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 05:01:22.324382 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:22.324335    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48f2ea38-5dae-48f5-9139-82f051c9b519-home" (OuterVolumeSpecName: "home") pod "48f2ea38-5dae-48f5-9139-82f051c9b519" (UID: "48f2ea38-5dae-48f5-9139-82f051c9b519"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 05:01:22.324608 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:22.324547    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48f2ea38-5dae-48f5-9139-82f051c9b519-model-cache\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:01:22.324608 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:22.324572    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48f2ea38-5dae-48f5-9139-82f051c9b519-home\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:01:22.324608 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:22.324561    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48f2ea38-5dae-48f5-9139-82f051c9b519-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "48f2ea38-5dae-48f5-9139-82f051c9b519" (UID: "48f2ea38-5dae-48f5-9139-82f051c9b519"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 05:01:22.326309 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:22.326284    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48f2ea38-5dae-48f5-9139-82f051c9b519-kube-api-access-7dmvh" (OuterVolumeSpecName: "kube-api-access-7dmvh") pod "48f2ea38-5dae-48f5-9139-82f051c9b519" (UID: "48f2ea38-5dae-48f5-9139-82f051c9b519"). InnerVolumeSpecName "kube-api-access-7dmvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 05:01:22.326399 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:22.326306    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48f2ea38-5dae-48f5-9139-82f051c9b519-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "48f2ea38-5dae-48f5-9139-82f051c9b519" (UID: "48f2ea38-5dae-48f5-9139-82f051c9b519"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 05:01:22.326399 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:22.326352    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48f2ea38-5dae-48f5-9139-82f051c9b519-dshm" (OuterVolumeSpecName: "dshm") pod "48f2ea38-5dae-48f5-9139-82f051c9b519" (UID: "48f2ea38-5dae-48f5-9139-82f051c9b519"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 05:01:22.425896 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:22.425861    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48f2ea38-5dae-48f5-9139-82f051c9b519-dshm\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:01:22.425896 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:22.425891    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48f2ea38-5dae-48f5-9139-82f051c9b519-tls-certs\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:01:22.425896 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:22.425901    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48f2ea38-5dae-48f5-9139-82f051c9b519-tmp-dir\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:01:22.426149 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:22.425910    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dmvh\" (UniqueName: \"kubernetes.io/projected/48f2ea38-5dae-48f5-9139-82f051c9b519-kube-api-access-7dmvh\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:01:22.552427 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:22.552390    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1badbb/auth-test-model-1badbb-kserve-65b655568c-7dk52"]
Jul 28 05:01:22.556341 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:22.556318    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-1badbb/auth-test-model-1badbb-kserve-65b655568c-7dk52"]
Jul 28 05:01:24.088545 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:24.088511    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48f2ea38-5dae-48f5-9139-82f051c9b519" path="/var/lib/kubelet/pods/48f2ea38-5dae-48f5-9139-82f051c9b519/volumes"
Jul 28 05:01:28.287872 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:28.287822    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-1badbb/sub-test-model-1badbb-kserve-7c7b8b4955-sdxss"]
Jul 28 05:01:28.288289 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:28.288274    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48f2ea38-5dae-48f5-9139-82f051c9b519" containerName="main"
Jul 28 05:01:28.288333 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:28.288290    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="48f2ea38-5dae-48f5-9139-82f051c9b519" containerName="main"
Jul 28 05:01:28.288365 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:28.288358    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="48f2ea38-5dae-48f5-9139-82f051c9b519" containerName="main"
Jul 28 05:01:28.292822 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:28.292802    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1badbb/sub-test-model-1badbb-kserve-7c7b8b4955-sdxss"
Jul 28 05:01:28.296760 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:28.296731    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1badbb\"/\"kube-root-ca.crt\""
Jul 28 05:01:28.296923 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:28.296768    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1badbb\"/\"sub-test-model-1badbb-kserve-self-signed-certs\""
Jul 28 05:01:28.296923 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:28.296814    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-1badbb\"/\"openshift-service-ca.crt\""
Jul 28 05:01:28.296923 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:28.296908    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-1badbb\"/\"default-dockercfg-z2fqf\""
Jul 28 05:01:28.299523 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:28.299500    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1badbb/sub-test-model-1badbb-kserve-7c7b8b4955-sdxss"]
Jul 28 05:01:28.388694 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:28.388658    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce9444de-e2bd-453b-abea-57de863a7455-tls-certs\") pod \"sub-test-model-1badbb-kserve-7c7b8b4955-sdxss\" (UID: \"ce9444de-e2bd-453b-abea-57de863a7455\") " pod="ai-tenant-e2e-shared-a-1badbb/sub-test-model-1badbb-kserve-7c7b8b4955-sdxss"
Jul 28 05:01:28.388867 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:28.388723    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce9444de-e2bd-453b-abea-57de863a7455-home\") pod \"sub-test-model-1badbb-kserve-7c7b8b4955-sdxss\" (UID: \"ce9444de-e2bd-453b-abea-57de863a7455\") " pod="ai-tenant-e2e-shared-a-1badbb/sub-test-model-1badbb-kserve-7c7b8b4955-sdxss"
Jul 28 05:01:28.388867 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:28.388778    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce9444de-e2bd-453b-abea-57de863a7455-model-cache\") pod \"sub-test-model-1badbb-kserve-7c7b8b4955-sdxss\" (UID: \"ce9444de-e2bd-453b-abea-57de863a7455\") " pod="ai-tenant-e2e-shared-a-1badbb/sub-test-model-1badbb-kserve-7c7b8b4955-sdxss"
Jul 28 05:01:28.388867 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:28.388806    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce9444de-e2bd-453b-abea-57de863a7455-dshm\") pod \"sub-test-model-1badbb-kserve-7c7b8b4955-sdxss\" (UID: \"ce9444de-e2bd-453b-abea-57de863a7455\") " pod="ai-tenant-e2e-shared-a-1badbb/sub-test-model-1badbb-kserve-7c7b8b4955-sdxss"
Jul 28 05:01:28.388867 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:28.388863    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce9444de-e2bd-453b-abea-57de863a7455-tmp-dir\") pod \"sub-test-model-1badbb-kserve-7c7b8b4955-sdxss\" (UID: \"ce9444de-e2bd-453b-abea-57de863a7455\") " pod="ai-tenant-e2e-shared-a-1badbb/sub-test-model-1badbb-kserve-7c7b8b4955-sdxss"
Jul 28 05:01:28.389097 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:28.388958    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ls44\" (UniqueName: \"kubernetes.io/projected/ce9444de-e2bd-453b-abea-57de863a7455-kube-api-access-5ls44\") pod \"sub-test-model-1badbb-kserve-7c7b8b4955-sdxss\" (UID: \"ce9444de-e2bd-453b-abea-57de863a7455\") " pod="ai-tenant-e2e-shared-a-1badbb/sub-test-model-1badbb-kserve-7c7b8b4955-sdxss"
Jul 28 05:01:28.490503 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:28.490466    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce9444de-e2bd-453b-abea-57de863a7455-dshm\") pod \"sub-test-model-1badbb-kserve-7c7b8b4955-sdxss\" (UID: \"ce9444de-e2bd-453b-abea-57de863a7455\") " pod="ai-tenant-e2e-shared-a-1badbb/sub-test-model-1badbb-kserve-7c7b8b4955-sdxss"
Jul 28 05:01:28.490708 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:28.490519    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce9444de-e2bd-453b-abea-57de863a7455-tmp-dir\") pod \"sub-test-model-1badbb-kserve-7c7b8b4955-sdxss\" (UID: \"ce9444de-e2bd-453b-abea-57de863a7455\") " pod="ai-tenant-e2e-shared-a-1badbb/sub-test-model-1badbb-kserve-7c7b8b4955-sdxss"
Jul 28 05:01:28.490708 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:28.490558    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ls44\" (UniqueName: \"kubernetes.io/projected/ce9444de-e2bd-453b-abea-57de863a7455-kube-api-access-5ls44\") pod \"sub-test-model-1badbb-kserve-7c7b8b4955-sdxss\" (UID: \"ce9444de-e2bd-453b-abea-57de863a7455\") " pod="ai-tenant-e2e-shared-a-1badbb/sub-test-model-1badbb-kserve-7c7b8b4955-sdxss"
Jul 28 05:01:28.490708 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:28.490626    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce9444de-e2bd-453b-abea-57de863a7455-tls-certs\") pod \"sub-test-model-1badbb-kserve-7c7b8b4955-sdxss\" (UID: \"ce9444de-e2bd-453b-abea-57de863a7455\") " pod="ai-tenant-e2e-shared-a-1badbb/sub-test-model-1badbb-kserve-7c7b8b4955-sdxss"
Jul 28 05:01:28.490708 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:28.490662    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce9444de-e2bd-453b-abea-57de863a7455-home\") pod \"sub-test-model-1badbb-kserve-7c7b8b4955-sdxss\" (UID: \"ce9444de-e2bd-453b-abea-57de863a7455\") " pod="ai-tenant-e2e-shared-a-1badbb/sub-test-model-1badbb-kserve-7c7b8b4955-sdxss"
Jul 28 05:01:28.490708 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:28.490686    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce9444de-e2bd-453b-abea-57de863a7455-model-cache\") pod \"sub-test-model-1badbb-kserve-7c7b8b4955-sdxss\" (UID: \"ce9444de-e2bd-453b-abea-57de863a7455\") " pod="ai-tenant-e2e-shared-a-1badbb/sub-test-model-1badbb-kserve-7c7b8b4955-sdxss"
Jul 28 05:01:28.491043 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:28.491021    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce9444de-e2bd-453b-abea-57de863a7455-tmp-dir\") pod \"sub-test-model-1badbb-kserve-7c7b8b4955-sdxss\" (UID: \"ce9444de-e2bd-453b-abea-57de863a7455\") " pod="ai-tenant-e2e-shared-a-1badbb/sub-test-model-1badbb-kserve-7c7b8b4955-sdxss"
Jul 28 05:01:28.491103 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:28.491084    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce9444de-e2bd-453b-abea-57de863a7455-model-cache\") pod \"sub-test-model-1badbb-kserve-7c7b8b4955-sdxss\" (UID: \"ce9444de-e2bd-453b-abea-57de863a7455\") " pod="ai-tenant-e2e-shared-a-1badbb/sub-test-model-1badbb-kserve-7c7b8b4955-sdxss"
Jul 28 05:01:28.491155 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:28.491140    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce9444de-e2bd-453b-abea-57de863a7455-home\") pod \"sub-test-model-1badbb-kserve-7c7b8b4955-sdxss\" (UID: \"ce9444de-e2bd-453b-abea-57de863a7455\") " pod="ai-tenant-e2e-shared-a-1badbb/sub-test-model-1badbb-kserve-7c7b8b4955-sdxss"
Jul 28 05:01:28.493169 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:28.493143    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce9444de-e2bd-453b-abea-57de863a7455-dshm\") pod \"sub-test-model-1badbb-kserve-7c7b8b4955-sdxss\" (UID: \"ce9444de-e2bd-453b-abea-57de863a7455\") " pod="ai-tenant-e2e-shared-a-1badbb/sub-test-model-1badbb-kserve-7c7b8b4955-sdxss"
Jul 28 05:01:28.493378 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:28.493363    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce9444de-e2bd-453b-abea-57de863a7455-tls-certs\") pod \"sub-test-model-1badbb-kserve-7c7b8b4955-sdxss\" (UID: \"ce9444de-e2bd-453b-abea-57de863a7455\") " pod="ai-tenant-e2e-shared-a-1badbb/sub-test-model-1badbb-kserve-7c7b8b4955-sdxss"
Jul 28 05:01:28.498699 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:28.498670    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ls44\" (UniqueName: \"kubernetes.io/projected/ce9444de-e2bd-453b-abea-57de863a7455-kube-api-access-5ls44\") pod \"sub-test-model-1badbb-kserve-7c7b8b4955-sdxss\" (UID: \"ce9444de-e2bd-453b-abea-57de863a7455\") " pod="ai-tenant-e2e-shared-a-1badbb/sub-test-model-1badbb-kserve-7c7b8b4955-sdxss"
Jul 28 05:01:28.604220 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:28.604130    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1badbb/sub-test-model-1badbb-kserve-7c7b8b4955-sdxss"
Jul 28 05:01:28.741747 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:28.741724    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-1badbb/sub-test-model-1badbb-kserve-7c7b8b4955-sdxss"]
Jul 28 05:01:28.743028 ip-10-0-129-187 kubenswrapper[2566]: W0728 05:01:28.743004    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce9444de_e2bd_453b_abea_57de863a7455.slice/crio-e69505d19034dbe1a87097059ac1416c79735f3be294bf870832ee3ba21823ce WatchSource:0}: Error finding container e69505d19034dbe1a87097059ac1416c79735f3be294bf870832ee3ba21823ce: Status 404 returned error can't find the container with id e69505d19034dbe1a87097059ac1416c79735f3be294bf870832ee3ba21823ce
Jul 28 05:01:29.256944 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:29.256906    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1badbb/sub-test-model-1badbb-kserve-7c7b8b4955-sdxss" event={"ID":"ce9444de-e2bd-453b-abea-57de863a7455","Type":"ContainerStarted","Data":"b21e142b7a6323ade9e17d647414aa11e5cabcef64f7a3a74de8b25a2d8b7cf7"}
Jul 28 05:01:29.257117 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:29.256953    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1badbb/sub-test-model-1badbb-kserve-7c7b8b4955-sdxss" event={"ID":"ce9444de-e2bd-453b-abea-57de863a7455","Type":"ContainerStarted","Data":"e69505d19034dbe1a87097059ac1416c79735f3be294bf870832ee3ba21823ce"}
Jul 28 05:01:29.272725 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:29.272668    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-1badbb/sub-test-model-1badbb-kserve-7c7b8b4955-sdxss" podStartSLOduration=1.272652711 podStartE2EDuration="1.272652711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:01:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:01:29.272340905 +0000 UTC m=+2989.662516937" watchObservedRunningTime="2026-07-28 05:01:29.272652711 +0000 UTC m=+2989.662828733"
Jul 28 05:01:38.604583 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:38.604538    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-1badbb/sub-test-model-1badbb-kserve-7c7b8b4955-sdxss"
Jul 28 05:01:38.605153 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:38.604604    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-1badbb/sub-test-model-1badbb-kserve-7c7b8b4955-sdxss"
Jul 28 05:01:38.617507 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:38.617481    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-1badbb/sub-test-model-1badbb-kserve-7c7b8b4955-sdxss"
Jul 28 05:01:39.320388 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:39.320362    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-1badbb/sub-test-model-1badbb-kserve-7c7b8b4955-sdxss"
Jul 28 05:01:43.789510 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:43.789477    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4glpr_bfd2d3d8-f287-42d8-83c7-f82185c96735/console-operator/2.log"
Jul 28 05:01:43.797994 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:01:43.797967    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4glpr_bfd2d3d8-f287-42d8-83c7-f82185c96735/console-operator/2.log"
Jul 28 05:03:13.247688 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:03:13.247644    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1badbb/sub-test-model-1badbb-kserve-7c7b8b4955-sdxss"]
Jul 28 05:03:28.266900 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:03:28.266853    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-1badbb/sub-test-model-1badbb-kserve-7c7b8b4955-sdxss" podUID="ce9444de-e2bd-453b-abea-57de863a7455" containerName="main" containerID="cri-o://b21e142b7a6323ade9e17d647414aa11e5cabcef64f7a3a74de8b25a2d8b7cf7" gracePeriod=45
Jul 28 05:03:28.534051 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:03:28.534028    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1badbb/sub-test-model-1badbb-kserve-7c7b8b4955-sdxss"
Jul 28 05:03:28.574053 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:03:28.574027    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce9444de-e2bd-453b-abea-57de863a7455-model-cache\") pod \"ce9444de-e2bd-453b-abea-57de863a7455\" (UID: \"ce9444de-e2bd-453b-abea-57de863a7455\") "
Jul 28 05:03:28.574172 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:03:28.574074    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce9444de-e2bd-453b-abea-57de863a7455-dshm\") pod \"ce9444de-e2bd-453b-abea-57de863a7455\" (UID: \"ce9444de-e2bd-453b-abea-57de863a7455\") "
Jul 28 05:03:28.574172 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:03:28.574099    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5ls44\" (UniqueName: \"kubernetes.io/projected/ce9444de-e2bd-453b-abea-57de863a7455-kube-api-access-5ls44\") pod \"ce9444de-e2bd-453b-abea-57de863a7455\" (UID: \"ce9444de-e2bd-453b-abea-57de863a7455\") "
Jul 28 05:03:28.574172 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:03:28.574137    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce9444de-e2bd-453b-abea-57de863a7455-tmp-dir\") pod \"ce9444de-e2bd-453b-abea-57de863a7455\" (UID: \"ce9444de-e2bd-453b-abea-57de863a7455\") "
Jul 28 05:03:28.574344 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:03:28.574192    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce9444de-e2bd-453b-abea-57de863a7455-tls-certs\") pod \"ce9444de-e2bd-453b-abea-57de863a7455\" (UID: \"ce9444de-e2bd-453b-abea-57de863a7455\") "
Jul 28 05:03:28.574344 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:03:28.574291    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce9444de-e2bd-453b-abea-57de863a7455-home\") pod \"ce9444de-e2bd-453b-abea-57de863a7455\" (UID: \"ce9444de-e2bd-453b-abea-57de863a7455\") "
Jul 28 05:03:28.574344 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:03:28.574313    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce9444de-e2bd-453b-abea-57de863a7455-model-cache" (OuterVolumeSpecName: "model-cache") pod "ce9444de-e2bd-453b-abea-57de863a7455" (UID: "ce9444de-e2bd-453b-abea-57de863a7455"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 05:03:28.574514 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:03:28.574451    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce9444de-e2bd-453b-abea-57de863a7455-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ce9444de-e2bd-453b-abea-57de863a7455" (UID: "ce9444de-e2bd-453b-abea-57de863a7455"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 05:03:28.574676 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:03:28.574653    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce9444de-e2bd-453b-abea-57de863a7455-home" (OuterVolumeSpecName: "home") pod "ce9444de-e2bd-453b-abea-57de863a7455" (UID: "ce9444de-e2bd-453b-abea-57de863a7455"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 05:03:28.574783 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:03:28.574723    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce9444de-e2bd-453b-abea-57de863a7455-model-cache\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:03:28.574783 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:03:28.574741    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce9444de-e2bd-453b-abea-57de863a7455-tmp-dir\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:03:28.576745 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:03:28.576719    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce9444de-e2bd-453b-abea-57de863a7455-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ce9444de-e2bd-453b-abea-57de863a7455" (UID: "ce9444de-e2bd-453b-abea-57de863a7455"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 05:03:28.576871 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:03:28.576752    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce9444de-e2bd-453b-abea-57de863a7455-kube-api-access-5ls44" (OuterVolumeSpecName: "kube-api-access-5ls44") pod "ce9444de-e2bd-453b-abea-57de863a7455" (UID: "ce9444de-e2bd-453b-abea-57de863a7455"). InnerVolumeSpecName "kube-api-access-5ls44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 05:03:28.576871 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:03:28.576861    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce9444de-e2bd-453b-abea-57de863a7455-dshm" (OuterVolumeSpecName: "dshm") pod "ce9444de-e2bd-453b-abea-57de863a7455" (UID: "ce9444de-e2bd-453b-abea-57de863a7455"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 05:03:28.676251 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:03:28.676219    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce9444de-e2bd-453b-abea-57de863a7455-home\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:03:28.676251 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:03:28.676244    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce9444de-e2bd-453b-abea-57de863a7455-dshm\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:03:28.676251 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:03:28.676254    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ls44\" (UniqueName: \"kubernetes.io/projected/ce9444de-e2bd-453b-abea-57de863a7455-kube-api-access-5ls44\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:03:28.676478 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:03:28.676264    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce9444de-e2bd-453b-abea-57de863a7455-tls-certs\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:03:28.854527 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:03:28.854430    2566 generic.go:356] "Generic (PLEG): container finished" podID="ce9444de-e2bd-453b-abea-57de863a7455" containerID="b21e142b7a6323ade9e17d647414aa11e5cabcef64f7a3a74de8b25a2d8b7cf7" exitCode=0
Jul 28 05:03:28.854527 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:03:28.854472    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-1badbb/sub-test-model-1badbb-kserve-7c7b8b4955-sdxss"
Jul 28 05:03:28.854527 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:03:28.854486    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1badbb/sub-test-model-1badbb-kserve-7c7b8b4955-sdxss" event={"ID":"ce9444de-e2bd-453b-abea-57de863a7455","Type":"ContainerDied","Data":"b21e142b7a6323ade9e17d647414aa11e5cabcef64f7a3a74de8b25a2d8b7cf7"}
Jul 28 05:03:28.854527 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:03:28.854525    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-1badbb/sub-test-model-1badbb-kserve-7c7b8b4955-sdxss" event={"ID":"ce9444de-e2bd-453b-abea-57de863a7455","Type":"ContainerDied","Data":"e69505d19034dbe1a87097059ac1416c79735f3be294bf870832ee3ba21823ce"}
Jul 28 05:03:28.854905 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:03:28.854548    2566 scope.go:117] "RemoveContainer" containerID="b21e142b7a6323ade9e17d647414aa11e5cabcef64f7a3a74de8b25a2d8b7cf7"
Jul 28 05:03:28.863997 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:03:28.863980    2566 scope.go:117] "RemoveContainer" containerID="b21e142b7a6323ade9e17d647414aa11e5cabcef64f7a3a74de8b25a2d8b7cf7"
Jul 28 05:03:28.864217 ip-10-0-129-187 kubenswrapper[2566]: E0728 05:03:28.864201    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b21e142b7a6323ade9e17d647414aa11e5cabcef64f7a3a74de8b25a2d8b7cf7\": container with ID starting with b21e142b7a6323ade9e17d647414aa11e5cabcef64f7a3a74de8b25a2d8b7cf7 not found: ID does not exist" containerID="b21e142b7a6323ade9e17d647414aa11e5cabcef64f7a3a74de8b25a2d8b7cf7"
Jul 28 05:03:28.864258 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:03:28.864225    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b21e142b7a6323ade9e17d647414aa11e5cabcef64f7a3a74de8b25a2d8b7cf7"} err="failed to get container status \"b21e142b7a6323ade9e17d647414aa11e5cabcef64f7a3a74de8b25a2d8b7cf7\": rpc error: code = NotFound desc = could not find container \"b21e142b7a6323ade9e17d647414aa11e5cabcef64f7a3a74de8b25a2d8b7cf7\": container with ID starting with b21e142b7a6323ade9e17d647414aa11e5cabcef64f7a3a74de8b25a2d8b7cf7 not found: ID does not exist"
Jul 28 05:03:28.890976 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:03:28.890950    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-1badbb/sub-test-model-1badbb-kserve-7c7b8b4955-sdxss"]
Jul 28 05:03:28.897088 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:03:28.897067    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-1badbb/sub-test-model-1badbb-kserve-7c7b8b4955-sdxss"]
Jul 28 05:03:30.088973 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:03:30.088941    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce9444de-e2bd-453b-abea-57de863a7455" path="/var/lib/kubelet/pods/ce9444de-e2bd-453b-abea-57de863a7455/volumes"
Jul 28 05:05:00.284933 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:00.284896    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7574-140712837611568"]
Jul 28 05:05:00.285575 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:00.285552    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce9444de-e2bd-453b-abea-57de863a7455" containerName="main"
Jul 28 05:05:00.285661 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:00.285580    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce9444de-e2bd-453b-abea-57de863a7455" containerName="main"
Jul 28 05:05:00.285717 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:00.285691    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce9444de-e2bd-453b-abea-57de863a7455" containerName="main"
Jul 28 05:05:00.288974 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:00.288927    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140712837611568"
Jul 28 05:05:00.291583 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:00.291557    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-gmrc7\""
Jul 28 05:05:00.291831 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:00.291812    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7574-140712837611568"]
Jul 28 05:05:00.368581 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:00.368554    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7884\" (UniqueName: \"kubernetes.io/projected/3b29d23b-a84d-440b-9fa2-8aa658207e77-kube-api-access-j7884\") pod \"test-curl-7574-140712837611568\" (UID: \"3b29d23b-a84d-440b-9fa2-8aa658207e77\") " pod="opendatahub/test-curl-7574-140712837611568"
Jul 28 05:05:00.470097 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:00.470063    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7884\" (UniqueName: \"kubernetes.io/projected/3b29d23b-a84d-440b-9fa2-8aa658207e77-kube-api-access-j7884\") pod \"test-curl-7574-140712837611568\" (UID: \"3b29d23b-a84d-440b-9fa2-8aa658207e77\") " pod="opendatahub/test-curl-7574-140712837611568"
Jul 28 05:05:00.478683 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:00.478658    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7884\" (UniqueName: \"kubernetes.io/projected/3b29d23b-a84d-440b-9fa2-8aa658207e77-kube-api-access-j7884\") pod \"test-curl-7574-140712837611568\" (UID: \"3b29d23b-a84d-440b-9fa2-8aa658207e77\") " pod="opendatahub/test-curl-7574-140712837611568"
Jul 28 05:05:00.600181 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:00.600085    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140712837611568"
Jul 28 05:05:00.740550 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:00.740521    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7574-140712837611568"]
Jul 28 05:05:00.742472 ip-10-0-129-187 kubenswrapper[2566]: W0728 05:05:00.742446    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b29d23b_a84d_440b_9fa2_8aa658207e77.slice/crio-7e40abddf19d0ef988fe5f0e0022171129a2b82bee1107f31406a64dad4dc229 WatchSource:0}: Error finding container 7e40abddf19d0ef988fe5f0e0022171129a2b82bee1107f31406a64dad4dc229: Status 404 returned error can't find the container with id 7e40abddf19d0ef988fe5f0e0022171129a2b82bee1107f31406a64dad4dc229
Jul 28 05:05:00.743965 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:00.743946    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 05:05:01.322706 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:01.322667    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7574-140712837611568" event={"ID":"3b29d23b-a84d-440b-9fa2-8aa658207e77","Type":"ContainerStarted","Data":"7e40abddf19d0ef988fe5f0e0022171129a2b82bee1107f31406a64dad4dc229"}
Jul 28 05:05:02.328934 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:02.328898    2566 generic.go:356] "Generic (PLEG): container finished" podID="3b29d23b-a84d-440b-9fa2-8aa658207e77" containerID="362aa5c4be803a7beef6abfffeadcedf74df2d72abc8ab19113b828cc8efecad" exitCode=0
Jul 28 05:05:02.329325 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:02.328985    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7574-140712837611568" event={"ID":"3b29d23b-a84d-440b-9fa2-8aa658207e77","Type":"ContainerDied","Data":"362aa5c4be803a7beef6abfffeadcedf74df2d72abc8ab19113b828cc8efecad"}
Jul 28 05:05:03.474856 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:03.474813    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140712837611568"
Jul 28 05:05:03.507426 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:03.507396    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7574-140712837611568_3b29d23b-a84d-440b-9fa2-8aa658207e77/test-curl-7574-140712837611568/0.log"
Jul 28 05:05:03.539655 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:03.539626    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7574-140712837611568"]
Jul 28 05:05:03.543221 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:03.543192    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7574-140712837611568"]
Jul 28 05:05:03.606265 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:03.606190    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7884\" (UniqueName: \"kubernetes.io/projected/3b29d23b-a84d-440b-9fa2-8aa658207e77-kube-api-access-j7884\") pod \"3b29d23b-a84d-440b-9fa2-8aa658207e77\" (UID: \"3b29d23b-a84d-440b-9fa2-8aa658207e77\") "
Jul 28 05:05:03.608196 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:03.608174    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b29d23b-a84d-440b-9fa2-8aa658207e77-kube-api-access-j7884" (OuterVolumeSpecName: "kube-api-access-j7884") pod "3b29d23b-a84d-440b-9fa2-8aa658207e77" (UID: "3b29d23b-a84d-440b-9fa2-8aa658207e77"). InnerVolumeSpecName "kube-api-access-j7884". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 05:05:03.707783 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:03.707748    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7884\" (UniqueName: \"kubernetes.io/projected/3b29d23b-a84d-440b-9fa2-8aa658207e77-kube-api-access-j7884\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:05:03.712780 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:03.712753    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7574-140712837499184"]
Jul 28 05:05:03.713175 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:03.713161    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b29d23b-a84d-440b-9fa2-8aa658207e77" containerName="test-curl-7574-140712837611568"
Jul 28 05:05:03.713231 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:03.713176    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b29d23b-a84d-440b-9fa2-8aa658207e77" containerName="test-curl-7574-140712837611568"
Jul 28 05:05:03.713265 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:03.713258    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b29d23b-a84d-440b-9fa2-8aa658207e77" containerName="test-curl-7574-140712837611568"
Jul 28 05:05:03.716352 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:03.716329    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140712837499184"
Jul 28 05:05:03.720102 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:03.720082    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7574-140712837499184"]
Jul 28 05:05:03.808764 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:03.808726    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7rvs\" (UniqueName: \"kubernetes.io/projected/02be9e24-dd1e-4ffd-9e7e-4df02a74612b-kube-api-access-x7rvs\") pod \"test-curl-7574-140712837499184\" (UID: \"02be9e24-dd1e-4ffd-9e7e-4df02a74612b\") " pod="opendatahub/test-curl-7574-140712837499184"
Jul 28 05:05:03.910003 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:03.909912    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7rvs\" (UniqueName: \"kubernetes.io/projected/02be9e24-dd1e-4ffd-9e7e-4df02a74612b-kube-api-access-x7rvs\") pod \"test-curl-7574-140712837499184\" (UID: \"02be9e24-dd1e-4ffd-9e7e-4df02a74612b\") " pod="opendatahub/test-curl-7574-140712837499184"
Jul 28 05:05:03.917866 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:03.917833    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7rvs\" (UniqueName: \"kubernetes.io/projected/02be9e24-dd1e-4ffd-9e7e-4df02a74612b-kube-api-access-x7rvs\") pod \"test-curl-7574-140712837499184\" (UID: \"02be9e24-dd1e-4ffd-9e7e-4df02a74612b\") " pod="opendatahub/test-curl-7574-140712837499184"
Jul 28 05:05:04.027028 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:04.026996    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140712837499184"
Jul 28 05:05:04.091755 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:04.091728    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b29d23b-a84d-440b-9fa2-8aa658207e77" path="/var/lib/kubelet/pods/3b29d23b-a84d-440b-9fa2-8aa658207e77/volumes"
Jul 28 05:05:04.161105 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:04.161082    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7574-140712837499184"]
Jul 28 05:05:04.163260 ip-10-0-129-187 kubenswrapper[2566]: W0728 05:05:04.163229    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02be9e24_dd1e_4ffd_9e7e_4df02a74612b.slice/crio-3eafc1a9256717363ab8bdbef0a1a61b09bd1b1c93e3161108e0648177c9426d WatchSource:0}: Error finding container 3eafc1a9256717363ab8bdbef0a1a61b09bd1b1c93e3161108e0648177c9426d: Status 404 returned error can't find the container with id 3eafc1a9256717363ab8bdbef0a1a61b09bd1b1c93e3161108e0648177c9426d
Jul 28 05:05:04.340399 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:04.340364    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140712837611568"
Jul 28 05:05:04.340590 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:04.340401    2566 scope.go:117] "RemoveContainer" containerID="362aa5c4be803a7beef6abfffeadcedf74df2d72abc8ab19113b828cc8efecad"
Jul 28 05:05:04.341726 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:04.341545    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7574-140712837499184" event={"ID":"02be9e24-dd1e-4ffd-9e7e-4df02a74612b","Type":"ContainerStarted","Data":"3eafc1a9256717363ab8bdbef0a1a61b09bd1b1c93e3161108e0648177c9426d"}
Jul 28 05:05:05.347964 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:05.347929    2566 generic.go:356] "Generic (PLEG): container finished" podID="02be9e24-dd1e-4ffd-9e7e-4df02a74612b" containerID="c5210c03688decfa1ce724436a277fcdcd16790bfeea8e8c8e5b61fef3679b5c" exitCode=0
Jul 28 05:05:05.348429 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:05.348019    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7574-140712837499184" event={"ID":"02be9e24-dd1e-4ffd-9e7e-4df02a74612b","Type":"ContainerDied","Data":"c5210c03688decfa1ce724436a277fcdcd16790bfeea8e8c8e5b61fef3679b5c"}
Jul 28 05:05:06.491601 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:06.491574    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140712837499184"
Jul 28 05:05:06.511049 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:06.510995    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7574-140712837499184_02be9e24-dd1e-4ffd-9e7e-4df02a74612b/test-curl-7574-140712837499184/0.log"
Jul 28 05:05:06.536033 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:06.536003    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7574-140712837499184"]
Jul 28 05:05:06.539301 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:06.539278    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7574-140712837499184"]
Jul 28 05:05:06.639892 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:06.639783    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7rvs\" (UniqueName: \"kubernetes.io/projected/02be9e24-dd1e-4ffd-9e7e-4df02a74612b-kube-api-access-x7rvs\") pod \"02be9e24-dd1e-4ffd-9e7e-4df02a74612b\" (UID: \"02be9e24-dd1e-4ffd-9e7e-4df02a74612b\") "
Jul 28 05:05:06.641970 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:06.641940    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02be9e24-dd1e-4ffd-9e7e-4df02a74612b-kube-api-access-x7rvs" (OuterVolumeSpecName: "kube-api-access-x7rvs") pod "02be9e24-dd1e-4ffd-9e7e-4df02a74612b" (UID: "02be9e24-dd1e-4ffd-9e7e-4df02a74612b"). InnerVolumeSpecName "kube-api-access-x7rvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 05:05:06.740611 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:06.740567    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7rvs\" (UniqueName: \"kubernetes.io/projected/02be9e24-dd1e-4ffd-9e7e-4df02a74612b-kube-api-access-x7rvs\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:05:07.360635 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:07.360608    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140712837499184"
Jul 28 05:05:07.360803 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:07.360646    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3eafc1a9256717363ab8bdbef0a1a61b09bd1b1c93e3161108e0648177c9426d"
Jul 28 05:05:07.371433 ip-10-0-129-187 kubenswrapper[2566]: E0728 05:05:07.371406    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7574-140712837499184\" is forbidden: User \"system:node:ip-10-0-129-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-187.ec2.internal' and this object" podUID="02be9e24-dd1e-4ffd-9e7e-4df02a74612b" pod="opendatahub/test-curl-7574-140712837499184"
Jul 28 05:05:08.088571 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:08.088538    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02be9e24-dd1e-4ffd-9e7e-4df02a74612b" path="/var/lib/kubelet/pods/02be9e24-dd1e-4ffd-9e7e-4df02a74612b/volumes"
Jul 28 05:05:12.348287 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:12.348257    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7574-140712837500768"]
Jul 28 05:05:12.348722 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:12.348708    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02be9e24-dd1e-4ffd-9e7e-4df02a74612b" containerName="test-curl-7574-140712837499184"
Jul 28 05:05:12.348722 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:12.348723    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="02be9e24-dd1e-4ffd-9e7e-4df02a74612b" containerName="test-curl-7574-140712837499184"
Jul 28 05:05:12.348801 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:12.348793    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="02be9e24-dd1e-4ffd-9e7e-4df02a74612b" containerName="test-curl-7574-140712837499184"
Jul 28 05:05:12.351899 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:12.351879    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140712837500768"
Jul 28 05:05:12.354853 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:12.354812    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-gmrc7\""
Jul 28 05:05:12.357765 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:12.357743    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7574-140712837500768"]
Jul 28 05:05:12.391370 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:12.391347    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cpk6\" (UniqueName: \"kubernetes.io/projected/2cb530ec-d282-4b57-b385-d1c25d500176-kube-api-access-7cpk6\") pod \"test-curl-7574-140712837500768\" (UID: \"2cb530ec-d282-4b57-b385-d1c25d500176\") " pod="opendatahub/test-curl-7574-140712837500768"
Jul 28 05:05:12.492163 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:12.492129    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cpk6\" (UniqueName: \"kubernetes.io/projected/2cb530ec-d282-4b57-b385-d1c25d500176-kube-api-access-7cpk6\") pod \"test-curl-7574-140712837500768\" (UID: \"2cb530ec-d282-4b57-b385-d1c25d500176\") " pod="opendatahub/test-curl-7574-140712837500768"
Jul 28 05:05:12.500227 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:12.500200    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cpk6\" (UniqueName: \"kubernetes.io/projected/2cb530ec-d282-4b57-b385-d1c25d500176-kube-api-access-7cpk6\") pod \"test-curl-7574-140712837500768\" (UID: \"2cb530ec-d282-4b57-b385-d1c25d500176\") " pod="opendatahub/test-curl-7574-140712837500768"
Jul 28 05:05:12.663533 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:12.663448    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140712837500768"
Jul 28 05:05:13.004251 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:13.004226    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7574-140712837500768"]
Jul 28 05:05:13.006555 ip-10-0-129-187 kubenswrapper[2566]: W0728 05:05:13.006529    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2cb530ec_d282_4b57_b385_d1c25d500176.slice/crio-14d945883190ba9b4c6c6a1d3a6433b12d20c29b1c3fcc0b5e7326002754f949 WatchSource:0}: Error finding container 14d945883190ba9b4c6c6a1d3a6433b12d20c29b1c3fcc0b5e7326002754f949: Status 404 returned error can't find the container with id 14d945883190ba9b4c6c6a1d3a6433b12d20c29b1c3fcc0b5e7326002754f949
Jul 28 05:05:13.392369 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:13.392342    2566 generic.go:356] "Generic (PLEG): container finished" podID="2cb530ec-d282-4b57-b385-d1c25d500176" containerID="9b80f021b7db7d9705d92f631685d69dec7a73d2cb2e30276e1d0da19b2c6408" exitCode=0
Jul 28 05:05:13.392673 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:13.392429    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7574-140712837500768" event={"ID":"2cb530ec-d282-4b57-b385-d1c25d500176","Type":"ContainerDied","Data":"9b80f021b7db7d9705d92f631685d69dec7a73d2cb2e30276e1d0da19b2c6408"}
Jul 28 05:05:13.392673 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:13.392470    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7574-140712837500768" event={"ID":"2cb530ec-d282-4b57-b385-d1c25d500176","Type":"ContainerStarted","Data":"14d945883190ba9b4c6c6a1d3a6433b12d20c29b1c3fcc0b5e7326002754f949"}
Jul 28 05:05:14.538157 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:14.538131    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140712837500768"
Jul 28 05:05:14.556521 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:14.556497    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7574-140712837500768_2cb530ec-d282-4b57-b385-d1c25d500176/test-curl-7574-140712837500768/0.log"
Jul 28 05:05:14.584408 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:14.584375    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7574-140712837500768"]
Jul 28 05:05:14.586243 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:14.586222    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7574-140712837500768"]
Jul 28 05:05:14.611519 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:14.611498    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7cpk6\" (UniqueName: \"kubernetes.io/projected/2cb530ec-d282-4b57-b385-d1c25d500176-kube-api-access-7cpk6\") pod \"2cb530ec-d282-4b57-b385-d1c25d500176\" (UID: \"2cb530ec-d282-4b57-b385-d1c25d500176\") "
Jul 28 05:05:14.613542 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:14.613513    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cb530ec-d282-4b57-b385-d1c25d500176-kube-api-access-7cpk6" (OuterVolumeSpecName: "kube-api-access-7cpk6") pod "2cb530ec-d282-4b57-b385-d1c25d500176" (UID: "2cb530ec-d282-4b57-b385-d1c25d500176"). InnerVolumeSpecName "kube-api-access-7cpk6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 05:05:14.712604 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:14.712529    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cpk6\" (UniqueName: \"kubernetes.io/projected/2cb530ec-d282-4b57-b385-d1c25d500176-kube-api-access-7cpk6\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:05:15.403505 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:15.403472    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7574-140712837500768"
Jul 28 05:05:15.403679 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:15.403509    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14d945883190ba9b4c6c6a1d3a6433b12d20c29b1c3fcc0b5e7326002754f949"
Jul 28 05:05:15.413887 ip-10-0-129-187 kubenswrapper[2566]: E0728 05:05:15.413859    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7574-140712837500768\" is forbidden: User \"system:node:ip-10-0-129-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-187.ec2.internal' and this object" podUID="2cb530ec-d282-4b57-b385-d1c25d500176" pod="opendatahub/test-curl-7574-140712837500768"
Jul 28 05:05:15.417137 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:15.417113    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx"]
Jul 28 05:05:15.417527 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:15.417513    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cb530ec-d282-4b57-b385-d1c25d500176" containerName="test-curl-7574-140712837500768"
Jul 28 05:05:15.417575 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:15.417528    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cb530ec-d282-4b57-b385-d1c25d500176" containerName="test-curl-7574-140712837500768"
Jul 28 05:05:15.417611 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:15.417584    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cb530ec-d282-4b57-b385-d1c25d500176" containerName="test-curl-7574-140712837500768"
Jul 28 05:05:15.427229 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:15.427207    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx"
Jul 28 05:05:15.430159 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:15.430139    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-dc1efb-openshift-default-dockercfg-gp9n4\""
Jul 28 05:05:15.433291 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:15.433271    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx"]
Jul 28 05:05:15.447556 ip-10-0-129-187 kubenswrapper[2566]: E0728 05:05:15.447526    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7574-140712837500768\" is forbidden: User \"system:node:ip-10-0-129-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-187.ec2.internal' and this object" podUID="2cb530ec-d282-4b57-b385-d1c25d500176" pod="opendatahub/test-curl-7574-140712837500768"
Jul 28 05:05:15.520313 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:15.520273    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e74aefee-4f89-462e-89b9-f98c6abbe624-istiod-ca-cert\") pod \"e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx\" (UID: \"e74aefee-4f89-462e-89b9-f98c6abbe624\") " pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx"
Jul 28 05:05:15.520460 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:15.520398    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n7ls\" (UniqueName: \"kubernetes.io/projected/e74aefee-4f89-462e-89b9-f98c6abbe624-kube-api-access-8n7ls\") pod \"e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx\" (UID: \"e74aefee-4f89-462e-89b9-f98c6abbe624\") " pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx"
Jul 28 05:05:15.520460 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:15.520434    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e74aefee-4f89-462e-89b9-f98c6abbe624-istio-podinfo\") pod \"e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx\" (UID: \"e74aefee-4f89-462e-89b9-f98c6abbe624\") " pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx"
Jul 28 05:05:15.520547 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:15.520490    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e74aefee-4f89-462e-89b9-f98c6abbe624-istio-envoy\") pod \"e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx\" (UID: \"e74aefee-4f89-462e-89b9-f98c6abbe624\") " pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx"
Jul 28 05:05:15.520547 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:15.520510    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e74aefee-4f89-462e-89b9-f98c6abbe624-istio-token\") pod \"e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx\" (UID: \"e74aefee-4f89-462e-89b9-f98c6abbe624\") " pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx"
Jul 28 05:05:15.520547 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:15.520538    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e74aefee-4f89-462e-89b9-f98c6abbe624-istio-data\") pod \"e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx\" (UID: \"e74aefee-4f89-462e-89b9-f98c6abbe624\") " pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx"
Jul 28 05:05:15.520661 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:15.520566    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74aefee-4f89-462e-89b9-f98c6abbe624-workload-socket\") pod \"e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx\" (UID: \"e74aefee-4f89-462e-89b9-f98c6abbe624\") " pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx"
Jul 28 05:05:15.520661 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:15.520600    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e74aefee-4f89-462e-89b9-f98c6abbe624-workload-certs\") pod \"e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx\" (UID: \"e74aefee-4f89-462e-89b9-f98c6abbe624\") " pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx"
Jul 28 05:05:15.520661 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:15.520623    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74aefee-4f89-462e-89b9-f98c6abbe624-credential-socket\") pod \"e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx\" (UID: \"e74aefee-4f89-462e-89b9-f98c6abbe624\") " pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx"
Jul 28 05:05:15.622220 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:15.622183    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e74aefee-4f89-462e-89b9-f98c6abbe624-istiod-ca-cert\") pod \"e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx\" (UID: \"e74aefee-4f89-462e-89b9-f98c6abbe624\") " pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx"
Jul 28 05:05:15.622689 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:15.622277    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n7ls\" (UniqueName: \"kubernetes.io/projected/e74aefee-4f89-462e-89b9-f98c6abbe624-kube-api-access-8n7ls\") pod \"e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx\" (UID: \"e74aefee-4f89-462e-89b9-f98c6abbe624\") " pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx"
Jul 28 05:05:15.622689 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:15.622308    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e74aefee-4f89-462e-89b9-f98c6abbe624-istio-podinfo\") pod \"e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx\" (UID: \"e74aefee-4f89-462e-89b9-f98c6abbe624\") " pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx"
Jul 28 05:05:15.622689 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:15.622366    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e74aefee-4f89-462e-89b9-f98c6abbe624-istio-envoy\") pod \"e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx\" (UID: \"e74aefee-4f89-462e-89b9-f98c6abbe624\") " pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx"
Jul 28 05:05:15.622689 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:15.622390    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e74aefee-4f89-462e-89b9-f98c6abbe624-istio-token\") pod \"e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx\" (UID: \"e74aefee-4f89-462e-89b9-f98c6abbe624\") " pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx"
Jul 28 05:05:15.622689 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:15.622426    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e74aefee-4f89-462e-89b9-f98c6abbe624-istio-data\") pod \"e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx\" (UID: \"e74aefee-4f89-462e-89b9-f98c6abbe624\") " pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx"
Jul 28 05:05:15.622689 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:15.622461    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74aefee-4f89-462e-89b9-f98c6abbe624-workload-socket\") pod \"e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx\" (UID: \"e74aefee-4f89-462e-89b9-f98c6abbe624\") " pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx"
Jul 28 05:05:15.622689 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:15.622492    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e74aefee-4f89-462e-89b9-f98c6abbe624-workload-certs\") pod \"e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx\" (UID: \"e74aefee-4f89-462e-89b9-f98c6abbe624\") " pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx"
Jul 28 05:05:15.622689 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:15.622604    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74aefee-4f89-462e-89b9-f98c6abbe624-credential-socket\") pod \"e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx\" (UID: \"e74aefee-4f89-462e-89b9-f98c6abbe624\") " pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx"
Jul 28 05:05:15.623142 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:15.622932    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74aefee-4f89-462e-89b9-f98c6abbe624-workload-socket\") pod \"e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx\" (UID: \"e74aefee-4f89-462e-89b9-f98c6abbe624\") " pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx"
Jul 28 05:05:15.623142 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:15.623016    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74aefee-4f89-462e-89b9-f98c6abbe624-credential-socket\") pod \"e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx\" (UID: \"e74aefee-4f89-462e-89b9-f98c6abbe624\") " pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx"
Jul 28 05:05:15.623142 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:15.623074    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e74aefee-4f89-462e-89b9-f98c6abbe624-istio-data\") pod \"e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx\" (UID: \"e74aefee-4f89-462e-89b9-f98c6abbe624\") " pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx"
Jul 28 05:05:15.623295 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:15.623185    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e74aefee-4f89-462e-89b9-f98c6abbe624-istiod-ca-cert\") pod \"e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx\" (UID: \"e74aefee-4f89-462e-89b9-f98c6abbe624\") " pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx"
Jul 28 05:05:15.623295 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:15.623244    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e74aefee-4f89-462e-89b9-f98c6abbe624-workload-certs\") pod \"e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx\" (UID: \"e74aefee-4f89-462e-89b9-f98c6abbe624\") " pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx"
Jul 28 05:05:15.625007 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:15.624985    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e74aefee-4f89-462e-89b9-f98c6abbe624-istio-envoy\") pod \"e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx\" (UID: \"e74aefee-4f89-462e-89b9-f98c6abbe624\") " pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx"
Jul 28 05:05:15.625125 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:15.625048    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e74aefee-4f89-462e-89b9-f98c6abbe624-istio-podinfo\") pod \"e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx\" (UID: \"e74aefee-4f89-462e-89b9-f98c6abbe624\") " pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx"
Jul 28 05:05:15.630077 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:15.630053    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e74aefee-4f89-462e-89b9-f98c6abbe624-istio-token\") pod \"e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx\" (UID: \"e74aefee-4f89-462e-89b9-f98c6abbe624\") " pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx"
Jul 28 05:05:15.630303 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:15.630282    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n7ls\" (UniqueName: \"kubernetes.io/projected/e74aefee-4f89-462e-89b9-f98c6abbe624-kube-api-access-8n7ls\") pod \"e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx\" (UID: \"e74aefee-4f89-462e-89b9-f98c6abbe624\") " pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx"
Jul 28 05:05:15.741499 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:15.741399    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx"
Jul 28 05:05:15.885990 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:15.885970    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx"]
Jul 28 05:05:15.888172 ip-10-0-129-187 kubenswrapper[2566]: W0728 05:05:15.888149    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode74aefee_4f89_462e_89b9_f98c6abbe624.slice/crio-43f2946cc0fa1149921574b1faeec583c3c11f12ec47b07e87680ce310869b69 WatchSource:0}: Error finding container 43f2946cc0fa1149921574b1faeec583c3c11f12ec47b07e87680ce310869b69: Status 404 returned error can't find the container with id 43f2946cc0fa1149921574b1faeec583c3c11f12ec47b07e87680ce310869b69
Jul 28 05:05:15.890464 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:15.890427    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 05:05:15.890579 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:15.890500    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 05:05:15.890579 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:15.890534    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 05:05:16.089849 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:16.089805    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cb530ec-d282-4b57-b385-d1c25d500176" path="/var/lib/kubelet/pods/2cb530ec-d282-4b57-b385-d1c25d500176/volumes"
Jul 28 05:05:16.410378 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:16.410291    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx" event={"ID":"e74aefee-4f89-462e-89b9-f98c6abbe624","Type":"ContainerStarted","Data":"158eace5807a18d93358a39a7c0d9f24c0e7ba4d3033fc8b3ef080a9b039b248"}
Jul 28 05:05:16.410378 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:16.410354    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx" event={"ID":"e74aefee-4f89-462e-89b9-f98c6abbe624","Type":"ContainerStarted","Data":"43f2946cc0fa1149921574b1faeec583c3c11f12ec47b07e87680ce310869b69"}
Jul 28 05:05:16.430268 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:16.430227    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx" podStartSLOduration=1.430211503 podStartE2EDuration="1.430211503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:05:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:05:16.42931677 +0000 UTC m=+3216.819492793" watchObservedRunningTime="2026-07-28 05:05:16.430211503 +0000 UTC m=+3216.820387522"
Jul 28 05:05:16.742336 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:16.742250    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx"
Jul 28 05:05:16.747458 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:16.747436    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx"
Jul 28 05:05:17.415717 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:17.415681    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx"
Jul 28 05:05:17.416645 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:17.416625    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx"
Jul 28 05:05:19.257932 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:19.257901    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dc1efb-6bb5d85cdd-w7l8z"]
Jul 28 05:05:19.262622 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:19.262604    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dc1efb-6bb5d85cdd-w7l8z"
Jul 28 05:05:19.265292 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:19.265272    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-dc1efb\""
Jul 28 05:05:19.265388 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:19.265367    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-dc1efb-dockercfg-7tcpk\""
Jul 28 05:05:19.269215 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:19.269186    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dc1efb-6bb5d85cdd-w7l8z"]
Jul 28 05:05:19.358883 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:19.358630    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dt5p\" (UniqueName: \"kubernetes.io/projected/e702cc51-e3a8-4c8e-84b7-d2774888e09c-kube-api-access-4dt5p\") pod \"payload-processing-e2e-ipp-a-dc1efb-6bb5d85cdd-w7l8z\" (UID: \"e702cc51-e3a8-4c8e-84b7-d2774888e09c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dc1efb-6bb5d85cdd-w7l8z"
Jul 28 05:05:19.358883 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:19.358783    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e702cc51-e3a8-4c8e-84b7-d2774888e09c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-dc1efb-6bb5d85cdd-w7l8z\" (UID: \"e702cc51-e3a8-4c8e-84b7-d2774888e09c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dc1efb-6bb5d85cdd-w7l8z"
Jul 28 05:05:19.460432 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:19.460400    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dt5p\" (UniqueName: \"kubernetes.io/projected/e702cc51-e3a8-4c8e-84b7-d2774888e09c-kube-api-access-4dt5p\") pod \"payload-processing-e2e-ipp-a-dc1efb-6bb5d85cdd-w7l8z\" (UID: \"e702cc51-e3a8-4c8e-84b7-d2774888e09c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dc1efb-6bb5d85cdd-w7l8z"
Jul 28 05:05:19.460616 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:19.460517    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e702cc51-e3a8-4c8e-84b7-d2774888e09c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-dc1efb-6bb5d85cdd-w7l8z\" (UID: \"e702cc51-e3a8-4c8e-84b7-d2774888e09c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dc1efb-6bb5d85cdd-w7l8z"
Jul 28 05:05:19.461444 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:19.461416    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e702cc51-e3a8-4c8e-84b7-d2774888e09c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-dc1efb-6bb5d85cdd-w7l8z\" (UID: \"e702cc51-e3a8-4c8e-84b7-d2774888e09c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dc1efb-6bb5d85cdd-w7l8z"
Jul 28 05:05:19.468963 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:19.468943    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dt5p\" (UniqueName: \"kubernetes.io/projected/e702cc51-e3a8-4c8e-84b7-d2774888e09c-kube-api-access-4dt5p\") pod \"payload-processing-e2e-ipp-a-dc1efb-6bb5d85cdd-w7l8z\" (UID: \"e702cc51-e3a8-4c8e-84b7-d2774888e09c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dc1efb-6bb5d85cdd-w7l8z"
Jul 28 05:05:19.576601 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:19.576569    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dc1efb-6bb5d85cdd-w7l8z"
Jul 28 05:05:19.715341 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:19.715314    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dc1efb-6bb5d85cdd-w7l8z"]
Jul 28 05:05:19.717123 ip-10-0-129-187 kubenswrapper[2566]: W0728 05:05:19.717088    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode702cc51_e3a8_4c8e_84b7_d2774888e09c.slice/crio-fd5d6641065cb06aee1f6f595e9037c57ee12227ba4d8fa1309096733d9c997e WatchSource:0}: Error finding container fd5d6641065cb06aee1f6f595e9037c57ee12227ba4d8fa1309096733d9c997e: Status 404 returned error can't find the container with id fd5d6641065cb06aee1f6f595e9037c57ee12227ba4d8fa1309096733d9c997e
Jul 28 05:05:20.437322 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:20.437291    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dc1efb-6bb5d85cdd-w7l8z" event={"ID":"e702cc51-e3a8-4c8e-84b7-d2774888e09c","Type":"ContainerStarted","Data":"a6737dab7437ff2ae11d755e4638e64bc19c24eca611be883b2b820f28715596"}
Jul 28 05:05:20.437792 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:20.437347    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dc1efb-6bb5d85cdd-w7l8z" event={"ID":"e702cc51-e3a8-4c8e-84b7-d2774888e09c","Type":"ContainerStarted","Data":"fd5d6641065cb06aee1f6f595e9037c57ee12227ba4d8fa1309096733d9c997e"}
Jul 28 05:05:20.437792 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:20.437370    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-dc1efb-6bb5d85cdd-w7l8z"
Jul 28 05:05:20.455703 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:20.455642    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-dc1efb-6bb5d85cdd-w7l8z" podStartSLOduration=1.455622518 podStartE2EDuration="1.455622518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:05:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:05:20.452175572 +0000 UTC m=+3220.842351595" watchObservedRunningTime="2026-07-28 05:05:20.455622518 +0000 UTC m=+3220.845798545"
Jul 28 05:05:31.446574 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:31.446542    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-dc1efb-6bb5d85cdd-w7l8z"
Jul 28 05:05:36.525490 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:36.525455    2566 generic.go:356] "Generic (PLEG): container finished" podID="e74aefee-4f89-462e-89b9-f98c6abbe624" containerID="158eace5807a18d93358a39a7c0d9f24c0e7ba4d3033fc8b3ef080a9b039b248" exitCode=0
Jul 28 05:05:36.525924 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:36.525526    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx" event={"ID":"e74aefee-4f89-462e-89b9-f98c6abbe624","Type":"ContainerDied","Data":"158eace5807a18d93358a39a7c0d9f24c0e7ba4d3033fc8b3ef080a9b039b248"}
Jul 28 05:05:36.525970 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:36.525958    2566 scope.go:117] "RemoveContainer" containerID="158eace5807a18d93358a39a7c0d9f24c0e7ba4d3033fc8b3ef080a9b039b248"
Jul 28 05:05:36.527209 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:36.527181    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 05:05:36.527287 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:36.527255    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 05:05:36.527325 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:36.527299    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 05:05:37.535689 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:37.535644    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx" event={"ID":"e74aefee-4f89-462e-89b9-f98c6abbe624","Type":"ContainerStarted","Data":"917ef30d9857eca03e3b3736517e14a6b07b3a4f8bc611684e7afca8132530d6"}
Jul 28 05:05:37.742539 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:37.742498    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx"
Jul 28 05:05:37.742745 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:37.742699    2566 patch_prober.go:28] interesting pod/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Jul 28 05:05:37.742831 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:37.742755    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx" podUID="e74aefee-4f89-462e-89b9-f98c6abbe624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Jul 28 05:05:38.243403 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:38.243367    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9b749c-78d7987948-4cz5k"]
Jul 28 05:05:38.247106 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:38.247089    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b749c-78d7987948-4cz5k"
Jul 28 05:05:38.249961 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:38.249934    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-9b749c\""
Jul 28 05:05:38.250101 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:38.249934    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-9b749c-dockercfg-wkcpw\""
Jul 28 05:05:38.255729 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:38.255703    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9b749c-78d7987948-4cz5k"]
Jul 28 05:05:38.326892 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:38.326830    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9b749c-5fc6885d68-j77b9"]
Jul 28 05:05:38.339755 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:38.338825    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9b749c-5fc6885d68-j77b9"
Jul 28 05:05:38.339755 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:38.339703    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9b749c-5fc6885d68-j77b9"]
Jul 28 05:05:38.352767 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:38.352733    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0c1aa7f-20cb-457c-84d6-17f1ba08e019-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9b749c-78d7987948-4cz5k\" (UID: \"d0c1aa7f-20cb-457c-84d6-17f1ba08e019\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b749c-78d7987948-4cz5k"
Jul 28 05:05:38.353192 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:38.353156    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcs4r\" (UniqueName: \"kubernetes.io/projected/d0c1aa7f-20cb-457c-84d6-17f1ba08e019-kube-api-access-kcs4r\") pod \"payload-pre-processing-e2e-ipp-b-9b749c-78d7987948-4cz5k\" (UID: \"d0c1aa7f-20cb-457c-84d6-17f1ba08e019\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b749c-78d7987948-4cz5k"
Jul 28 05:05:38.454173 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:38.454139    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcs4r\" (UniqueName: \"kubernetes.io/projected/d0c1aa7f-20cb-457c-84d6-17f1ba08e019-kube-api-access-kcs4r\") pod \"payload-pre-processing-e2e-ipp-b-9b749c-78d7987948-4cz5k\" (UID: \"d0c1aa7f-20cb-457c-84d6-17f1ba08e019\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b749c-78d7987948-4cz5k"
Jul 28 05:05:38.454380 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:38.454313    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pz6s\" (UniqueName: \"kubernetes.io/projected/725cf087-0eeb-43a8-80a0-a1fa3d8a011a-kube-api-access-6pz6s\") pod \"payload-processing-e2e-ipp-b-9b749c-5fc6885d68-j77b9\" (UID: \"725cf087-0eeb-43a8-80a0-a1fa3d8a011a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9b749c-5fc6885d68-j77b9"
Jul 28 05:05:38.454446 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:38.454386    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0c1aa7f-20cb-457c-84d6-17f1ba08e019-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9b749c-78d7987948-4cz5k\" (UID: \"d0c1aa7f-20cb-457c-84d6-17f1ba08e019\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b749c-78d7987948-4cz5k"
Jul 28 05:05:38.454446 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:38.454410    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/725cf087-0eeb-43a8-80a0-a1fa3d8a011a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9b749c-5fc6885d68-j77b9\" (UID: \"725cf087-0eeb-43a8-80a0-a1fa3d8a011a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9b749c-5fc6885d68-j77b9"
Jul 28 05:05:38.455099 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:38.455081    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0c1aa7f-20cb-457c-84d6-17f1ba08e019-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9b749c-78d7987948-4cz5k\" (UID: \"d0c1aa7f-20cb-457c-84d6-17f1ba08e019\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b749c-78d7987948-4cz5k"
Jul 28 05:05:38.462015 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:38.461990    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcs4r\" (UniqueName: \"kubernetes.io/projected/d0c1aa7f-20cb-457c-84d6-17f1ba08e019-kube-api-access-kcs4r\") pod \"payload-pre-processing-e2e-ipp-b-9b749c-78d7987948-4cz5k\" (UID: \"d0c1aa7f-20cb-457c-84d6-17f1ba08e019\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b749c-78d7987948-4cz5k"
Jul 28 05:05:38.555150 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:38.555114    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pz6s\" (UniqueName: \"kubernetes.io/projected/725cf087-0eeb-43a8-80a0-a1fa3d8a011a-kube-api-access-6pz6s\") pod \"payload-processing-e2e-ipp-b-9b749c-5fc6885d68-j77b9\" (UID: \"725cf087-0eeb-43a8-80a0-a1fa3d8a011a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9b749c-5fc6885d68-j77b9"
Jul 28 05:05:38.555546 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:38.555223    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/725cf087-0eeb-43a8-80a0-a1fa3d8a011a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9b749c-5fc6885d68-j77b9\" (UID: \"725cf087-0eeb-43a8-80a0-a1fa3d8a011a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9b749c-5fc6885d68-j77b9"
Jul 28 05:05:38.555996 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:38.555978    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/725cf087-0eeb-43a8-80a0-a1fa3d8a011a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9b749c-5fc6885d68-j77b9\" (UID: \"725cf087-0eeb-43a8-80a0-a1fa3d8a011a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9b749c-5fc6885d68-j77b9"
Jul 28 05:05:38.558455 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:38.558438    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b749c-78d7987948-4cz5k"
Jul 28 05:05:38.563444 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:38.563421    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pz6s\" (UniqueName: \"kubernetes.io/projected/725cf087-0eeb-43a8-80a0-a1fa3d8a011a-kube-api-access-6pz6s\") pod \"payload-processing-e2e-ipp-b-9b749c-5fc6885d68-j77b9\" (UID: \"725cf087-0eeb-43a8-80a0-a1fa3d8a011a\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9b749c-5fc6885d68-j77b9"
Jul 28 05:05:38.657460 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:38.657356    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9b749c-5fc6885d68-j77b9"
Jul 28 05:05:38.742306 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:38.742264    2566 patch_prober.go:28] interesting pod/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Jul 28 05:05:38.742506 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:38.742330    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx" podUID="e74aefee-4f89-462e-89b9-f98c6abbe624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Jul 28 05:05:38.756297 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:38.755896    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9b749c-78d7987948-4cz5k"]
Jul 28 05:05:38.761615 ip-10-0-129-187 kubenswrapper[2566]: W0728 05:05:38.761557    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0c1aa7f_20cb_457c_84d6_17f1ba08e019.slice/crio-a7d1105dddb65dbfa4b9247466579cdefc60b766a0304bfa22ef12b163926885 WatchSource:0}: Error finding container a7d1105dddb65dbfa4b9247466579cdefc60b766a0304bfa22ef12b163926885: Status 404 returned error can't find the container with id a7d1105dddb65dbfa4b9247466579cdefc60b766a0304bfa22ef12b163926885
Jul 28 05:05:39.089065 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:39.089035    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9b749c-5fc6885d68-j77b9"]
Jul 28 05:05:39.090393 ip-10-0-129-187 kubenswrapper[2566]: W0728 05:05:39.090364    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod725cf087_0eeb_43a8_80a0_a1fa3d8a011a.slice/crio-bab0c7df4127e00f970556a7db003a0c2dfe6f9b4d048cc4b45865b99063d0fd WatchSource:0}: Error finding container bab0c7df4127e00f970556a7db003a0c2dfe6f9b4d048cc4b45865b99063d0fd: Status 404 returned error can't find the container with id bab0c7df4127e00f970556a7db003a0c2dfe6f9b4d048cc4b45865b99063d0fd
Jul 28 05:05:39.550978 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:39.550824    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9b749c-5fc6885d68-j77b9" event={"ID":"725cf087-0eeb-43a8-80a0-a1fa3d8a011a","Type":"ContainerStarted","Data":"b2e187c32f574be4e66fe2c3fc2e7f92e222affdc23e2019ac7a2479875c3edb"}
Jul 28 05:05:39.550978 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:39.550922    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9b749c-5fc6885d68-j77b9"
Jul 28 05:05:39.550978 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:39.550944    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9b749c-5fc6885d68-j77b9" event={"ID":"725cf087-0eeb-43a8-80a0-a1fa3d8a011a","Type":"ContainerStarted","Data":"bab0c7df4127e00f970556a7db003a0c2dfe6f9b4d048cc4b45865b99063d0fd"}
Jul 28 05:05:39.552570 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:39.552547    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b749c-78d7987948-4cz5k" event={"ID":"d0c1aa7f-20cb-457c-84d6-17f1ba08e019","Type":"ContainerStarted","Data":"92973813c19cae8a498bd553a5abb302f01ff34ec37273ea5c263edc60710d1e"}
Jul 28 05:05:39.552680 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:39.552591    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b749c-78d7987948-4cz5k"
Jul 28 05:05:39.552680 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:39.552607    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b749c-78d7987948-4cz5k" event={"ID":"d0c1aa7f-20cb-457c-84d6-17f1ba08e019","Type":"ContainerStarted","Data":"a7d1105dddb65dbfa4b9247466579cdefc60b766a0304bfa22ef12b163926885"}
Jul 28 05:05:39.568184 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:39.568138    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-9b749c-5fc6885d68-j77b9" podStartSLOduration=1.5681217840000001 podStartE2EDuration="1.568121784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:05:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:05:39.565791974 +0000 UTC m=+3239.955967995" watchObservedRunningTime="2026-07-28 05:05:39.568121784 +0000 UTC m=+3239.958297804"
Jul 28 05:05:39.581308 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:39.581265    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b749c-78d7987948-4cz5k" podStartSLOduration=1.581252686 podStartE2EDuration="1.581252686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:05:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:05:39.580440035 +0000 UTC m=+3239.970616069" watchObservedRunningTime="2026-07-28 05:05:39.581252686 +0000 UTC m=+3239.971428706"
Jul 28 05:05:39.742772 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:39.742730    2566 patch_prober.go:28] interesting pod/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Jul 28 05:05:39.743088 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:39.743048    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx" podUID="e74aefee-4f89-462e-89b9-f98c6abbe624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Jul 28 05:05:40.742473 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:40.742372    2566 patch_prober.go:28] interesting pod/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Jul 28 05:05:40.742473 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:40.742428    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx" podUID="e74aefee-4f89-462e-89b9-f98c6abbe624" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Jul 28 05:05:41.746042 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:41.746009    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx"
Jul 28 05:05:41.746516 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:41.746409    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx"
Jul 28 05:05:41.747085 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:41.747066    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx"
Jul 28 05:05:50.559634 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:50.559601    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b749c-78d7987948-4cz5k"
Jul 28 05:05:50.560111 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:50.559813    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9b749c-5fc6885d68-j77b9"
Jul 28 05:05:59.242465 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:05:59.242430    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-9b749c-5fc6885d68-j77b9_725cf087-0eeb-43a8-80a0-a1fa3d8a011a/payload-processing/0.log"
Jul 28 05:06:43.823415 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:06:43.823374    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4glpr_bfd2d3d8-f287-42d8-83c7-f82185c96735/console-operator/2.log"
Jul 28 05:06:43.831977 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:06:43.831958    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4glpr_bfd2d3d8-f287-42d8-83c7-f82185c96735/console-operator/2.log"
Jul 28 05:06:56.627799 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:06:56.627770    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-dc1efb-6bb5d85cdd-w7l8z_e702cc51-e3a8-4c8e-84b7-d2774888e09c/payload-processing/0.log"
Jul 28 05:06:56.900438 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:06:56.900343    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-9b749c-5fc6885d68-j77b9_725cf087-0eeb-43a8-80a0-a1fa3d8a011a/payload-processing/0.log"
Jul 28 05:06:57.486787 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:06:57.486749    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm"]
Jul 28 05:06:57.490372 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:06:57.490353    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm"
Jul 28 05:06:57.492947 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:06:57.492919    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-29b6e5-openshift-default-dockercfg-622cb\""
Jul 28 05:06:57.500630 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:06:57.500605    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm"]
Jul 28 05:06:57.556321 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:06:57.556288    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/914ffbb7-7664-4519-94d4-6758257c5863-credential-socket\") pod \"e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm\" (UID: \"914ffbb7-7664-4519-94d4-6758257c5863\") " pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm"
Jul 28 05:06:57.556526 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:06:57.556361    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/914ffbb7-7664-4519-94d4-6758257c5863-istiod-ca-cert\") pod \"e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm\" (UID: \"914ffbb7-7664-4519-94d4-6758257c5863\") " pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm"
Jul 28 05:06:57.556526 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:06:57.556406    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/914ffbb7-7664-4519-94d4-6758257c5863-workload-socket\") pod \"e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm\" (UID: \"914ffbb7-7664-4519-94d4-6758257c5863\") " pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm"
Jul 28 05:06:57.556526 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:06:57.556467    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/914ffbb7-7664-4519-94d4-6758257c5863-istio-data\") pod \"e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm\" (UID: \"914ffbb7-7664-4519-94d4-6758257c5863\") " pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm"
Jul 28 05:06:57.556709 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:06:57.556536    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/914ffbb7-7664-4519-94d4-6758257c5863-workload-certs\") pod \"e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm\" (UID: \"914ffbb7-7664-4519-94d4-6758257c5863\") " pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm"
Jul 28 05:06:57.556709 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:06:57.556580    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/914ffbb7-7664-4519-94d4-6758257c5863-istio-podinfo\") pod \"e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm\" (UID: \"914ffbb7-7664-4519-94d4-6758257c5863\") " pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm"
Jul 28 05:06:57.556709 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:06:57.556609    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj964\" (UniqueName: \"kubernetes.io/projected/914ffbb7-7664-4519-94d4-6758257c5863-kube-api-access-fj964\") pod \"e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm\" (UID: \"914ffbb7-7664-4519-94d4-6758257c5863\") " pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm"
Jul 28 05:06:57.556709 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:06:57.556683    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/914ffbb7-7664-4519-94d4-6758257c5863-istio-token\") pod \"e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm\" (UID: \"914ffbb7-7664-4519-94d4-6758257c5863\") " pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm"
Jul 28 05:06:57.556918 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:06:57.556734    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/914ffbb7-7664-4519-94d4-6758257c5863-istio-envoy\") pod \"e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm\" (UID: \"914ffbb7-7664-4519-94d4-6758257c5863\") " pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm"
Jul 28 05:06:57.657468 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:06:57.657432    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/914ffbb7-7664-4519-94d4-6758257c5863-istio-envoy\") pod \"e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm\" (UID: \"914ffbb7-7664-4519-94d4-6758257c5863\") " pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm"
Jul 28 05:06:57.657924 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:06:57.657477    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/914ffbb7-7664-4519-94d4-6758257c5863-credential-socket\") pod \"e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm\" (UID: \"914ffbb7-7664-4519-94d4-6758257c5863\") " pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm"
Jul 28 05:06:57.657924 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:06:57.657546    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/914ffbb7-7664-4519-94d4-6758257c5863-istiod-ca-cert\") pod \"e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm\" (UID: \"914ffbb7-7664-4519-94d4-6758257c5863\") " pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm"
Jul 28 05:06:57.657924 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:06:57.657580    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/914ffbb7-7664-4519-94d4-6758257c5863-workload-socket\") pod \"e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm\" (UID: \"914ffbb7-7664-4519-94d4-6758257c5863\") " pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm"
Jul 28 05:06:57.657924 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:06:57.657656    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/914ffbb7-7664-4519-94d4-6758257c5863-istio-data\") pod \"e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm\" (UID: \"914ffbb7-7664-4519-94d4-6758257c5863\") " pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm"
Jul 28 05:06:57.657924 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:06:57.657692    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/914ffbb7-7664-4519-94d4-6758257c5863-workload-certs\") pod \"e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm\" (UID: \"914ffbb7-7664-4519-94d4-6758257c5863\") " pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm"
Jul 28 05:06:57.657924 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:06:57.657717    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/914ffbb7-7664-4519-94d4-6758257c5863-istio-podinfo\") pod \"e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm\" (UID: \"914ffbb7-7664-4519-94d4-6758257c5863\") " pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm"
Jul 28 05:06:57.657924 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:06:57.657740    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj964\" (UniqueName: \"kubernetes.io/projected/914ffbb7-7664-4519-94d4-6758257c5863-kube-api-access-fj964\") pod \"e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm\" (UID: \"914ffbb7-7664-4519-94d4-6758257c5863\") " pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm"
Jul 28 05:06:57.657924 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:06:57.657810    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/914ffbb7-7664-4519-94d4-6758257c5863-istio-token\") pod \"e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm\" (UID: \"914ffbb7-7664-4519-94d4-6758257c5863\") " pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm"
Jul 28 05:06:57.658406 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:06:57.658119    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/914ffbb7-7664-4519-94d4-6758257c5863-workload-certs\") pod \"e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm\" (UID: \"914ffbb7-7664-4519-94d4-6758257c5863\") " pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm"
Jul 28 05:06:57.658406 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:06:57.658332    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/914ffbb7-7664-4519-94d4-6758257c5863-credential-socket\") pod \"e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm\" (UID: \"914ffbb7-7664-4519-94d4-6758257c5863\") " pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm"
Jul 28 05:06:57.658406 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:06:57.658367    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/914ffbb7-7664-4519-94d4-6758257c5863-istiod-ca-cert\") pod \"e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm\" (UID: \"914ffbb7-7664-4519-94d4-6758257c5863\") " pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm"
Jul 28 05:06:57.658406 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:06:57.658367    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/914ffbb7-7664-4519-94d4-6758257c5863-istio-data\") pod \"e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm\" (UID: \"914ffbb7-7664-4519-94d4-6758257c5863\") " pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm"
Jul 28 05:06:57.658588 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:06:57.658571    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/914ffbb7-7664-4519-94d4-6758257c5863-workload-socket\") pod \"e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm\" (UID: \"914ffbb7-7664-4519-94d4-6758257c5863\") " pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm"
Jul 28 05:06:57.659809 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:06:57.659783    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/914ffbb7-7664-4519-94d4-6758257c5863-istio-envoy\") pod \"e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm\" (UID: \"914ffbb7-7664-4519-94d4-6758257c5863\") " pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm"
Jul 28 05:06:57.660155 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:06:57.660135    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/914ffbb7-7664-4519-94d4-6758257c5863-istio-podinfo\") pod \"e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm\" (UID: \"914ffbb7-7664-4519-94d4-6758257c5863\") " pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm"
Jul 28 05:06:57.667402 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:06:57.667379    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj964\" (UniqueName: \"kubernetes.io/projected/914ffbb7-7664-4519-94d4-6758257c5863-kube-api-access-fj964\") pod \"e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm\" (UID: \"914ffbb7-7664-4519-94d4-6758257c5863\") " pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm"
Jul 28 05:06:57.667493 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:06:57.667469    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/914ffbb7-7664-4519-94d4-6758257c5863-istio-token\") pod \"e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm\" (UID: \"914ffbb7-7664-4519-94d4-6758257c5863\") " pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm"
Jul 28 05:06:57.803391 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:06:57.803348    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm"
Jul 28 05:06:57.944814 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:06:57.944782    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm"]
Jul 28 05:06:57.946354 ip-10-0-129-187 kubenswrapper[2566]: W0728 05:06:57.946326    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod914ffbb7_7664_4519_94d4_6758257c5863.slice/crio-d77851504a3ff00185b3841ba7b224be544ca529ac8e8a0787a9f08d76d66c03 WatchSource:0}: Error finding container d77851504a3ff00185b3841ba7b224be544ca529ac8e8a0787a9f08d76d66c03: Status 404 returned error can't find the container with id d77851504a3ff00185b3841ba7b224be544ca529ac8e8a0787a9f08d76d66c03
Jul 28 05:06:57.948641 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:06:57.948609    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 05:06:57.948700 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:06:57.948679    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 05:06:57.948737 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:06:57.948706    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 05:06:57.979158 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:06:57.979123    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm" event={"ID":"914ffbb7-7664-4519-94d4-6758257c5863","Type":"ContainerStarted","Data":"d77851504a3ff00185b3841ba7b224be544ca529ac8e8a0787a9f08d76d66c03"}
Jul 28 05:06:58.985572 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:06:58.985535    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm" event={"ID":"914ffbb7-7664-4519-94d4-6758257c5863","Type":"ContainerStarted","Data":"ce56d40ccf687eea112d6c240ddc86b2ce88bed5d51309e9ee4cdcde6d73ab3c"}
Jul 28 05:06:59.004601 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:06:59.004547    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm" podStartSLOduration=2.004529635 podStartE2EDuration="2.004529635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:06:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:06:59.003879499 +0000 UTC m=+3319.394055535" watchObservedRunningTime="2026-07-28 05:06:59.004529635 +0000 UTC m=+3319.394705657"
Jul 28 05:06:59.804480 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:06:59.804448    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm"
Jul 28 05:06:59.809722 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:06:59.809694    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm"
Jul 28 05:06:59.990592 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:06:59.990555    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm"
Jul 28 05:06:59.991595 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:06:59.991575    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm"
Jul 28 05:07:01.997562 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:01.997534    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-29b6e5-9b4f89c98-z7chq"]
Jul 28 05:07:02.005143 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:02.005065    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-29b6e5-9b4f89c98-z7chq"
Jul 28 05:07:02.008621 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:02.008543    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-29b6e5-dockercfg-rqmm5\""
Jul 28 05:07:02.008831 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:02.008811    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-29b6e5\""
Jul 28 05:07:02.011254 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:02.011224    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-29b6e5-9b4f89c98-z7chq"]
Jul 28 05:07:02.105131 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:02.105093    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srx8r\" (UniqueName: \"kubernetes.io/projected/e4d45902-7d49-4bdf-81f4-06fc25c43587-kube-api-access-srx8r\") pod \"payload-processing-e2e-ipp-cleanup-29b6e5-9b4f89c98-z7chq\" (UID: \"e4d45902-7d49-4bdf-81f4-06fc25c43587\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-29b6e5-9b4f89c98-z7chq"
Jul 28 05:07:02.105315 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:02.105148    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d45902-7d49-4bdf-81f4-06fc25c43587-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-29b6e5-9b4f89c98-z7chq\" (UID: \"e4d45902-7d49-4bdf-81f4-06fc25c43587\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-29b6e5-9b4f89c98-z7chq"
Jul 28 05:07:02.206473 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:02.206430    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srx8r\" (UniqueName: \"kubernetes.io/projected/e4d45902-7d49-4bdf-81f4-06fc25c43587-kube-api-access-srx8r\") pod \"payload-processing-e2e-ipp-cleanup-29b6e5-9b4f89c98-z7chq\" (UID: \"e4d45902-7d49-4bdf-81f4-06fc25c43587\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-29b6e5-9b4f89c98-z7chq"
Jul 28 05:07:02.206651 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:02.206490    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d45902-7d49-4bdf-81f4-06fc25c43587-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-29b6e5-9b4f89c98-z7chq\" (UID: \"e4d45902-7d49-4bdf-81f4-06fc25c43587\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-29b6e5-9b4f89c98-z7chq"
Jul 28 05:07:02.207308 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:02.207288    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d45902-7d49-4bdf-81f4-06fc25c43587-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-29b6e5-9b4f89c98-z7chq\" (UID: \"e4d45902-7d49-4bdf-81f4-06fc25c43587\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-29b6e5-9b4f89c98-z7chq"
Jul 28 05:07:02.215217 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:02.215190    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srx8r\" (UniqueName: \"kubernetes.io/projected/e4d45902-7d49-4bdf-81f4-06fc25c43587-kube-api-access-srx8r\") pod \"payload-processing-e2e-ipp-cleanup-29b6e5-9b4f89c98-z7chq\" (UID: \"e4d45902-7d49-4bdf-81f4-06fc25c43587\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-29b6e5-9b4f89c98-z7chq"
Jul 28 05:07:02.318469 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:02.318426    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-29b6e5-9b4f89c98-z7chq"
Jul 28 05:07:02.466796 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:02.466714    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-29b6e5-9b4f89c98-z7chq"]
Jul 28 05:07:02.467506 ip-10-0-129-187 kubenswrapper[2566]: W0728 05:07:02.467475    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4d45902_7d49_4bdf_81f4_06fc25c43587.slice/crio-d7abb9615dd767b52e9733ad3dc563a676110724b54a3a3c35412a3ea1788e7b WatchSource:0}: Error finding container d7abb9615dd767b52e9733ad3dc563a676110724b54a3a3c35412a3ea1788e7b: Status 404 returned error can't find the container with id d7abb9615dd767b52e9733ad3dc563a676110724b54a3a3c35412a3ea1788e7b
Jul 28 05:07:03.018937 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:03.018897    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-29b6e5-9b4f89c98-z7chq" event={"ID":"e4d45902-7d49-4bdf-81f4-06fc25c43587","Type":"ContainerStarted","Data":"ce0aed803e54e0ffa2a47c7e9a5fd878c477e4f5404ad1c5cedfa7a9d9a03cfc"}
Jul 28 05:07:03.019297 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:03.018960    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-29b6e5-9b4f89c98-z7chq" event={"ID":"e4d45902-7d49-4bdf-81f4-06fc25c43587","Type":"ContainerStarted","Data":"d7abb9615dd767b52e9733ad3dc563a676110724b54a3a3c35412a3ea1788e7b"}
Jul 28 05:07:03.019297 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:03.018983    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-29b6e5-9b4f89c98-z7chq"
Jul 28 05:07:03.036409 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:03.036359    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-29b6e5-9b4f89c98-z7chq" podStartSLOduration=2.036342908 podStartE2EDuration="2.036342908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:07:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:07:03.033988905 +0000 UTC m=+3323.424164926" watchObservedRunningTime="2026-07-28 05:07:03.036342908 +0000 UTC m=+3323.426518942"
Jul 28 05:07:14.027046 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:14.027010    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-29b6e5-9b4f89c98-z7chq"
Jul 28 05:07:23.137322 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:23.137287    2566 generic.go:356] "Generic (PLEG): container finished" podID="914ffbb7-7664-4519-94d4-6758257c5863" containerID="ce56d40ccf687eea112d6c240ddc86b2ce88bed5d51309e9ee4cdcde6d73ab3c" exitCode=0
Jul 28 05:07:23.137750 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:23.137338    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm" event={"ID":"914ffbb7-7664-4519-94d4-6758257c5863","Type":"ContainerDied","Data":"ce56d40ccf687eea112d6c240ddc86b2ce88bed5d51309e9ee4cdcde6d73ab3c"}
Jul 28 05:07:23.137868 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:23.137827    2566 scope.go:117] "RemoveContainer" containerID="ce56d40ccf687eea112d6c240ddc86b2ce88bed5d51309e9ee4cdcde6d73ab3c"
Jul 28 05:07:23.139156 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:23.139126    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 05:07:23.139253 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:23.139202    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 05:07:23.139253 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:23.139238    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 28 05:07:24.144830 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:24.144788    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm" event={"ID":"914ffbb7-7664-4519-94d4-6758257c5863","Type":"ContainerStarted","Data":"35e9e2db12f8f3ae3f4c0bdd86d3fc1aa8333985976e82cc1fea429951909e6c"}
Jul 28 05:07:24.804288 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:24.804253    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm"
Jul 28 05:07:24.804471 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:24.804437    2566 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Jul 28 05:07:24.804517 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:24.804481    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm" podUID="914ffbb7-7664-4519-94d4-6758257c5863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Jul 28 05:07:25.804088 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:25.804049    2566 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Jul 28 05:07:25.804490 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:25.804107    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm" podUID="914ffbb7-7664-4519-94d4-6758257c5863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Jul 28 05:07:26.804067 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:26.804029    2566 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Jul 28 05:07:26.804267 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:26.804088    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm" podUID="914ffbb7-7664-4519-94d4-6758257c5863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Jul 28 05:07:27.097242 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:27.097145    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-29b6e5-9b4f89c98-z7chq"]
Jul 28 05:07:27.097410 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:27.097368    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-29b6e5-9b4f89c98-z7chq" podUID="e4d45902-7d49-4bdf-81f4-06fc25c43587" containerName="payload-processing" containerID="cri-o://ce0aed803e54e0ffa2a47c7e9a5fd878c477e4f5404ad1c5cedfa7a9d9a03cfc" gracePeriod=30
Jul 28 05:07:27.405350 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:27.405320    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-29b6e5-9b4f89c98-z7chq"
Jul 28 05:07:27.462991 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:27.462953    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d45902-7d49-4bdf-81f4-06fc25c43587-plugins-config-volume\") pod \"e4d45902-7d49-4bdf-81f4-06fc25c43587\" (UID: \"e4d45902-7d49-4bdf-81f4-06fc25c43587\") "
Jul 28 05:07:27.463151 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:27.463030    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-srx8r\" (UniqueName: \"kubernetes.io/projected/e4d45902-7d49-4bdf-81f4-06fc25c43587-kube-api-access-srx8r\") pod \"e4d45902-7d49-4bdf-81f4-06fc25c43587\" (UID: \"e4d45902-7d49-4bdf-81f4-06fc25c43587\") "
Jul 28 05:07:27.463332 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:27.463308    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4d45902-7d49-4bdf-81f4-06fc25c43587-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e4d45902-7d49-4bdf-81f4-06fc25c43587" (UID: "e4d45902-7d49-4bdf-81f4-06fc25c43587"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 05:07:27.463588 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:27.463558    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4d45902-7d49-4bdf-81f4-06fc25c43587-plugins-config-volume\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:07:27.465136 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:27.465109    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4d45902-7d49-4bdf-81f4-06fc25c43587-kube-api-access-srx8r" (OuterVolumeSpecName: "kube-api-access-srx8r") pod "e4d45902-7d49-4bdf-81f4-06fc25c43587" (UID: "e4d45902-7d49-4bdf-81f4-06fc25c43587"). InnerVolumeSpecName "kube-api-access-srx8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 05:07:27.564893 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:27.564864    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srx8r\" (UniqueName: \"kubernetes.io/projected/e4d45902-7d49-4bdf-81f4-06fc25c43587-kube-api-access-srx8r\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:07:27.803912 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:27.803877    2566 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Jul 28 05:07:27.804102 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:27.803929    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm" podUID="914ffbb7-7664-4519-94d4-6758257c5863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Jul 28 05:07:28.173297 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:28.173265    2566 generic.go:356] "Generic (PLEG): container finished" podID="e4d45902-7d49-4bdf-81f4-06fc25c43587" containerID="ce0aed803e54e0ffa2a47c7e9a5fd878c477e4f5404ad1c5cedfa7a9d9a03cfc" exitCode=0
Jul 28 05:07:28.173712 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:28.173312    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-29b6e5-9b4f89c98-z7chq"
Jul 28 05:07:28.173712 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:28.173343    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-29b6e5-9b4f89c98-z7chq" event={"ID":"e4d45902-7d49-4bdf-81f4-06fc25c43587","Type":"ContainerDied","Data":"ce0aed803e54e0ffa2a47c7e9a5fd878c477e4f5404ad1c5cedfa7a9d9a03cfc"}
Jul 28 05:07:28.173712 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:28.173394    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-29b6e5-9b4f89c98-z7chq" event={"ID":"e4d45902-7d49-4bdf-81f4-06fc25c43587","Type":"ContainerDied","Data":"d7abb9615dd767b52e9733ad3dc563a676110724b54a3a3c35412a3ea1788e7b"}
Jul 28 05:07:28.173712 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:28.173416    2566 scope.go:117] "RemoveContainer" containerID="ce0aed803e54e0ffa2a47c7e9a5fd878c477e4f5404ad1c5cedfa7a9d9a03cfc"
Jul 28 05:07:28.182241 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:28.182215    2566 scope.go:117] "RemoveContainer" containerID="ce0aed803e54e0ffa2a47c7e9a5fd878c477e4f5404ad1c5cedfa7a9d9a03cfc"
Jul 28 05:07:28.182496 ip-10-0-129-187 kubenswrapper[2566]: E0728 05:07:28.182479    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce0aed803e54e0ffa2a47c7e9a5fd878c477e4f5404ad1c5cedfa7a9d9a03cfc\": container with ID starting with ce0aed803e54e0ffa2a47c7e9a5fd878c477e4f5404ad1c5cedfa7a9d9a03cfc not found: ID does not exist" containerID="ce0aed803e54e0ffa2a47c7e9a5fd878c477e4f5404ad1c5cedfa7a9d9a03cfc"
Jul 28 05:07:28.182548 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:28.182506    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce0aed803e54e0ffa2a47c7e9a5fd878c477e4f5404ad1c5cedfa7a9d9a03cfc"} err="failed to get container status \"ce0aed803e54e0ffa2a47c7e9a5fd878c477e4f5404ad1c5cedfa7a9d9a03cfc\": rpc error: code = NotFound desc = could not find container \"ce0aed803e54e0ffa2a47c7e9a5fd878c477e4f5404ad1c5cedfa7a9d9a03cfc\": container with ID starting with ce0aed803e54e0ffa2a47c7e9a5fd878c477e4f5404ad1c5cedfa7a9d9a03cfc not found: ID does not exist"
Jul 28 05:07:28.190128 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:28.190100    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-29b6e5-9b4f89c98-z7chq"]
Jul 28 05:07:28.193944 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:28.193923    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-29b6e5-9b4f89c98-z7chq"]
Jul 28 05:07:28.808102 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:28.808076    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm"
Jul 28 05:07:28.808462 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:28.808438    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm"
Jul 28 05:07:28.809187 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:28.809171    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm"
Jul 28 05:07:30.089280 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:30.089243    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4d45902-7d49-4bdf-81f4-06fc25c43587" path="/var/lib/kubelet/pods/e4d45902-7d49-4bdf-81f4-06fc25c43587/volumes"
Jul 28 05:07:37.950465 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:37.950419    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm"]
Jul 28 05:07:37.950960 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:37.950663    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm" podUID="914ffbb7-7664-4519-94d4-6758257c5863" containerName="istio-proxy" containerID="cri-o://35e9e2db12f8f3ae3f4c0bdd86d3fc1aa8333985976e82cc1fea429951909e6c" gracePeriod=30
Jul 28 05:07:43.222640 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:43.222572    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm"
Jul 28 05:07:43.261564 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:43.261531    2566 generic.go:356] "Generic (PLEG): container finished" podID="914ffbb7-7664-4519-94d4-6758257c5863" containerID="35e9e2db12f8f3ae3f4c0bdd86d3fc1aa8333985976e82cc1fea429951909e6c" exitCode=0
Jul 28 05:07:43.261750 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:43.261579    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm"
Jul 28 05:07:43.261750 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:43.261629    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm" event={"ID":"914ffbb7-7664-4519-94d4-6758257c5863","Type":"ContainerDied","Data":"35e9e2db12f8f3ae3f4c0bdd86d3fc1aa8333985976e82cc1fea429951909e6c"}
Jul 28 05:07:43.261750 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:43.261685    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm" event={"ID":"914ffbb7-7664-4519-94d4-6758257c5863","Type":"ContainerDied","Data":"d77851504a3ff00185b3841ba7b224be544ca529ac8e8a0787a9f08d76d66c03"}
Jul 28 05:07:43.261750 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:43.261708    2566 scope.go:117] "RemoveContainer" containerID="35e9e2db12f8f3ae3f4c0bdd86d3fc1aa8333985976e82cc1fea429951909e6c"
Jul 28 05:07:43.272769 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:43.272750    2566 scope.go:117] "RemoveContainer" containerID="ce56d40ccf687eea112d6c240ddc86b2ce88bed5d51309e9ee4cdcde6d73ab3c"
Jul 28 05:07:43.281053 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:43.281031    2566 scope.go:117] "RemoveContainer" containerID="35e9e2db12f8f3ae3f4c0bdd86d3fc1aa8333985976e82cc1fea429951909e6c"
Jul 28 05:07:43.281332 ip-10-0-129-187 kubenswrapper[2566]: E0728 05:07:43.281311    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35e9e2db12f8f3ae3f4c0bdd86d3fc1aa8333985976e82cc1fea429951909e6c\": container with ID starting with 35e9e2db12f8f3ae3f4c0bdd86d3fc1aa8333985976e82cc1fea429951909e6c not found: ID does not exist" containerID="35e9e2db12f8f3ae3f4c0bdd86d3fc1aa8333985976e82cc1fea429951909e6c"
Jul 28 05:07:43.281419 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:43.281344    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35e9e2db12f8f3ae3f4c0bdd86d3fc1aa8333985976e82cc1fea429951909e6c"} err="failed to get container status \"35e9e2db12f8f3ae3f4c0bdd86d3fc1aa8333985976e82cc1fea429951909e6c\": rpc error: code = NotFound desc = could not find container \"35e9e2db12f8f3ae3f4c0bdd86d3fc1aa8333985976e82cc1fea429951909e6c\": container with ID starting with 35e9e2db12f8f3ae3f4c0bdd86d3fc1aa8333985976e82cc1fea429951909e6c not found: ID does not exist"
Jul 28 05:07:43.281419 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:43.281368    2566 scope.go:117] "RemoveContainer" containerID="ce56d40ccf687eea112d6c240ddc86b2ce88bed5d51309e9ee4cdcde6d73ab3c"
Jul 28 05:07:43.281627 ip-10-0-129-187 kubenswrapper[2566]: E0728 05:07:43.281599    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce56d40ccf687eea112d6c240ddc86b2ce88bed5d51309e9ee4cdcde6d73ab3c\": container with ID starting with ce56d40ccf687eea112d6c240ddc86b2ce88bed5d51309e9ee4cdcde6d73ab3c not found: ID does not exist" containerID="ce56d40ccf687eea112d6c240ddc86b2ce88bed5d51309e9ee4cdcde6d73ab3c"
Jul 28 05:07:43.281715 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:43.281635    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce56d40ccf687eea112d6c240ddc86b2ce88bed5d51309e9ee4cdcde6d73ab3c"} err="failed to get container status \"ce56d40ccf687eea112d6c240ddc86b2ce88bed5d51309e9ee4cdcde6d73ab3c\": rpc error: code = NotFound desc = could not find container \"ce56d40ccf687eea112d6c240ddc86b2ce88bed5d51309e9ee4cdcde6d73ab3c\": container with ID starting with ce56d40ccf687eea112d6c240ddc86b2ce88bed5d51309e9ee4cdcde6d73ab3c not found: ID does not exist"
Jul 28 05:07:43.323688 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:43.323656    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fj964\" (UniqueName: \"kubernetes.io/projected/914ffbb7-7664-4519-94d4-6758257c5863-kube-api-access-fj964\") pod \"914ffbb7-7664-4519-94d4-6758257c5863\" (UID: \"914ffbb7-7664-4519-94d4-6758257c5863\") "
Jul 28 05:07:43.323862 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:43.323701    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/914ffbb7-7664-4519-94d4-6758257c5863-istiod-ca-cert\") pod \"914ffbb7-7664-4519-94d4-6758257c5863\" (UID: \"914ffbb7-7664-4519-94d4-6758257c5863\") "
Jul 28 05:07:43.323862 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:43.323726    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/914ffbb7-7664-4519-94d4-6758257c5863-workload-certs\") pod \"914ffbb7-7664-4519-94d4-6758257c5863\" (UID: \"914ffbb7-7664-4519-94d4-6758257c5863\") "
Jul 28 05:07:43.323862 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:43.323763    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/914ffbb7-7664-4519-94d4-6758257c5863-istio-data\") pod \"914ffbb7-7664-4519-94d4-6758257c5863\" (UID: \"914ffbb7-7664-4519-94d4-6758257c5863\") "
Jul 28 05:07:43.323862 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:43.323777    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/914ffbb7-7664-4519-94d4-6758257c5863-istio-token\") pod \"914ffbb7-7664-4519-94d4-6758257c5863\" (UID: \"914ffbb7-7664-4519-94d4-6758257c5863\") "
Jul 28 05:07:43.323862 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:43.323830    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/914ffbb7-7664-4519-94d4-6758257c5863-istio-envoy\") pod \"914ffbb7-7664-4519-94d4-6758257c5863\" (UID: \"914ffbb7-7664-4519-94d4-6758257c5863\") "
Jul 28 05:07:43.324153 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:43.323918    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/914ffbb7-7664-4519-94d4-6758257c5863-credential-socket\") pod \"914ffbb7-7664-4519-94d4-6758257c5863\" (UID: \"914ffbb7-7664-4519-94d4-6758257c5863\") "
Jul 28 05:07:43.324153 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:43.323969    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/914ffbb7-7664-4519-94d4-6758257c5863-workload-socket\") pod \"914ffbb7-7664-4519-94d4-6758257c5863\" (UID: \"914ffbb7-7664-4519-94d4-6758257c5863\") "
Jul 28 05:07:43.324153 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:43.323991    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/914ffbb7-7664-4519-94d4-6758257c5863-istio-podinfo\") pod \"914ffbb7-7664-4519-94d4-6758257c5863\" (UID: \"914ffbb7-7664-4519-94d4-6758257c5863\") "
Jul 28 05:07:43.324153 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:43.324047    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914ffbb7-7664-4519-94d4-6758257c5863-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "914ffbb7-7664-4519-94d4-6758257c5863" (UID: "914ffbb7-7664-4519-94d4-6758257c5863"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 05:07:43.324378 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:43.324287    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914ffbb7-7664-4519-94d4-6758257c5863-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "914ffbb7-7664-4519-94d4-6758257c5863" (UID: "914ffbb7-7664-4519-94d4-6758257c5863"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 05:07:43.324378 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:43.324330    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914ffbb7-7664-4519-94d4-6758257c5863-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "914ffbb7-7664-4519-94d4-6758257c5863" (UID: "914ffbb7-7664-4519-94d4-6758257c5863"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 05:07:43.324378 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:43.324341    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/914ffbb7-7664-4519-94d4-6758257c5863-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "914ffbb7-7664-4519-94d4-6758257c5863" (UID: "914ffbb7-7664-4519-94d4-6758257c5863"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 05:07:43.324539 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:43.324436    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/914ffbb7-7664-4519-94d4-6758257c5863-credential-socket\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:07:43.324539 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:43.324460    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/914ffbb7-7664-4519-94d4-6758257c5863-workload-socket\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:07:43.324539 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:43.324476    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/914ffbb7-7664-4519-94d4-6758257c5863-istiod-ca-cert\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:07:43.324539 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:43.324489    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/914ffbb7-7664-4519-94d4-6758257c5863-workload-certs\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:07:43.324872 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:43.324819    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914ffbb7-7664-4519-94d4-6758257c5863-istio-data" (OuterVolumeSpecName: "istio-data") pod "914ffbb7-7664-4519-94d4-6758257c5863" (UID: "914ffbb7-7664-4519-94d4-6758257c5863"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 05:07:43.326085 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:43.326053    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/914ffbb7-7664-4519-94d4-6758257c5863-kube-api-access-fj964" (OuterVolumeSpecName: "kube-api-access-fj964") pod "914ffbb7-7664-4519-94d4-6758257c5863" (UID: "914ffbb7-7664-4519-94d4-6758257c5863"). InnerVolumeSpecName "kube-api-access-fj964". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 05:07:43.326210 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:43.326126    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/914ffbb7-7664-4519-94d4-6758257c5863-istio-token" (OuterVolumeSpecName: "istio-token") pod "914ffbb7-7664-4519-94d4-6758257c5863" (UID: "914ffbb7-7664-4519-94d4-6758257c5863"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 05:07:43.326344 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:43.326318    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/914ffbb7-7664-4519-94d4-6758257c5863-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "914ffbb7-7664-4519-94d4-6758257c5863" (UID: "914ffbb7-7664-4519-94d4-6758257c5863"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 05:07:43.326405 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:43.326367    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/914ffbb7-7664-4519-94d4-6758257c5863-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "914ffbb7-7664-4519-94d4-6758257c5863" (UID: "914ffbb7-7664-4519-94d4-6758257c5863"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 05:07:43.425420 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:43.425388    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/914ffbb7-7664-4519-94d4-6758257c5863-istio-data\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:07:43.425420 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:43.425414    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/914ffbb7-7664-4519-94d4-6758257c5863-istio-token\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:07:43.425420 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:43.425425    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/914ffbb7-7664-4519-94d4-6758257c5863-istio-envoy\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:07:43.425654 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:43.425433    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/914ffbb7-7664-4519-94d4-6758257c5863-istio-podinfo\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:07:43.425654 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:43.425443    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj964\" (UniqueName: \"kubernetes.io/projected/914ffbb7-7664-4519-94d4-6758257c5863-kube-api-access-fj964\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:07:43.584017 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:43.583986    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm"]
Jul 28 05:07:43.587733 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:43.587709    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-29b6e5-openshift-default-747c7c56cd-7gknm"]
Jul 28 05:07:44.088895 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:44.088863    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="914ffbb7-7664-4519-94d4-6758257c5863" path="/var/lib/kubelet/pods/914ffbb7-7664-4519-94d4-6758257c5863/volumes"
Jul 28 05:07:57.381960 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:57.381924    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dc1efb-6bb5d85cdd-w7l8z"]
Jul 28 05:07:57.382470 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:57.382179    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-dc1efb-6bb5d85cdd-w7l8z" podUID="e702cc51-e3a8-4c8e-84b7-d2774888e09c" containerName="payload-processing" containerID="cri-o://a6737dab7437ff2ae11d755e4638e64bc19c24eca611be883b2b820f28715596" gracePeriod=30
Jul 28 05:07:57.633371 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:57.633316    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dc1efb-6bb5d85cdd-w7l8z"
Jul 28 05:07:57.668222 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:57.668189    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e702cc51-e3a8-4c8e-84b7-d2774888e09c-plugins-config-volume\") pod \"e702cc51-e3a8-4c8e-84b7-d2774888e09c\" (UID: \"e702cc51-e3a8-4c8e-84b7-d2774888e09c\") "
Jul 28 05:07:57.668356 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:57.668340    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4dt5p\" (UniqueName: \"kubernetes.io/projected/e702cc51-e3a8-4c8e-84b7-d2774888e09c-kube-api-access-4dt5p\") pod \"e702cc51-e3a8-4c8e-84b7-d2774888e09c\" (UID: \"e702cc51-e3a8-4c8e-84b7-d2774888e09c\") "
Jul 28 05:07:57.668649 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:57.668624    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e702cc51-e3a8-4c8e-84b7-d2774888e09c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e702cc51-e3a8-4c8e-84b7-d2774888e09c" (UID: "e702cc51-e3a8-4c8e-84b7-d2774888e09c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 05:07:57.670380 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:57.670339    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e702cc51-e3a8-4c8e-84b7-d2774888e09c-kube-api-access-4dt5p" (OuterVolumeSpecName: "kube-api-access-4dt5p") pod "e702cc51-e3a8-4c8e-84b7-d2774888e09c" (UID: "e702cc51-e3a8-4c8e-84b7-d2774888e09c"). InnerVolumeSpecName "kube-api-access-4dt5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 05:07:57.770023 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:57.769989    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dt5p\" (UniqueName: \"kubernetes.io/projected/e702cc51-e3a8-4c8e-84b7-d2774888e09c-kube-api-access-4dt5p\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:07:57.770023 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:57.770018    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e702cc51-e3a8-4c8e-84b7-d2774888e09c-plugins-config-volume\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:07:58.351813 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:58.351718    2566 generic.go:356] "Generic (PLEG): container finished" podID="e702cc51-e3a8-4c8e-84b7-d2774888e09c" containerID="a6737dab7437ff2ae11d755e4638e64bc19c24eca611be883b2b820f28715596" exitCode=0
Jul 28 05:07:58.351813 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:58.351759    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dc1efb-6bb5d85cdd-w7l8z"
Jul 28 05:07:58.352066 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:58.351810    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dc1efb-6bb5d85cdd-w7l8z" event={"ID":"e702cc51-e3a8-4c8e-84b7-d2774888e09c","Type":"ContainerDied","Data":"a6737dab7437ff2ae11d755e4638e64bc19c24eca611be883b2b820f28715596"}
Jul 28 05:07:58.352066 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:58.351891    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dc1efb-6bb5d85cdd-w7l8z" event={"ID":"e702cc51-e3a8-4c8e-84b7-d2774888e09c","Type":"ContainerDied","Data":"fd5d6641065cb06aee1f6f595e9037c57ee12227ba4d8fa1309096733d9c997e"}
Jul 28 05:07:58.352066 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:58.351920    2566 scope.go:117] "RemoveContainer" containerID="a6737dab7437ff2ae11d755e4638e64bc19c24eca611be883b2b820f28715596"
Jul 28 05:07:58.360893 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:58.360854    2566 scope.go:117] "RemoveContainer" containerID="a6737dab7437ff2ae11d755e4638e64bc19c24eca611be883b2b820f28715596"
Jul 28 05:07:58.361189 ip-10-0-129-187 kubenswrapper[2566]: E0728 05:07:58.361160    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6737dab7437ff2ae11d755e4638e64bc19c24eca611be883b2b820f28715596\": container with ID starting with a6737dab7437ff2ae11d755e4638e64bc19c24eca611be883b2b820f28715596 not found: ID does not exist" containerID="a6737dab7437ff2ae11d755e4638e64bc19c24eca611be883b2b820f28715596"
Jul 28 05:07:58.361298 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:58.361200    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6737dab7437ff2ae11d755e4638e64bc19c24eca611be883b2b820f28715596"} err="failed to get container status \"a6737dab7437ff2ae11d755e4638e64bc19c24eca611be883b2b820f28715596\": rpc error: code = NotFound desc = could not find container \"a6737dab7437ff2ae11d755e4638e64bc19c24eca611be883b2b820f28715596\": container with ID starting with a6737dab7437ff2ae11d755e4638e64bc19c24eca611be883b2b820f28715596 not found: ID does not exist"
Jul 28 05:07:58.367249 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:58.367227    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dc1efb-6bb5d85cdd-w7l8z"]
Jul 28 05:07:58.371198 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:07:58.371175    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dc1efb-6bb5d85cdd-w7l8z"]
Jul 28 05:08:00.088945 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:08:00.088911    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e702cc51-e3a8-4c8e-84b7-d2774888e09c" path="/var/lib/kubelet/pods/e702cc51-e3a8-4c8e-84b7-d2774888e09c/volumes"
Jul 28 05:08:49.965018 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:08:49.964982    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx"]
Jul 28 05:08:49.965470 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:08:49.965239    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx" podUID="e74aefee-4f89-462e-89b9-f98c6abbe624" containerName="istio-proxy" containerID="cri-o://917ef30d9857eca03e3b3736517e14a6b07b3a4f8bc611684e7afca8132530d6" gracePeriod=30
Jul 28 05:08:55.221312 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:08:55.221290    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx"
Jul 28 05:08:55.310654 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:08:55.310625    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e74aefee-4f89-462e-89b9-f98c6abbe624-workload-certs\") pod \"e74aefee-4f89-462e-89b9-f98c6abbe624\" (UID: \"e74aefee-4f89-462e-89b9-f98c6abbe624\") "
Jul 28 05:08:55.310807 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:08:55.310658    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e74aefee-4f89-462e-89b9-f98c6abbe624-istiod-ca-cert\") pod \"e74aefee-4f89-462e-89b9-f98c6abbe624\" (UID: \"e74aefee-4f89-462e-89b9-f98c6abbe624\") "
Jul 28 05:08:55.310807 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:08:55.310686    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e74aefee-4f89-462e-89b9-f98c6abbe624-istio-envoy\") pod \"e74aefee-4f89-462e-89b9-f98c6abbe624\" (UID: \"e74aefee-4f89-462e-89b9-f98c6abbe624\") "
Jul 28 05:08:55.310807 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:08:55.310744    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e74aefee-4f89-462e-89b9-f98c6abbe624-istio-token\") pod \"e74aefee-4f89-462e-89b9-f98c6abbe624\" (UID: \"e74aefee-4f89-462e-89b9-f98c6abbe624\") "
Jul 28 05:08:55.310807 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:08:55.310777    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e74aefee-4f89-462e-89b9-f98c6abbe624-istio-data\") pod \"e74aefee-4f89-462e-89b9-f98c6abbe624\" (UID: \"e74aefee-4f89-462e-89b9-f98c6abbe624\") "
Jul 28 05:08:55.311050 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:08:55.310810    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8n7ls\" (UniqueName: \"kubernetes.io/projected/e74aefee-4f89-462e-89b9-f98c6abbe624-kube-api-access-8n7ls\") pod \"e74aefee-4f89-462e-89b9-f98c6abbe624\" (UID: \"e74aefee-4f89-462e-89b9-f98c6abbe624\") "
Jul 28 05:08:55.311050 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:08:55.310870    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74aefee-4f89-462e-89b9-f98c6abbe624-workload-socket\") pod \"e74aefee-4f89-462e-89b9-f98c6abbe624\" (UID: \"e74aefee-4f89-462e-89b9-f98c6abbe624\") "
Jul 28 05:08:55.311050 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:08:55.310894    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74aefee-4f89-462e-89b9-f98c6abbe624-credential-socket\") pod \"e74aefee-4f89-462e-89b9-f98c6abbe624\" (UID: \"e74aefee-4f89-462e-89b9-f98c6abbe624\") "
Jul 28 05:08:55.311050 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:08:55.310906    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e74aefee-4f89-462e-89b9-f98c6abbe624-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e74aefee-4f89-462e-89b9-f98c6abbe624" (UID: "e74aefee-4f89-462e-89b9-f98c6abbe624"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 05:08:55.311050 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:08:55.310933    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e74aefee-4f89-462e-89b9-f98c6abbe624-istio-podinfo\") pod \"e74aefee-4f89-462e-89b9-f98c6abbe624\" (UID: \"e74aefee-4f89-462e-89b9-f98c6abbe624\") "
Jul 28 05:08:55.311313 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:08:55.311166    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e74aefee-4f89-462e-89b9-f98c6abbe624-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e74aefee-4f89-462e-89b9-f98c6abbe624" (UID: "e74aefee-4f89-462e-89b9-f98c6abbe624"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 05:08:55.311313 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:08:55.311182    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e74aefee-4f89-462e-89b9-f98c6abbe624-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e74aefee-4f89-462e-89b9-f98c6abbe624" (UID: "e74aefee-4f89-462e-89b9-f98c6abbe624"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 05:08:55.311417 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:08:55.311392    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e74aefee-4f89-462e-89b9-f98c6abbe624-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e74aefee-4f89-462e-89b9-f98c6abbe624" (UID: "e74aefee-4f89-462e-89b9-f98c6abbe624"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 05:08:55.311494 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:08:55.311471    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e74aefee-4f89-462e-89b9-f98c6abbe624-workload-certs\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:08:55.311551 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:08:55.311498    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e74aefee-4f89-462e-89b9-f98c6abbe624-istiod-ca-cert\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:08:55.311551 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:08:55.311511    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74aefee-4f89-462e-89b9-f98c6abbe624-workload-socket\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:08:55.311691 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:08:55.311669    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e74aefee-4f89-462e-89b9-f98c6abbe624-istio-data" (OuterVolumeSpecName: "istio-data") pod "e74aefee-4f89-462e-89b9-f98c6abbe624" (UID: "e74aefee-4f89-462e-89b9-f98c6abbe624"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 05:08:55.312958 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:08:55.312926    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e74aefee-4f89-462e-89b9-f98c6abbe624-istio-token" (OuterVolumeSpecName: "istio-token") pod "e74aefee-4f89-462e-89b9-f98c6abbe624" (UID: "e74aefee-4f89-462e-89b9-f98c6abbe624"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 05:08:55.313274 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:08:55.313248    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e74aefee-4f89-462e-89b9-f98c6abbe624-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e74aefee-4f89-462e-89b9-f98c6abbe624" (UID: "e74aefee-4f89-462e-89b9-f98c6abbe624"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 05:08:55.313423 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:08:55.313333    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e74aefee-4f89-462e-89b9-f98c6abbe624-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e74aefee-4f89-462e-89b9-f98c6abbe624" (UID: "e74aefee-4f89-462e-89b9-f98c6abbe624"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 05:08:55.313423 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:08:55.313351    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e74aefee-4f89-462e-89b9-f98c6abbe624-kube-api-access-8n7ls" (OuterVolumeSpecName: "kube-api-access-8n7ls") pod "e74aefee-4f89-462e-89b9-f98c6abbe624" (UID: "e74aefee-4f89-462e-89b9-f98c6abbe624"). InnerVolumeSpecName "kube-api-access-8n7ls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 05:08:55.412634 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:08:55.412600    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n7ls\" (UniqueName: \"kubernetes.io/projected/e74aefee-4f89-462e-89b9-f98c6abbe624-kube-api-access-8n7ls\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:08:55.412634 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:08:55.412629    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74aefee-4f89-462e-89b9-f98c6abbe624-credential-socket\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:08:55.412804 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:08:55.412642    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e74aefee-4f89-462e-89b9-f98c6abbe624-istio-podinfo\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:08:55.412804 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:08:55.412654    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e74aefee-4f89-462e-89b9-f98c6abbe624-istio-envoy\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:08:55.412804 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:08:55.412665    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e74aefee-4f89-462e-89b9-f98c6abbe624-istio-token\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:08:55.412804 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:08:55.412675    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e74aefee-4f89-462e-89b9-f98c6abbe624-istio-data\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:08:55.651316 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:08:55.651241    2566 generic.go:356] "Generic (PLEG): container finished" podID="e74aefee-4f89-462e-89b9-f98c6abbe624" containerID="917ef30d9857eca03e3b3736517e14a6b07b3a4f8bc611684e7afca8132530d6" exitCode=0
Jul 28 05:08:55.651316 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:08:55.651296    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx"
Jul 28 05:08:55.651492 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:08:55.651329    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx" event={"ID":"e74aefee-4f89-462e-89b9-f98c6abbe624","Type":"ContainerDied","Data":"917ef30d9857eca03e3b3736517e14a6b07b3a4f8bc611684e7afca8132530d6"}
Jul 28 05:08:55.651492 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:08:55.651394    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx" event={"ID":"e74aefee-4f89-462e-89b9-f98c6abbe624","Type":"ContainerDied","Data":"43f2946cc0fa1149921574b1faeec583c3c11f12ec47b07e87680ce310869b69"}
Jul 28 05:08:55.651492 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:08:55.651426    2566 scope.go:117] "RemoveContainer" containerID="917ef30d9857eca03e3b3736517e14a6b07b3a4f8bc611684e7afca8132530d6"
Jul 28 05:08:55.660696 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:08:55.660680    2566 scope.go:117] "RemoveContainer" containerID="158eace5807a18d93358a39a7c0d9f24c0e7ba4d3033fc8b3ef080a9b039b248"
Jul 28 05:08:55.668751 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:08:55.668735    2566 scope.go:117] "RemoveContainer" containerID="917ef30d9857eca03e3b3736517e14a6b07b3a4f8bc611684e7afca8132530d6"
Jul 28 05:08:55.669016 ip-10-0-129-187 kubenswrapper[2566]: E0728 05:08:55.668997    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"917ef30d9857eca03e3b3736517e14a6b07b3a4f8bc611684e7afca8132530d6\": container with ID starting with 917ef30d9857eca03e3b3736517e14a6b07b3a4f8bc611684e7afca8132530d6 not found: ID does not exist" containerID="917ef30d9857eca03e3b3736517e14a6b07b3a4f8bc611684e7afca8132530d6"
Jul 28 05:08:55.669092 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:08:55.669027    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"917ef30d9857eca03e3b3736517e14a6b07b3a4f8bc611684e7afca8132530d6"} err="failed to get container status \"917ef30d9857eca03e3b3736517e14a6b07b3a4f8bc611684e7afca8132530d6\": rpc error: code = NotFound desc = could not find container \"917ef30d9857eca03e3b3736517e14a6b07b3a4f8bc611684e7afca8132530d6\": container with ID starting with 917ef30d9857eca03e3b3736517e14a6b07b3a4f8bc611684e7afca8132530d6 not found: ID does not exist"
Jul 28 05:08:55.669092 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:08:55.669050    2566 scope.go:117] "RemoveContainer" containerID="158eace5807a18d93358a39a7c0d9f24c0e7ba4d3033fc8b3ef080a9b039b248"
Jul 28 05:08:55.669310 ip-10-0-129-187 kubenswrapper[2566]: E0728 05:08:55.669288    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"158eace5807a18d93358a39a7c0d9f24c0e7ba4d3033fc8b3ef080a9b039b248\": container with ID starting with 158eace5807a18d93358a39a7c0d9f24c0e7ba4d3033fc8b3ef080a9b039b248 not found: ID does not exist" containerID="158eace5807a18d93358a39a7c0d9f24c0e7ba4d3033fc8b3ef080a9b039b248"
Jul 28 05:08:55.669416 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:08:55.669315    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"158eace5807a18d93358a39a7c0d9f24c0e7ba4d3033fc8b3ef080a9b039b248"} err="failed to get container status \"158eace5807a18d93358a39a7c0d9f24c0e7ba4d3033fc8b3ef080a9b039b248\": rpc error: code = NotFound desc = could not find container \"158eace5807a18d93358a39a7c0d9f24c0e7ba4d3033fc8b3ef080a9b039b248\": container with ID starting with 158eace5807a18d93358a39a7c0d9f24c0e7ba4d3033fc8b3ef080a9b039b248 not found: ID does not exist"
Jul 28 05:08:55.673186 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:08:55.673166    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx"]
Jul 28 05:08:55.676305 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:08:55.676286    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-dc1efb-openshift-default-66b79c4b5b-hpfsx"]
Jul 28 05:08:56.088400 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:08:56.088367    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e74aefee-4f89-462e-89b9-f98c6abbe624" path="/var/lib/kubelet/pods/e74aefee-4f89-462e-89b9-f98c6abbe624/volumes"
Jul 28 05:09:00.941922 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:00.941890    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9b749c-5fc6885d68-j77b9"]
Jul 28 05:09:00.942382 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:00.942124    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-9b749c-5fc6885d68-j77b9" podUID="725cf087-0eeb-43a8-80a0-a1fa3d8a011a" containerName="payload-processing" containerID="cri-o://b2e187c32f574be4e66fe2c3fc2e7f92e222affdc23e2019ac7a2479875c3edb" gracePeriod=30
Jul 28 05:09:00.949078 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:00.949051    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9b749c-78d7987948-4cz5k"]
Jul 28 05:09:00.949504 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:00.949472    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b749c-78d7987948-4cz5k" podUID="d0c1aa7f-20cb-457c-84d6-17f1ba08e019" containerName="payload-pre-processing" containerID="cri-o://92973813c19cae8a498bd553a5abb302f01ff34ec37273ea5c263edc60710d1e" gracePeriod=30
Jul 28 05:09:01.233911 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:01.233887    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9b749c-5fc6885d68-j77b9"
Jul 28 05:09:01.237013 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:01.236992    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b749c-78d7987948-4cz5k"
Jul 28 05:09:01.373355 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:01.373323    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/725cf087-0eeb-43a8-80a0-a1fa3d8a011a-plugins-config-volume\") pod \"725cf087-0eeb-43a8-80a0-a1fa3d8a011a\" (UID: \"725cf087-0eeb-43a8-80a0-a1fa3d8a011a\") "
Jul 28 05:09:01.373531 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:01.373394    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0c1aa7f-20cb-457c-84d6-17f1ba08e019-plugins-config-volume\") pod \"d0c1aa7f-20cb-457c-84d6-17f1ba08e019\" (UID: \"d0c1aa7f-20cb-457c-84d6-17f1ba08e019\") "
Jul 28 05:09:01.373531 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:01.373428    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcs4r\" (UniqueName: \"kubernetes.io/projected/d0c1aa7f-20cb-457c-84d6-17f1ba08e019-kube-api-access-kcs4r\") pod \"d0c1aa7f-20cb-457c-84d6-17f1ba08e019\" (UID: \"d0c1aa7f-20cb-457c-84d6-17f1ba08e019\") "
Jul 28 05:09:01.373531 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:01.373488    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6pz6s\" (UniqueName: \"kubernetes.io/projected/725cf087-0eeb-43a8-80a0-a1fa3d8a011a-kube-api-access-6pz6s\") pod \"725cf087-0eeb-43a8-80a0-a1fa3d8a011a\" (UID: \"725cf087-0eeb-43a8-80a0-a1fa3d8a011a\") "
Jul 28 05:09:01.373741 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:01.373712    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/725cf087-0eeb-43a8-80a0-a1fa3d8a011a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "725cf087-0eeb-43a8-80a0-a1fa3d8a011a" (UID: "725cf087-0eeb-43a8-80a0-a1fa3d8a011a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 05:09:01.373825 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:01.373800    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0c1aa7f-20cb-457c-84d6-17f1ba08e019-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d0c1aa7f-20cb-457c-84d6-17f1ba08e019" (UID: "d0c1aa7f-20cb-457c-84d6-17f1ba08e019"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 05:09:01.373901 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:01.373831    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/725cf087-0eeb-43a8-80a0-a1fa3d8a011a-plugins-config-volume\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:09:01.375467 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:01.375446    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c1aa7f-20cb-457c-84d6-17f1ba08e019-kube-api-access-kcs4r" (OuterVolumeSpecName: "kube-api-access-kcs4r") pod "d0c1aa7f-20cb-457c-84d6-17f1ba08e019" (UID: "d0c1aa7f-20cb-457c-84d6-17f1ba08e019"). InnerVolumeSpecName "kube-api-access-kcs4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 05:09:01.375673 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:01.375652    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/725cf087-0eeb-43a8-80a0-a1fa3d8a011a-kube-api-access-6pz6s" (OuterVolumeSpecName: "kube-api-access-6pz6s") pod "725cf087-0eeb-43a8-80a0-a1fa3d8a011a" (UID: "725cf087-0eeb-43a8-80a0-a1fa3d8a011a"). InnerVolumeSpecName "kube-api-access-6pz6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 05:09:01.474461 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:01.474395    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pz6s\" (UniqueName: \"kubernetes.io/projected/725cf087-0eeb-43a8-80a0-a1fa3d8a011a-kube-api-access-6pz6s\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:09:01.474461 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:01.474431    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0c1aa7f-20cb-457c-84d6-17f1ba08e019-plugins-config-volume\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:09:01.474461 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:01.474441    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcs4r\" (UniqueName: \"kubernetes.io/projected/d0c1aa7f-20cb-457c-84d6-17f1ba08e019-kube-api-access-kcs4r\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:09:01.686309 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:01.686279    2566 generic.go:356] "Generic (PLEG): container finished" podID="725cf087-0eeb-43a8-80a0-a1fa3d8a011a" containerID="b2e187c32f574be4e66fe2c3fc2e7f92e222affdc23e2019ac7a2479875c3edb" exitCode=0
Jul 28 05:09:01.686480 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:01.686323    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9b749c-5fc6885d68-j77b9"
Jul 28 05:09:01.686480 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:01.686373    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9b749c-5fc6885d68-j77b9" event={"ID":"725cf087-0eeb-43a8-80a0-a1fa3d8a011a","Type":"ContainerDied","Data":"b2e187c32f574be4e66fe2c3fc2e7f92e222affdc23e2019ac7a2479875c3edb"}
Jul 28 05:09:01.686480 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:01.686435    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9b749c-5fc6885d68-j77b9" event={"ID":"725cf087-0eeb-43a8-80a0-a1fa3d8a011a","Type":"ContainerDied","Data":"bab0c7df4127e00f970556a7db003a0c2dfe6f9b4d048cc4b45865b99063d0fd"}
Jul 28 05:09:01.686480 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:01.686464    2566 scope.go:117] "RemoveContainer" containerID="b2e187c32f574be4e66fe2c3fc2e7f92e222affdc23e2019ac7a2479875c3edb"
Jul 28 05:09:01.687861 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:01.687817    2566 generic.go:356] "Generic (PLEG): container finished" podID="d0c1aa7f-20cb-457c-84d6-17f1ba08e019" containerID="92973813c19cae8a498bd553a5abb302f01ff34ec37273ea5c263edc60710d1e" exitCode=0
Jul 28 05:09:01.687974 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:01.687874    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b749c-78d7987948-4cz5k"
Jul 28 05:09:01.687974 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:01.687912    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b749c-78d7987948-4cz5k" event={"ID":"d0c1aa7f-20cb-457c-84d6-17f1ba08e019","Type":"ContainerDied","Data":"92973813c19cae8a498bd553a5abb302f01ff34ec37273ea5c263edc60710d1e"}
Jul 28 05:09:01.687974 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:01.687947    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9b749c-78d7987948-4cz5k" event={"ID":"d0c1aa7f-20cb-457c-84d6-17f1ba08e019","Type":"ContainerDied","Data":"a7d1105dddb65dbfa4b9247466579cdefc60b766a0304bfa22ef12b163926885"}
Jul 28 05:09:01.696886 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:01.696860    2566 scope.go:117] "RemoveContainer" containerID="b2e187c32f574be4e66fe2c3fc2e7f92e222affdc23e2019ac7a2479875c3edb"
Jul 28 05:09:01.697156 ip-10-0-129-187 kubenswrapper[2566]: E0728 05:09:01.697135    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2e187c32f574be4e66fe2c3fc2e7f92e222affdc23e2019ac7a2479875c3edb\": container with ID starting with b2e187c32f574be4e66fe2c3fc2e7f92e222affdc23e2019ac7a2479875c3edb not found: ID does not exist" containerID="b2e187c32f574be4e66fe2c3fc2e7f92e222affdc23e2019ac7a2479875c3edb"
Jul 28 05:09:01.697232 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:01.697166    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2e187c32f574be4e66fe2c3fc2e7f92e222affdc23e2019ac7a2479875c3edb"} err="failed to get container status \"b2e187c32f574be4e66fe2c3fc2e7f92e222affdc23e2019ac7a2479875c3edb\": rpc error: code = NotFound desc = could not find container \"b2e187c32f574be4e66fe2c3fc2e7f92e222affdc23e2019ac7a2479875c3edb\": container with ID starting with b2e187c32f574be4e66fe2c3fc2e7f92e222affdc23e2019ac7a2479875c3edb not found: ID does not exist"
Jul 28 05:09:01.697232 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:01.697191    2566 scope.go:117] "RemoveContainer" containerID="92973813c19cae8a498bd553a5abb302f01ff34ec37273ea5c263edc60710d1e"
Jul 28 05:09:01.705007 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:01.704990    2566 scope.go:117] "RemoveContainer" containerID="92973813c19cae8a498bd553a5abb302f01ff34ec37273ea5c263edc60710d1e"
Jul 28 05:09:01.705261 ip-10-0-129-187 kubenswrapper[2566]: E0728 05:09:01.705242    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92973813c19cae8a498bd553a5abb302f01ff34ec37273ea5c263edc60710d1e\": container with ID starting with 92973813c19cae8a498bd553a5abb302f01ff34ec37273ea5c263edc60710d1e not found: ID does not exist" containerID="92973813c19cae8a498bd553a5abb302f01ff34ec37273ea5c263edc60710d1e"
Jul 28 05:09:01.705324 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:01.705270    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92973813c19cae8a498bd553a5abb302f01ff34ec37273ea5c263edc60710d1e"} err="failed to get container status \"92973813c19cae8a498bd553a5abb302f01ff34ec37273ea5c263edc60710d1e\": rpc error: code = NotFound desc = could not find container \"92973813c19cae8a498bd553a5abb302f01ff34ec37273ea5c263edc60710d1e\": container with ID starting with 92973813c19cae8a498bd553a5abb302f01ff34ec37273ea5c263edc60710d1e not found: ID does not exist"
Jul 28 05:09:01.708386 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:01.708366    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9b749c-5fc6885d68-j77b9"]
Jul 28 05:09:01.711568 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:01.711548    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9b749c-5fc6885d68-j77b9"]
Jul 28 05:09:01.720824 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:01.720804    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9b749c-78d7987948-4cz5k"]
Jul 28 05:09:01.724631 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:01.724582    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9b749c-78d7987948-4cz5k"]
Jul 28 05:09:02.089237 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:02.089196    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="725cf087-0eeb-43a8-80a0-a1fa3d8a011a" path="/var/lib/kubelet/pods/725cf087-0eeb-43a8-80a0-a1fa3d8a011a/volumes"
Jul 28 05:09:02.089755 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:02.089734    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0c1aa7f-20cb-457c-84d6-17f1ba08e019" path="/var/lib/kubelet/pods/d0c1aa7f-20cb-457c-84d6-17f1ba08e019/volumes"
Jul 28 05:09:22.812956 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:22.812924    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-1badbb-5fd5858c-sjvg9"]
Jul 28 05:09:22.813679 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:22.813155    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1badbb-5fd5858c-sjvg9" podUID="c9bb777e-25c1-4454-b326-96ef0a1941d4" containerName="maas-api" containerID="cri-o://8e2d67b88f3f110ad5d488a59aacff341c94ae4fe3bc8e402cc8b2d67658aa62" gracePeriod=30
Jul 28 05:09:22.870580 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:22.870492    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-1badbb-7ff76ff76b-mklzc"]
Jul 28 05:09:22.870771 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:22.870726    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-1badbb-7ff76ff76b-mklzc" podUID="907caf08-9b81-40eb-937f-777750801bfe" containerName="payload-processing" containerID="cri-o://ea3d5d30451c5155b32a3cbf0d0a5e0e806bef194116ca0b220917c43f616d93" gracePeriod=30
Jul 28 05:09:22.879430 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:22.878430    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-1badbb-595f44df4d-mm2k8"]
Jul 28 05:09:22.879430 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:22.878636    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1badbb-595f44df4d-mm2k8" podUID="f2ddf640-7e9d-4af8-8702-9d50954a2fe8" containerName="payload-pre-processing" containerID="cri-o://d05aa4f6e7689160c429d49dcc3a892ee89484ca70d75997ad803cdec5c1cef1" gracePeriod=30
Jul 28 05:09:23.183987 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.183952    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-1badbb-7ff76ff76b-mklzc"
Jul 28 05:09:23.187124 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.187102    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1badbb-5fd5858c-sjvg9"
Jul 28 05:09:23.192713 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.192692    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1badbb-595f44df4d-mm2k8"
Jul 28 05:09:23.268072 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.268043    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9bb777e-25c1-4454-b326-96ef0a1941d4-openshift-service-ca\") pod \"c9bb777e-25c1-4454-b326-96ef0a1941d4\" (UID: \"c9bb777e-25c1-4454-b326-96ef0a1941d4\") "
Jul 28 05:09:23.268072 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.268075    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-grt7q\" (UniqueName: \"kubernetes.io/projected/c9bb777e-25c1-4454-b326-96ef0a1941d4-kube-api-access-grt7q\") pod \"c9bb777e-25c1-4454-b326-96ef0a1941d4\" (UID: \"c9bb777e-25c1-4454-b326-96ef0a1941d4\") "
Jul 28 05:09:23.268313 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.268117    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/907caf08-9b81-40eb-937f-777750801bfe-plugins-config-volume\") pod \"907caf08-9b81-40eb-937f-777750801bfe\" (UID: \"907caf08-9b81-40eb-937f-777750801bfe\") "
Jul 28 05:09:23.268313 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.268149    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2ddf640-7e9d-4af8-8702-9d50954a2fe8-plugins-config-volume\") pod \"f2ddf640-7e9d-4af8-8702-9d50954a2fe8\" (UID: \"f2ddf640-7e9d-4af8-8702-9d50954a2fe8\") "
Jul 28 05:09:23.268313 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.268175    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9bb777e-25c1-4454-b326-96ef0a1941d4-maas-api-tls\") pod \"c9bb777e-25c1-4454-b326-96ef0a1941d4\" (UID: \"c9bb777e-25c1-4454-b326-96ef0a1941d4\") "
Jul 28 05:09:23.268510 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.268370    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8pk25\" (UniqueName: \"kubernetes.io/projected/f2ddf640-7e9d-4af8-8702-9d50954a2fe8-kube-api-access-8pk25\") pod \"f2ddf640-7e9d-4af8-8702-9d50954a2fe8\" (UID: \"f2ddf640-7e9d-4af8-8702-9d50954a2fe8\") "
Jul 28 05:09:23.268510 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.268466    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vd82k\" (UniqueName: \"kubernetes.io/projected/907caf08-9b81-40eb-937f-777750801bfe-kube-api-access-vd82k\") pod \"907caf08-9b81-40eb-937f-777750801bfe\" (UID: \"907caf08-9b81-40eb-937f-777750801bfe\") "
Jul 28 05:09:23.268510 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.268481    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9bb777e-25c1-4454-b326-96ef0a1941d4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c9bb777e-25c1-4454-b326-96ef0a1941d4" (UID: "c9bb777e-25c1-4454-b326-96ef0a1941d4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 05:09:23.268510 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.268501    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/907caf08-9b81-40eb-937f-777750801bfe-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "907caf08-9b81-40eb-937f-777750801bfe" (UID: "907caf08-9b81-40eb-937f-777750801bfe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 05:09:23.268820 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.268795    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2ddf640-7e9d-4af8-8702-9d50954a2fe8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f2ddf640-7e9d-4af8-8702-9d50954a2fe8" (UID: "f2ddf640-7e9d-4af8-8702-9d50954a2fe8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 05:09:23.268924 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.268890    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9bb777e-25c1-4454-b326-96ef0a1941d4-openshift-service-ca\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:09:23.268924 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.268904    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/907caf08-9b81-40eb-937f-777750801bfe-plugins-config-volume\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:09:23.270358 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.270333    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9bb777e-25c1-4454-b326-96ef0a1941d4-kube-api-access-grt7q" (OuterVolumeSpecName: "kube-api-access-grt7q") pod "c9bb777e-25c1-4454-b326-96ef0a1941d4" (UID: "c9bb777e-25c1-4454-b326-96ef0a1941d4"). InnerVolumeSpecName "kube-api-access-grt7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 05:09:23.270571 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.270549    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/907caf08-9b81-40eb-937f-777750801bfe-kube-api-access-vd82k" (OuterVolumeSpecName: "kube-api-access-vd82k") pod "907caf08-9b81-40eb-937f-777750801bfe" (UID: "907caf08-9b81-40eb-937f-777750801bfe"). InnerVolumeSpecName "kube-api-access-vd82k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 05:09:23.270833 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.270812    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9bb777e-25c1-4454-b326-96ef0a1941d4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c9bb777e-25c1-4454-b326-96ef0a1941d4" (UID: "c9bb777e-25c1-4454-b326-96ef0a1941d4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 05:09:23.270833 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.270822    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2ddf640-7e9d-4af8-8702-9d50954a2fe8-kube-api-access-8pk25" (OuterVolumeSpecName: "kube-api-access-8pk25") pod "f2ddf640-7e9d-4af8-8702-9d50954a2fe8" (UID: "f2ddf640-7e9d-4af8-8702-9d50954a2fe8"). InnerVolumeSpecName "kube-api-access-8pk25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 05:09:23.369666 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.369574    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pk25\" (UniqueName: \"kubernetes.io/projected/f2ddf640-7e9d-4af8-8702-9d50954a2fe8-kube-api-access-8pk25\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:09:23.369666 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.369614    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd82k\" (UniqueName: \"kubernetes.io/projected/907caf08-9b81-40eb-937f-777750801bfe-kube-api-access-vd82k\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:09:23.369666 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.369624    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grt7q\" (UniqueName: \"kubernetes.io/projected/c9bb777e-25c1-4454-b326-96ef0a1941d4-kube-api-access-grt7q\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:09:23.369666 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.369636    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2ddf640-7e9d-4af8-8702-9d50954a2fe8-plugins-config-volume\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:09:23.369666 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.369646    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c9bb777e-25c1-4454-b326-96ef0a1941d4-maas-api-tls\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:09:23.829194 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.829155    2566 generic.go:356] "Generic (PLEG): container finished" podID="c9bb777e-25c1-4454-b326-96ef0a1941d4" containerID="8e2d67b88f3f110ad5d488a59aacff341c94ae4fe3bc8e402cc8b2d67658aa62" exitCode=0
Jul 28 05:09:23.829662 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.829211    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1badbb-5fd5858c-sjvg9"
Jul 28 05:09:23.829662 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.829252    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1badbb-5fd5858c-sjvg9" event={"ID":"c9bb777e-25c1-4454-b326-96ef0a1941d4","Type":"ContainerDied","Data":"8e2d67b88f3f110ad5d488a59aacff341c94ae4fe3bc8e402cc8b2d67658aa62"}
Jul 28 05:09:23.829662 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.829365    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-1badbb-5fd5858c-sjvg9" event={"ID":"c9bb777e-25c1-4454-b326-96ef0a1941d4","Type":"ContainerDied","Data":"05478c885b5065902d692ad781ce03f0bca84619546c114e8a5008a7484f5bb1"}
Jul 28 05:09:23.829662 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.829400    2566 scope.go:117] "RemoveContainer" containerID="8e2d67b88f3f110ad5d488a59aacff341c94ae4fe3bc8e402cc8b2d67658aa62"
Jul 28 05:09:23.830995 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.830614    2566 generic.go:356] "Generic (PLEG): container finished" podID="f2ddf640-7e9d-4af8-8702-9d50954a2fe8" containerID="d05aa4f6e7689160c429d49dcc3a892ee89484ca70d75997ad803cdec5c1cef1" exitCode=0
Jul 28 05:09:23.830995 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.830662    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1badbb-595f44df4d-mm2k8"
Jul 28 05:09:23.830995 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.830713    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1badbb-595f44df4d-mm2k8" event={"ID":"f2ddf640-7e9d-4af8-8702-9d50954a2fe8","Type":"ContainerDied","Data":"d05aa4f6e7689160c429d49dcc3a892ee89484ca70d75997ad803cdec5c1cef1"}
Jul 28 05:09:23.830995 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.830765    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-1badbb-595f44df4d-mm2k8" event={"ID":"f2ddf640-7e9d-4af8-8702-9d50954a2fe8","Type":"ContainerDied","Data":"60b60028695ac923744032bc57daf4a5766ea98ae402ce6a78e2c9cc2e7eb371"}
Jul 28 05:09:23.834473 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.832786    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-1badbb-7ff76ff76b-mklzc"
Jul 28 05:09:23.834473 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.832909    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-1badbb-7ff76ff76b-mklzc" event={"ID":"907caf08-9b81-40eb-937f-777750801bfe","Type":"ContainerDied","Data":"ea3d5d30451c5155b32a3cbf0d0a5e0e806bef194116ca0b220917c43f616d93"}
Jul 28 05:09:23.834473 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.832655    2566 generic.go:356] "Generic (PLEG): container finished" podID="907caf08-9b81-40eb-937f-777750801bfe" containerID="ea3d5d30451c5155b32a3cbf0d0a5e0e806bef194116ca0b220917c43f616d93" exitCode=0
Jul 28 05:09:23.834473 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.834419    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-1badbb-7ff76ff76b-mklzc" event={"ID":"907caf08-9b81-40eb-937f-777750801bfe","Type":"ContainerDied","Data":"9e07166e7b0505e49c2ddddb741b5330aa2e881f5ad09efcbfc02bbb2990dd85"}
Jul 28 05:09:23.842752 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.842736    2566 scope.go:117] "RemoveContainer" containerID="8e2d67b88f3f110ad5d488a59aacff341c94ae4fe3bc8e402cc8b2d67658aa62"
Jul 28 05:09:23.843037 ip-10-0-129-187 kubenswrapper[2566]: E0728 05:09:23.843017    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e2d67b88f3f110ad5d488a59aacff341c94ae4fe3bc8e402cc8b2d67658aa62\": container with ID starting with 8e2d67b88f3f110ad5d488a59aacff341c94ae4fe3bc8e402cc8b2d67658aa62 not found: ID does not exist" containerID="8e2d67b88f3f110ad5d488a59aacff341c94ae4fe3bc8e402cc8b2d67658aa62"
Jul 28 05:09:23.843111 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.843046    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e2d67b88f3f110ad5d488a59aacff341c94ae4fe3bc8e402cc8b2d67658aa62"} err="failed to get container status \"8e2d67b88f3f110ad5d488a59aacff341c94ae4fe3bc8e402cc8b2d67658aa62\": rpc error: code = NotFound desc = could not find container \"8e2d67b88f3f110ad5d488a59aacff341c94ae4fe3bc8e402cc8b2d67658aa62\": container with ID starting with 8e2d67b88f3f110ad5d488a59aacff341c94ae4fe3bc8e402cc8b2d67658aa62 not found: ID does not exist"
Jul 28 05:09:23.843111 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.843063    2566 scope.go:117] "RemoveContainer" containerID="d05aa4f6e7689160c429d49dcc3a892ee89484ca70d75997ad803cdec5c1cef1"
Jul 28 05:09:23.851063 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.851027    2566 scope.go:117] "RemoveContainer" containerID="d05aa4f6e7689160c429d49dcc3a892ee89484ca70d75997ad803cdec5c1cef1"
Jul 28 05:09:23.851319 ip-10-0-129-187 kubenswrapper[2566]: E0728 05:09:23.851297    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d05aa4f6e7689160c429d49dcc3a892ee89484ca70d75997ad803cdec5c1cef1\": container with ID starting with d05aa4f6e7689160c429d49dcc3a892ee89484ca70d75997ad803cdec5c1cef1 not found: ID does not exist" containerID="d05aa4f6e7689160c429d49dcc3a892ee89484ca70d75997ad803cdec5c1cef1"
Jul 28 05:09:23.851373 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.851330    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d05aa4f6e7689160c429d49dcc3a892ee89484ca70d75997ad803cdec5c1cef1"} err="failed to get container status \"d05aa4f6e7689160c429d49dcc3a892ee89484ca70d75997ad803cdec5c1cef1\": rpc error: code = NotFound desc = could not find container \"d05aa4f6e7689160c429d49dcc3a892ee89484ca70d75997ad803cdec5c1cef1\": container with ID starting with d05aa4f6e7689160c429d49dcc3a892ee89484ca70d75997ad803cdec5c1cef1 not found: ID does not exist"
Jul 28 05:09:23.851373 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.851352    2566 scope.go:117] "RemoveContainer" containerID="ea3d5d30451c5155b32a3cbf0d0a5e0e806bef194116ca0b220917c43f616d93"
Jul 28 05:09:23.853950 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.853930    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-1badbb-5fd5858c-sjvg9"]
Jul 28 05:09:23.855767 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.855747    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-1badbb-5fd5858c-sjvg9"]
Jul 28 05:09:23.859166 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.859149    2566 scope.go:117] "RemoveContainer" containerID="ea3d5d30451c5155b32a3cbf0d0a5e0e806bef194116ca0b220917c43f616d93"
Jul 28 05:09:23.859401 ip-10-0-129-187 kubenswrapper[2566]: E0728 05:09:23.859383    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea3d5d30451c5155b32a3cbf0d0a5e0e806bef194116ca0b220917c43f616d93\": container with ID starting with ea3d5d30451c5155b32a3cbf0d0a5e0e806bef194116ca0b220917c43f616d93 not found: ID does not exist" containerID="ea3d5d30451c5155b32a3cbf0d0a5e0e806bef194116ca0b220917c43f616d93"
Jul 28 05:09:23.859439 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.859409    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea3d5d30451c5155b32a3cbf0d0a5e0e806bef194116ca0b220917c43f616d93"} err="failed to get container status \"ea3d5d30451c5155b32a3cbf0d0a5e0e806bef194116ca0b220917c43f616d93\": rpc error: code = NotFound desc = could not find container \"ea3d5d30451c5155b32a3cbf0d0a5e0e806bef194116ca0b220917c43f616d93\": container with ID starting with ea3d5d30451c5155b32a3cbf0d0a5e0e806bef194116ca0b220917c43f616d93 not found: ID does not exist"
Jul 28 05:09:23.864551 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.864532    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-1badbb-595f44df4d-mm2k8"]
Jul 28 05:09:23.868277 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.868258    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-1badbb-595f44df4d-mm2k8"]
Jul 28 05:09:23.879130 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.879112    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-1badbb-7ff76ff76b-mklzc"]
Jul 28 05:09:23.884660 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:23.884638    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-1badbb-7ff76ff76b-mklzc"]
Jul 28 05:09:24.088370 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:24.088286    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="907caf08-9b81-40eb-937f-777750801bfe" path="/var/lib/kubelet/pods/907caf08-9b81-40eb-937f-777750801bfe/volumes"
Jul 28 05:09:24.088762 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:24.088747    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9bb777e-25c1-4454-b326-96ef0a1941d4" path="/var/lib/kubelet/pods/c9bb777e-25c1-4454-b326-96ef0a1941d4/volumes"
Jul 28 05:09:24.089157 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:24.089144    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2ddf640-7e9d-4af8-8702-9d50954a2fe8" path="/var/lib/kubelet/pods/f2ddf640-7e9d-4af8-8702-9d50954a2fe8/volumes"
Jul 28 05:09:33.672896 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:33.672866    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz"]
Jul 28 05:09:33.673343 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:33.673061    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz" podUID="abc8eda5-4cd4-4570-b474-c525a74eb3c6" containerName="istio-proxy" containerID="cri-o://2b8afaa917f287b96c09e2645a0cd180f42c0781c27ceded912494bde94ccb4e" gracePeriod=30
Jul 28 05:09:38.915034 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:38.915005    2566 generic.go:356] "Generic (PLEG): container finished" podID="abc8eda5-4cd4-4570-b474-c525a74eb3c6" containerID="2b8afaa917f287b96c09e2645a0cd180f42c0781c27ceded912494bde94ccb4e" exitCode=0
Jul 28 05:09:38.915377 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:38.915060    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz" event={"ID":"abc8eda5-4cd4-4570-b474-c525a74eb3c6","Type":"ContainerDied","Data":"2b8afaa917f287b96c09e2645a0cd180f42c0781c27ceded912494bde94ccb4e"}
Jul 28 05:09:38.915377 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:38.915113    2566 scope.go:117] "RemoveContainer" containerID="fb4f2119ef0b5ff6db30de8277e23a49fe72bdd60d16dc12deeb8688ecadf4dd"
Jul 28 05:09:38.937128 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:38.937075    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz"
Jul 28 05:09:39.021259 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:39.021232    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abc8eda5-4cd4-4570-b474-c525a74eb3c6-istio-data\") pod \"abc8eda5-4cd4-4570-b474-c525a74eb3c6\" (UID: \"abc8eda5-4cd4-4570-b474-c525a74eb3c6\") "
Jul 28 05:09:39.021443 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:39.021279    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abc8eda5-4cd4-4570-b474-c525a74eb3c6-istio-token\") pod \"abc8eda5-4cd4-4570-b474-c525a74eb3c6\" (UID: \"abc8eda5-4cd4-4570-b474-c525a74eb3c6\") "
Jul 28 05:09:39.021443 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:39.021303    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abc8eda5-4cd4-4570-b474-c525a74eb3c6-istio-podinfo\") pod \"abc8eda5-4cd4-4570-b474-c525a74eb3c6\" (UID: \"abc8eda5-4cd4-4570-b474-c525a74eb3c6\") "
Jul 28 05:09:39.021443 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:39.021330    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc8eda5-4cd4-4570-b474-c525a74eb3c6-workload-socket\") pod \"abc8eda5-4cd4-4570-b474-c525a74eb3c6\" (UID: \"abc8eda5-4cd4-4570-b474-c525a74eb3c6\") "
Jul 28 05:09:39.021443 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:39.021385    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abc8eda5-4cd4-4570-b474-c525a74eb3c6-workload-certs\") pod \"abc8eda5-4cd4-4570-b474-c525a74eb3c6\" (UID: \"abc8eda5-4cd4-4570-b474-c525a74eb3c6\") "
Jul 28 05:09:39.021443 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:39.021410    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9h9hp\" (UniqueName: \"kubernetes.io/projected/abc8eda5-4cd4-4570-b474-c525a74eb3c6-kube-api-access-9h9hp\") pod \"abc8eda5-4cd4-4570-b474-c525a74eb3c6\" (UID: \"abc8eda5-4cd4-4570-b474-c525a74eb3c6\") "
Jul 28 05:09:39.021443 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:39.021443    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abc8eda5-4cd4-4570-b474-c525a74eb3c6-istiod-ca-cert\") pod \"abc8eda5-4cd4-4570-b474-c525a74eb3c6\" (UID: \"abc8eda5-4cd4-4570-b474-c525a74eb3c6\") "
Jul 28 05:09:39.021788 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:39.021480    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc8eda5-4cd4-4570-b474-c525a74eb3c6-credential-socket\") pod \"abc8eda5-4cd4-4570-b474-c525a74eb3c6\" (UID: \"abc8eda5-4cd4-4570-b474-c525a74eb3c6\") "
Jul 28 05:09:39.021788 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:39.021511    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abc8eda5-4cd4-4570-b474-c525a74eb3c6-istio-envoy\") pod \"abc8eda5-4cd4-4570-b474-c525a74eb3c6\" (UID: \"abc8eda5-4cd4-4570-b474-c525a74eb3c6\") "
Jul 28 05:09:39.021788 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:39.021683    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc8eda5-4cd4-4570-b474-c525a74eb3c6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "abc8eda5-4cd4-4570-b474-c525a74eb3c6" (UID: "abc8eda5-4cd4-4570-b474-c525a74eb3c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 05:09:39.021978 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:39.021905    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc8eda5-4cd4-4570-b474-c525a74eb3c6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "abc8eda5-4cd4-4570-b474-c525a74eb3c6" (UID: "abc8eda5-4cd4-4570-b474-c525a74eb3c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 05:09:39.021978 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:39.021692    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc8eda5-4cd4-4570-b474-c525a74eb3c6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "abc8eda5-4cd4-4570-b474-c525a74eb3c6" (UID: "abc8eda5-4cd4-4570-b474-c525a74eb3c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 05:09:39.021978 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:39.021956    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc8eda5-4cd4-4570-b474-c525a74eb3c6-workload-socket\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:09:39.022141 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:39.021972    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abc8eda5-4cd4-4570-b474-c525a74eb3c6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "abc8eda5-4cd4-4570-b474-c525a74eb3c6" (UID: "abc8eda5-4cd4-4570-b474-c525a74eb3c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 05:09:39.022362 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:39.022332    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc8eda5-4cd4-4570-b474-c525a74eb3c6-istio-data" (OuterVolumeSpecName: "istio-data") pod "abc8eda5-4cd4-4570-b474-c525a74eb3c6" (UID: "abc8eda5-4cd4-4570-b474-c525a74eb3c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 05:09:39.023550 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:39.023530    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc8eda5-4cd4-4570-b474-c525a74eb3c6-istio-token" (OuterVolumeSpecName: "istio-token") pod "abc8eda5-4cd4-4570-b474-c525a74eb3c6" (UID: "abc8eda5-4cd4-4570-b474-c525a74eb3c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 05:09:39.023638 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:39.023560    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/abc8eda5-4cd4-4570-b474-c525a74eb3c6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "abc8eda5-4cd4-4570-b474-c525a74eb3c6" (UID: "abc8eda5-4cd4-4570-b474-c525a74eb3c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 28 05:09:39.023638 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:39.023629    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abc8eda5-4cd4-4570-b474-c525a74eb3c6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "abc8eda5-4cd4-4570-b474-c525a74eb3c6" (UID: "abc8eda5-4cd4-4570-b474-c525a74eb3c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 05:09:39.023778 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:39.023756    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abc8eda5-4cd4-4570-b474-c525a74eb3c6-kube-api-access-9h9hp" (OuterVolumeSpecName: "kube-api-access-9h9hp") pod "abc8eda5-4cd4-4570-b474-c525a74eb3c6" (UID: "abc8eda5-4cd4-4570-b474-c525a74eb3c6"). InnerVolumeSpecName "kube-api-access-9h9hp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 05:09:39.122906 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:39.122872    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/abc8eda5-4cd4-4570-b474-c525a74eb3c6-istio-podinfo\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:09:39.122906 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:39.122903    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/abc8eda5-4cd4-4570-b474-c525a74eb3c6-workload-certs\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:09:39.122906 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:39.122913    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h9hp\" (UniqueName: \"kubernetes.io/projected/abc8eda5-4cd4-4570-b474-c525a74eb3c6-kube-api-access-9h9hp\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:09:39.123160 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:39.122921    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/abc8eda5-4cd4-4570-b474-c525a74eb3c6-istiod-ca-cert\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:09:39.123160 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:39.122931    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/abc8eda5-4cd4-4570-b474-c525a74eb3c6-credential-socket\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:09:39.123160 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:39.122939    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/abc8eda5-4cd4-4570-b474-c525a74eb3c6-istio-envoy\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:09:39.123160 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:39.122947    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/abc8eda5-4cd4-4570-b474-c525a74eb3c6-istio-data\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:09:39.123160 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:39.122954    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/abc8eda5-4cd4-4570-b474-c525a74eb3c6-istio-token\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:09:39.921358 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:39.921332    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz"
Jul 28 05:09:39.921863 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:39.921389    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz" event={"ID":"abc8eda5-4cd4-4570-b474-c525a74eb3c6","Type":"ContainerDied","Data":"e3c22d9efe46ae3d1c25b0651b2982ec5ac09130f952342321454b811620904c"}
Jul 28 05:09:39.921863 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:39.921463    2566 scope.go:117] "RemoveContainer" containerID="2b8afaa917f287b96c09e2645a0cd180f42c0781c27ceded912494bde94ccb4e"
Jul 28 05:09:39.944601 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:39.944576    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz"]
Jul 28 05:09:39.949414 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:39.949394    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-1badbb-openshift-default-6448f7dfb8-lnggz"]
Jul 28 05:09:40.088718 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:40.088687    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abc8eda5-4cd4-4570-b474-c525a74eb3c6" path="/var/lib/kubelet/pods/abc8eda5-4cd4-4570-b474-c525a74eb3c6/volumes"
Jul 28 05:09:44.564776 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:44.564743    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-92c557-858bb566bf-t5xks"]
Jul 28 05:09:44.565264 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:44.564972    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-92c557-858bb566bf-t5xks" podUID="aea644ae-603c-4f83-aded-079a83d3441f" containerName="maas-api" containerID="cri-o://67726e08c494dc30b011db96a4e33f12a00aa29cf091f3565924b81b4e2a50fc" gracePeriod=30
Jul 28 05:09:44.622399 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:44.622357    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-92c557-6f4d7cf958-mxbgg"]
Jul 28 05:09:44.622651 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:44.622615    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-92c557-6f4d7cf958-mxbgg" podUID="d17d74fd-cd53-4292-a641-f8ca86d4b0d5" containerName="payload-processing" containerID="cri-o://6e77dd842d7d08b72cc8101d8ca79379a047da7e326703e489a860ba21907613" gracePeriod=30
Jul 28 05:09:44.898515 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:44.898490    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-92c557-858bb566bf-t5xks"
Jul 28 05:09:44.901414 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:44.901389    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-92c557-6f4d7cf958-mxbgg"
Jul 28 05:09:44.949782 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:44.949746    2566 generic.go:356] "Generic (PLEG): container finished" podID="d17d74fd-cd53-4292-a641-f8ca86d4b0d5" containerID="6e77dd842d7d08b72cc8101d8ca79379a047da7e326703e489a860ba21907613" exitCode=0
Jul 28 05:09:44.949990 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:44.949795    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-92c557-6f4d7cf958-mxbgg"
Jul 28 05:09:44.949990 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:44.949818    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-92c557-6f4d7cf958-mxbgg" event={"ID":"d17d74fd-cd53-4292-a641-f8ca86d4b0d5","Type":"ContainerDied","Data":"6e77dd842d7d08b72cc8101d8ca79379a047da7e326703e489a860ba21907613"}
Jul 28 05:09:44.949990 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:44.949912    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-92c557-6f4d7cf958-mxbgg" event={"ID":"d17d74fd-cd53-4292-a641-f8ca86d4b0d5","Type":"ContainerDied","Data":"c9aae3d95d3a91e0a9f756a38085ba3dcf40c859a7aabc6c87ce55e6bd6260e2"}
Jul 28 05:09:44.949990 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:44.949962    2566 scope.go:117] "RemoveContainer" containerID="6e77dd842d7d08b72cc8101d8ca79379a047da7e326703e489a860ba21907613"
Jul 28 05:09:44.951471 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:44.951445    2566 generic.go:356] "Generic (PLEG): container finished" podID="aea644ae-603c-4f83-aded-079a83d3441f" containerID="67726e08c494dc30b011db96a4e33f12a00aa29cf091f3565924b81b4e2a50fc" exitCode=0
Jul 28 05:09:44.951583 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:44.951489    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-92c557-858bb566bf-t5xks"
Jul 28 05:09:44.951583 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:44.951518    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-92c557-858bb566bf-t5xks" event={"ID":"aea644ae-603c-4f83-aded-079a83d3441f","Type":"ContainerDied","Data":"67726e08c494dc30b011db96a4e33f12a00aa29cf091f3565924b81b4e2a50fc"}
Jul 28 05:09:44.951583 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:44.951559    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-92c557-858bb566bf-t5xks" event={"ID":"aea644ae-603c-4f83-aded-079a83d3441f","Type":"ContainerDied","Data":"987f15e3c509408e2cd2152f74a83c8a84b90b9ee5707e1e5fbceaede7f159dd"}
Jul 28 05:09:44.960179 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:44.960155    2566 scope.go:117] "RemoveContainer" containerID="6e77dd842d7d08b72cc8101d8ca79379a047da7e326703e489a860ba21907613"
Jul 28 05:09:44.960507 ip-10-0-129-187 kubenswrapper[2566]: E0728 05:09:44.960478    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e77dd842d7d08b72cc8101d8ca79379a047da7e326703e489a860ba21907613\": container with ID starting with 6e77dd842d7d08b72cc8101d8ca79379a047da7e326703e489a860ba21907613 not found: ID does not exist" containerID="6e77dd842d7d08b72cc8101d8ca79379a047da7e326703e489a860ba21907613"
Jul 28 05:09:44.960607 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:44.960519    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e77dd842d7d08b72cc8101d8ca79379a047da7e326703e489a860ba21907613"} err="failed to get container status \"6e77dd842d7d08b72cc8101d8ca79379a047da7e326703e489a860ba21907613\": rpc error: code = NotFound desc = could not find container \"6e77dd842d7d08b72cc8101d8ca79379a047da7e326703e489a860ba21907613\": container with ID starting with 6e77dd842d7d08b72cc8101d8ca79379a047da7e326703e489a860ba21907613 not found: ID does not exist"
Jul 28 05:09:44.960607 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:44.960545    2566 scope.go:117] "RemoveContainer" containerID="67726e08c494dc30b011db96a4e33f12a00aa29cf091f3565924b81b4e2a50fc"
Jul 28 05:09:44.967816 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:44.967801    2566 scope.go:117] "RemoveContainer" containerID="67726e08c494dc30b011db96a4e33f12a00aa29cf091f3565924b81b4e2a50fc"
Jul 28 05:09:44.968085 ip-10-0-129-187 kubenswrapper[2566]: E0728 05:09:44.968069    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67726e08c494dc30b011db96a4e33f12a00aa29cf091f3565924b81b4e2a50fc\": container with ID starting with 67726e08c494dc30b011db96a4e33f12a00aa29cf091f3565924b81b4e2a50fc not found: ID does not exist" containerID="67726e08c494dc30b011db96a4e33f12a00aa29cf091f3565924b81b4e2a50fc"
Jul 28 05:09:44.968146 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:44.968093    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67726e08c494dc30b011db96a4e33f12a00aa29cf091f3565924b81b4e2a50fc"} err="failed to get container status \"67726e08c494dc30b011db96a4e33f12a00aa29cf091f3565924b81b4e2a50fc\": rpc error: code = NotFound desc = could not find container \"67726e08c494dc30b011db96a4e33f12a00aa29cf091f3565924b81b4e2a50fc\": container with ID starting with 67726e08c494dc30b011db96a4e33f12a00aa29cf091f3565924b81b4e2a50fc not found: ID does not exist"
Jul 28 05:09:45.082832 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:45.082747    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aea644ae-603c-4f83-aded-079a83d3441f-openshift-service-ca\") pod \"aea644ae-603c-4f83-aded-079a83d3441f\" (UID: \"aea644ae-603c-4f83-aded-079a83d3441f\") "
Jul 28 05:09:45.082832 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:45.082850    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n65g2\" (UniqueName: \"kubernetes.io/projected/aea644ae-603c-4f83-aded-079a83d3441f-kube-api-access-n65g2\") pod \"aea644ae-603c-4f83-aded-079a83d3441f\" (UID: \"aea644ae-603c-4f83-aded-079a83d3441f\") "
Jul 28 05:09:45.083086 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:45.082884    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jtsz6\" (UniqueName: \"kubernetes.io/projected/d17d74fd-cd53-4292-a641-f8ca86d4b0d5-kube-api-access-jtsz6\") pod \"d17d74fd-cd53-4292-a641-f8ca86d4b0d5\" (UID: \"d17d74fd-cd53-4292-a641-f8ca86d4b0d5\") "
Jul 28 05:09:45.083086 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:45.082945    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aea644ae-603c-4f83-aded-079a83d3441f-maas-api-tls\") pod \"aea644ae-603c-4f83-aded-079a83d3441f\" (UID: \"aea644ae-603c-4f83-aded-079a83d3441f\") "
Jul 28 05:09:45.083086 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:45.082978    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d17d74fd-cd53-4292-a641-f8ca86d4b0d5-plugins-config-volume\") pod \"d17d74fd-cd53-4292-a641-f8ca86d4b0d5\" (UID: \"d17d74fd-cd53-4292-a641-f8ca86d4b0d5\") "
Jul 28 05:09:45.083258 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:45.083211    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aea644ae-603c-4f83-aded-079a83d3441f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "aea644ae-603c-4f83-aded-079a83d3441f" (UID: "aea644ae-603c-4f83-aded-079a83d3441f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 05:09:45.083418 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:45.083393    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aea644ae-603c-4f83-aded-079a83d3441f-openshift-service-ca\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:09:45.083573 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:45.083452    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d17d74fd-cd53-4292-a641-f8ca86d4b0d5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d17d74fd-cd53-4292-a641-f8ca86d4b0d5" (UID: "d17d74fd-cd53-4292-a641-f8ca86d4b0d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 05:09:45.085037 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:45.085014    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aea644ae-603c-4f83-aded-079a83d3441f-kube-api-access-n65g2" (OuterVolumeSpecName: "kube-api-access-n65g2") pod "aea644ae-603c-4f83-aded-079a83d3441f" (UID: "aea644ae-603c-4f83-aded-079a83d3441f"). InnerVolumeSpecName "kube-api-access-n65g2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 05:09:45.085037 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:45.085020    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aea644ae-603c-4f83-aded-079a83d3441f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "aea644ae-603c-4f83-aded-079a83d3441f" (UID: "aea644ae-603c-4f83-aded-079a83d3441f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 05:09:45.085149 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:45.085025    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d17d74fd-cd53-4292-a641-f8ca86d4b0d5-kube-api-access-jtsz6" (OuterVolumeSpecName: "kube-api-access-jtsz6") pod "d17d74fd-cd53-4292-a641-f8ca86d4b0d5" (UID: "d17d74fd-cd53-4292-a641-f8ca86d4b0d5"). InnerVolumeSpecName "kube-api-access-jtsz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 05:09:45.184344 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:45.184309    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n65g2\" (UniqueName: \"kubernetes.io/projected/aea644ae-603c-4f83-aded-079a83d3441f-kube-api-access-n65g2\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:09:45.184344 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:45.184338    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtsz6\" (UniqueName: \"kubernetes.io/projected/d17d74fd-cd53-4292-a641-f8ca86d4b0d5-kube-api-access-jtsz6\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:09:45.184344 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:45.184347    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aea644ae-603c-4f83-aded-079a83d3441f-maas-api-tls\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:09:45.184344 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:45.184356    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d17d74fd-cd53-4292-a641-f8ca86d4b0d5-plugins-config-volume\") on node \"ip-10-0-129-187.ec2.internal\" DevicePath \"\""
Jul 28 05:09:45.273262 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:45.273235    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-92c557-6f4d7cf958-mxbgg"]
Jul 28 05:09:45.275031 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:45.275011    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-92c557-6f4d7cf958-mxbgg"]
Jul 28 05:09:45.284238 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:45.284217    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-92c557-858bb566bf-t5xks"]
Jul 28 05:09:45.287786 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:45.287766    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-92c557-858bb566bf-t5xks"]
Jul 28 05:09:46.089079 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:46.089046    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aea644ae-603c-4f83-aded-079a83d3441f" path="/var/lib/kubelet/pods/aea644ae-603c-4f83-aded-079a83d3441f/volumes"
Jul 28 05:09:46.089468 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:09:46.089445    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d17d74fd-cd53-4292-a641-f8ca86d4b0d5" path="/var/lib/kubelet/pods/d17d74fd-cd53-4292-a641-f8ca86d4b0d5/volumes"
Jul 28 05:10:02.220993 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:02.220911    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7847888f58-4nr6t_15ba97a6-0f40-4e47-a51a-2e23a16d5175/manager/0.log"
Jul 28 05:10:02.855276 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:02.855244    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-69d65f9dd8-lgq4p_3eb1967e-94c7-450b-82d7-076c52b46af1/copy-manifests/0.log"
Jul 28 05:10:02.861988 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:02.861957    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-69d65f9dd8-lgq4p_3eb1967e-94c7-450b-82d7-076c52b46af1/manager/0.log"
Jul 28 05:10:05.349207 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:05.349176    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-chdx9_58a62ded-ca6b-41f7-9e33-16f1702ecdf7/manager/0.log"
Jul 28 05:10:06.604400 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:06.604359    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58d69b594d-jqh6r_baf9fb3c-ce25-4d8c-a06d-ce4a4aa666c5/kube-auth-proxy/0.log"
Jul 28 05:10:06.990280 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:06.990178    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-kncv4_fc1e8560-ba15-4aeb-8b1c-760913971991/payload-pre-processing/0.log"
Jul 28 05:10:07.245526 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:07.245433    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-95c8f4d49-5ncng_5c7f7f74-e140-4d88-bc93-c029ae1c18a7/router/0.log"
Jul 28 05:10:07.608381 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:07.608349    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-ngc2f_d2c9cd70-f242-4eb1-8fa3-5ed2a9e9ad70/main/0.log"
Jul 28 05:10:07.846489 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:07.846462    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-qqcd4_dc87f1c9-5ac5-41ff-b653-701ce5e0b760/main/0.log"
Jul 28 05:10:08.100336 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:08.100307    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-tdmg6_173039f3-efce-40f1-b977-4a0c921c0bfb/main/0.log"
Jul 28 05:10:08.221873 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:08.221826    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-c4h7c_cd436090-b85d-4d4a-af2e-dbacf3e3c3e4/main/0.log"
Jul 28 05:10:14.240151 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.240113    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5wtbr/must-gather-lm7xv"]
Jul 28 05:10:14.240527 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.240467    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abc8eda5-4cd4-4570-b474-c525a74eb3c6" containerName="istio-proxy"
Jul 28 05:10:14.240527 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.240478    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="abc8eda5-4cd4-4570-b474-c525a74eb3c6" containerName="istio-proxy"
Jul 28 05:10:14.240527 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.240486    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0c1aa7f-20cb-457c-84d6-17f1ba08e019" containerName="payload-pre-processing"
Jul 28 05:10:14.240527 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.240492    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0c1aa7f-20cb-457c-84d6-17f1ba08e019" containerName="payload-pre-processing"
Jul 28 05:10:14.240527 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.240501    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e74aefee-4f89-462e-89b9-f98c6abbe624" containerName="istio-proxy"
Jul 28 05:10:14.240527 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.240506    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e74aefee-4f89-462e-89b9-f98c6abbe624" containerName="istio-proxy"
Jul 28 05:10:14.240527 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.240514    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4d45902-7d49-4bdf-81f4-06fc25c43587" containerName="payload-processing"
Jul 28 05:10:14.240527 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.240520    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4d45902-7d49-4bdf-81f4-06fc25c43587" containerName="payload-processing"
Jul 28 05:10:14.240527 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.240530    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="725cf087-0eeb-43a8-80a0-a1fa3d8a011a" containerName="payload-processing"
Jul 28 05:10:14.240881 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.240535    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="725cf087-0eeb-43a8-80a0-a1fa3d8a011a" containerName="payload-processing"
Jul 28 05:10:14.240881 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.240543    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d17d74fd-cd53-4292-a641-f8ca86d4b0d5" containerName="payload-processing"
Jul 28 05:10:14.240881 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.240548    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d17d74fd-cd53-4292-a641-f8ca86d4b0d5" containerName="payload-processing"
Jul 28 05:10:14.240881 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.240557    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abc8eda5-4cd4-4570-b474-c525a74eb3c6" containerName="istio-proxy"
Jul 28 05:10:14.240881 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.240562    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="abc8eda5-4cd4-4570-b474-c525a74eb3c6" containerName="istio-proxy"
Jul 28 05:10:14.240881 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.240572    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="914ffbb7-7664-4519-94d4-6758257c5863" containerName="istio-proxy"
Jul 28 05:10:14.240881 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.240577    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="914ffbb7-7664-4519-94d4-6758257c5863" containerName="istio-proxy"
Jul 28 05:10:14.240881 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.240583    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9bb777e-25c1-4454-b326-96ef0a1941d4" containerName="maas-api"
Jul 28 05:10:14.240881 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.240588    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9bb777e-25c1-4454-b326-96ef0a1941d4" containerName="maas-api"
Jul 28 05:10:14.240881 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.240593    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="907caf08-9b81-40eb-937f-777750801bfe" containerName="payload-processing"
Jul 28 05:10:14.240881 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.240598    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="907caf08-9b81-40eb-937f-777750801bfe" containerName="payload-processing"
Jul 28 05:10:14.240881 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.240603    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aea644ae-603c-4f83-aded-079a83d3441f" containerName="maas-api"
Jul 28 05:10:14.240881 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.240608    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="aea644ae-603c-4f83-aded-079a83d3441f" containerName="maas-api"
Jul 28 05:10:14.240881 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.240617    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e702cc51-e3a8-4c8e-84b7-d2774888e09c" containerName="payload-processing"
Jul 28 05:10:14.240881 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.240623    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e702cc51-e3a8-4c8e-84b7-d2774888e09c" containerName="payload-processing"
Jul 28 05:10:14.240881 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.240633    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2ddf640-7e9d-4af8-8702-9d50954a2fe8" containerName="payload-pre-processing"
Jul 28 05:10:14.240881 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.240637    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2ddf640-7e9d-4af8-8702-9d50954a2fe8" containerName="payload-pre-processing"
Jul 28 05:10:14.240881 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.240643    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="914ffbb7-7664-4519-94d4-6758257c5863" containerName="istio-proxy"
Jul 28 05:10:14.240881 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.240648    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="914ffbb7-7664-4519-94d4-6758257c5863" containerName="istio-proxy"
Jul 28 05:10:14.240881 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.240702    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="abc8eda5-4cd4-4570-b474-c525a74eb3c6" containerName="istio-proxy"
Jul 28 05:10:14.240881 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.240708    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="aea644ae-603c-4f83-aded-079a83d3441f" containerName="maas-api"
Jul 28 05:10:14.240881 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.240715    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d17d74fd-cd53-4292-a641-f8ca86d4b0d5" containerName="payload-processing"
Jul 28 05:10:14.240881 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.240721    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0c1aa7f-20cb-457c-84d6-17f1ba08e019" containerName="payload-pre-processing"
Jul 28 05:10:14.240881 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.240728    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e74aefee-4f89-462e-89b9-f98c6abbe624" containerName="istio-proxy"
Jul 28 05:10:14.240881 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.240734    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9bb777e-25c1-4454-b326-96ef0a1941d4" containerName="maas-api"
Jul 28 05:10:14.240881 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.240740    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2ddf640-7e9d-4af8-8702-9d50954a2fe8" containerName="payload-pre-processing"
Jul 28 05:10:14.240881 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.240744    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4d45902-7d49-4bdf-81f4-06fc25c43587" containerName="payload-processing"
Jul 28 05:10:14.240881 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.240750    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="914ffbb7-7664-4519-94d4-6758257c5863" containerName="istio-proxy"
Jul 28 05:10:14.240881 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.240756    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e74aefee-4f89-462e-89b9-f98c6abbe624" containerName="istio-proxy"
Jul 28 05:10:14.240881 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.240761    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="725cf087-0eeb-43a8-80a0-a1fa3d8a011a" containerName="payload-processing"
Jul 28 05:10:14.240881 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.240766    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="914ffbb7-7664-4519-94d4-6758257c5863" containerName="istio-proxy"
Jul 28 05:10:14.240881 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.240774    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="907caf08-9b81-40eb-937f-777750801bfe" containerName="payload-processing"
Jul 28 05:10:14.240881 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.240779    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e702cc51-e3a8-4c8e-84b7-d2774888e09c" containerName="payload-processing"
Jul 28 05:10:14.240881 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.240785    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="abc8eda5-4cd4-4570-b474-c525a74eb3c6" containerName="istio-proxy"
Jul 28 05:10:14.240881 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.240864    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e74aefee-4f89-462e-89b9-f98c6abbe624" containerName="istio-proxy"
Jul 28 05:10:14.240881 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.240871    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e74aefee-4f89-462e-89b9-f98c6abbe624" containerName="istio-proxy"
Jul 28 05:10:14.244469 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.244451    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5wtbr/must-gather-lm7xv"
Jul 28 05:10:14.247269 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.247249    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5wtbr\"/\"openshift-service-ca.crt\""
Jul 28 05:10:14.247491 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.247476    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5wtbr\"/\"kube-root-ca.crt\""
Jul 28 05:10:14.251795 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.251776    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5wtbr/must-gather-lm7xv"]
Jul 28 05:10:14.347042 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.347012    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b188ef41-d701-4333-a9b5-cbe841056efd-must-gather-output\") pod \"must-gather-lm7xv\" (UID: \"b188ef41-d701-4333-a9b5-cbe841056efd\") " pod="openshift-must-gather-5wtbr/must-gather-lm7xv"
Jul 28 05:10:14.347200 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.347056    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m8fw\" (UniqueName: \"kubernetes.io/projected/b188ef41-d701-4333-a9b5-cbe841056efd-kube-api-access-8m8fw\") pod \"must-gather-lm7xv\" (UID: \"b188ef41-d701-4333-a9b5-cbe841056efd\") " pod="openshift-must-gather-5wtbr/must-gather-lm7xv"
Jul 28 05:10:14.447926 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.447889    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b188ef41-d701-4333-a9b5-cbe841056efd-must-gather-output\") pod \"must-gather-lm7xv\" (UID: \"b188ef41-d701-4333-a9b5-cbe841056efd\") " pod="openshift-must-gather-5wtbr/must-gather-lm7xv"
Jul 28 05:10:14.448085 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.447944    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m8fw\" (UniqueName: \"kubernetes.io/projected/b188ef41-d701-4333-a9b5-cbe841056efd-kube-api-access-8m8fw\") pod \"must-gather-lm7xv\" (UID: \"b188ef41-d701-4333-a9b5-cbe841056efd\") " pod="openshift-must-gather-5wtbr/must-gather-lm7xv"
Jul 28 05:10:14.448270 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.448232    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b188ef41-d701-4333-a9b5-cbe841056efd-must-gather-output\") pod \"must-gather-lm7xv\" (UID: \"b188ef41-d701-4333-a9b5-cbe841056efd\") " pod="openshift-must-gather-5wtbr/must-gather-lm7xv"
Jul 28 05:10:14.456022 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.455998    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m8fw\" (UniqueName: \"kubernetes.io/projected/b188ef41-d701-4333-a9b5-cbe841056efd-kube-api-access-8m8fw\") pod \"must-gather-lm7xv\" (UID: \"b188ef41-d701-4333-a9b5-cbe841056efd\") " pod="openshift-must-gather-5wtbr/must-gather-lm7xv"
Jul 28 05:10:14.554434 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.554400    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5wtbr/must-gather-lm7xv"
Jul 28 05:10:14.687151 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.687127    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5wtbr/must-gather-lm7xv"]
Jul 28 05:10:14.689522 ip-10-0-129-187 kubenswrapper[2566]: W0728 05:10:14.689496    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb188ef41_d701_4333_a9b5_cbe841056efd.slice/crio-f2df885951e3309ceb9f4735809dd4895cbb1b07dce9efd87ce93bd5d281eb83 WatchSource:0}: Error finding container f2df885951e3309ceb9f4735809dd4895cbb1b07dce9efd87ce93bd5d281eb83: Status 404 returned error can't find the container with id f2df885951e3309ceb9f4735809dd4895cbb1b07dce9efd87ce93bd5d281eb83
Jul 28 05:10:14.691370 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:14.691355    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 05:10:15.098698 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:15.098655    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5wtbr/must-gather-lm7xv" event={"ID":"b188ef41-d701-4333-a9b5-cbe841056efd","Type":"ContainerStarted","Data":"f2df885951e3309ceb9f4735809dd4895cbb1b07dce9efd87ce93bd5d281eb83"}
Jul 28 05:10:16.108208 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:16.108161    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5wtbr/must-gather-lm7xv" event={"ID":"b188ef41-d701-4333-a9b5-cbe841056efd","Type":"ContainerStarted","Data":"4a144ec434f0975026f70498917d21f9a22fb412a6edc4c329551650ac3ee7d2"}
Jul 28 05:10:16.108698 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:16.108223    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5wtbr/must-gather-lm7xv" event={"ID":"b188ef41-d701-4333-a9b5-cbe841056efd","Type":"ContainerStarted","Data":"871d924ec9c148939002ac1d0afd88f4fcef65d0417efac1aba1f18d32ec1f45"}
Jul 28 05:10:16.126524 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:16.126476    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5wtbr/must-gather-lm7xv" podStartSLOduration=1.472297149 podStartE2EDuration="2.126458893s" totalImagesPullingTime="654.161744ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:10:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:10:16.125115495 +0000 UTC m=+3516.515291518" watchObservedRunningTime="2026-07-28 05:10:16.126458893 +0000 UTC m=+3516.516634905"
Jul 28 05:10:16.950409 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:16.950379    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-z2gn6_accccd5b-603e-49d5-b9fc-b328dcb7e959/global-pull-secret-syncer/0.log"
Jul 28 05:10:17.070754 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:17.070721    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vfg8j_2c4c0738-fb7c-4610-aab6-7cd762ebaf83/konnectivity-agent/0.log"
Jul 28 05:10:17.101143 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:17.101110    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-187.ec2.internal_c9650e72cdb5110eb610920a5f32b652/haproxy/0.log"
Jul 28 05:10:22.279249 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:22.279145    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-chdx9_58a62ded-ca6b-41f7-9e33-16f1702ecdf7/manager/0.log"
Jul 28 05:10:24.458787 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:24.458698    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-mfd2q_edec740c-2287-4ba9-ae6b-0f9da7320a48/cluster-monitoring-operator/0.log"
Jul 28 05:10:24.558118 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:24.558092    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-67c5866966-hssnd_819152a5-89ff-4974-bfb7-7025060775b1/metrics-server/0.log"
Jul 28 05:10:24.616481 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:24.616448    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4sdhs_71b82494-fe22-4a00-9e59-44d7d84fb7a5/node-exporter/0.log"
Jul 28 05:10:24.637614 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:24.637586    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4sdhs_71b82494-fe22-4a00-9e59-44d7d84fb7a5/kube-rbac-proxy/0.log"
Jul 28 05:10:24.668678 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:24.668650    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4sdhs_71b82494-fe22-4a00-9e59-44d7d84fb7a5/init-textfile/0.log"
Jul 28 05:10:24.946997 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:24.946972    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_834f4106-f9fe-43f0-99e6-79ed625abab3/prometheus/0.log"
Jul 28 05:10:24.970632 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:24.970612    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_834f4106-f9fe-43f0-99e6-79ed625abab3/config-reloader/0.log"
Jul 28 05:10:24.993396 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:24.993348    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_834f4106-f9fe-43f0-99e6-79ed625abab3/thanos-sidecar/0.log"
Jul 28 05:10:25.029132 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:25.029101    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_834f4106-f9fe-43f0-99e6-79ed625abab3/kube-rbac-proxy-web/0.log"
Jul 28 05:10:25.049430 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:25.049401    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_834f4106-f9fe-43f0-99e6-79ed625abab3/kube-rbac-proxy/0.log"
Jul 28 05:10:25.071829 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:25.071722    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_834f4106-f9fe-43f0-99e6-79ed625abab3/kube-rbac-proxy-thanos/0.log"
Jul 28 05:10:25.093667 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:25.093623    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_834f4106-f9fe-43f0-99e6-79ed625abab3/init-config-reloader/0.log"
Jul 28 05:10:25.135725 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:25.135622    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-65jws_fa80dba1-35cc-4655-9984-d85d596e27d8/prometheus-operator/0.log"
Jul 28 05:10:25.157029 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:25.156977    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-65jws_fa80dba1-35cc-4655-9984-d85d596e27d8/kube-rbac-proxy/0.log"
Jul 28 05:10:26.602256 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:26.602229    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-jxd2j_c58f3317-e522-41a3-848f-906af440a23b/networking-console-plugin/0.log"
Jul 28 05:10:27.216463 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:27.216434    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4glpr_bfd2d3d8-f287-42d8-83c7-f82185c96735/console-operator/2.log"
Jul 28 05:10:27.222713 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:27.222678    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-4glpr_bfd2d3d8-f287-42d8-83c7-f82185c96735/console-operator/3.log"
Jul 28 05:10:27.246078 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:27.246044    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5wtbr/perf-node-gather-daemonset-x26cx"]
Jul 28 05:10:27.251480 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:27.251456    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5wtbr/perf-node-gather-daemonset-x26cx"
Jul 28 05:10:27.257612 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:27.257573    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5wtbr\"/\"default-dockercfg-8sxlx\""
Jul 28 05:10:27.257777 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:27.257575    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5wtbr/perf-node-gather-daemonset-x26cx"]
Jul 28 05:10:27.302152 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:27.302088    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6a0d7258-1285-4771-ab5f-1f05e666424d-proc\") pod \"perf-node-gather-daemonset-x26cx\" (UID: \"6a0d7258-1285-4771-ab5f-1f05e666424d\") " pod="openshift-must-gather-5wtbr/perf-node-gather-daemonset-x26cx"
Jul 28 05:10:27.302356 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:27.302164    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6a0d7258-1285-4771-ab5f-1f05e666424d-podres\") pod \"perf-node-gather-daemonset-x26cx\" (UID: \"6a0d7258-1285-4771-ab5f-1f05e666424d\") " pod="openshift-must-gather-5wtbr/perf-node-gather-daemonset-x26cx"
Jul 28 05:10:27.302356 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:27.302222    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6a0d7258-1285-4771-ab5f-1f05e666424d-lib-modules\") pod \"perf-node-gather-daemonset-x26cx\" (UID: \"6a0d7258-1285-4771-ab5f-1f05e666424d\") " pod="openshift-must-gather-5wtbr/perf-node-gather-daemonset-x26cx"
Jul 28 05:10:27.302543 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:27.302357    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25cjd\" (UniqueName: \"kubernetes.io/projected/6a0d7258-1285-4771-ab5f-1f05e666424d-kube-api-access-25cjd\") pod \"perf-node-gather-daemonset-x26cx\" (UID: \"6a0d7258-1285-4771-ab5f-1f05e666424d\") " pod="openshift-must-gather-5wtbr/perf-node-gather-daemonset-x26cx"
Jul 28 05:10:27.302543 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:27.302469    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6a0d7258-1285-4771-ab5f-1f05e666424d-sys\") pod \"perf-node-gather-daemonset-x26cx\" (UID: \"6a0d7258-1285-4771-ab5f-1f05e666424d\") " pod="openshift-must-gather-5wtbr/perf-node-gather-daemonset-x26cx"
Jul 28 05:10:27.403916 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:27.403880    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6a0d7258-1285-4771-ab5f-1f05e666424d-proc\") pod \"perf-node-gather-daemonset-x26cx\" (UID: \"6a0d7258-1285-4771-ab5f-1f05e666424d\") " pod="openshift-must-gather-5wtbr/perf-node-gather-daemonset-x26cx"
Jul 28 05:10:27.404092 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:27.403928    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6a0d7258-1285-4771-ab5f-1f05e666424d-podres\") pod \"perf-node-gather-daemonset-x26cx\" (UID: \"6a0d7258-1285-4771-ab5f-1f05e666424d\") " pod="openshift-must-gather-5wtbr/perf-node-gather-daemonset-x26cx"
Jul 28 05:10:27.404092 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:27.404040    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6a0d7258-1285-4771-ab5f-1f05e666424d-proc\") pod \"perf-node-gather-daemonset-x26cx\" (UID: \"6a0d7258-1285-4771-ab5f-1f05e666424d\") " pod="openshift-must-gather-5wtbr/perf-node-gather-daemonset-x26cx"
Jul 28 05:10:27.404182 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:27.404109    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6a0d7258-1285-4771-ab5f-1f05e666424d-podres\") pod \"perf-node-gather-daemonset-x26cx\" (UID: \"6a0d7258-1285-4771-ab5f-1f05e666424d\") " pod="openshift-must-gather-5wtbr/perf-node-gather-daemonset-x26cx"
Jul 28 05:10:27.404182 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:27.404116    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6a0d7258-1285-4771-ab5f-1f05e666424d-lib-modules\") pod \"perf-node-gather-daemonset-x26cx\" (UID: \"6a0d7258-1285-4771-ab5f-1f05e666424d\") " pod="openshift-must-gather-5wtbr/perf-node-gather-daemonset-x26cx"
Jul 28 05:10:27.404256 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:27.404185    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6a0d7258-1285-4771-ab5f-1f05e666424d-lib-modules\") pod \"perf-node-gather-daemonset-x26cx\" (UID: \"6a0d7258-1285-4771-ab5f-1f05e666424d\") " pod="openshift-must-gather-5wtbr/perf-node-gather-daemonset-x26cx"
Jul 28 05:10:27.404256 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:27.404195    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25cjd\" (UniqueName: \"kubernetes.io/projected/6a0d7258-1285-4771-ab5f-1f05e666424d-kube-api-access-25cjd\") pod \"perf-node-gather-daemonset-x26cx\" (UID: \"6a0d7258-1285-4771-ab5f-1f05e666424d\") " pod="openshift-must-gather-5wtbr/perf-node-gather-daemonset-x26cx"
Jul 28 05:10:27.404337 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:27.404322    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6a0d7258-1285-4771-ab5f-1f05e666424d-sys\") pod \"perf-node-gather-daemonset-x26cx\" (UID: \"6a0d7258-1285-4771-ab5f-1f05e666424d\") " pod="openshift-must-gather-5wtbr/perf-node-gather-daemonset-x26cx"
Jul 28 05:10:27.404422 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:27.404407    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6a0d7258-1285-4771-ab5f-1f05e666424d-sys\") pod \"perf-node-gather-daemonset-x26cx\" (UID: \"6a0d7258-1285-4771-ab5f-1f05e666424d\") " pod="openshift-must-gather-5wtbr/perf-node-gather-daemonset-x26cx"
Jul 28 05:10:27.412730 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:27.412703    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25cjd\" (UniqueName: \"kubernetes.io/projected/6a0d7258-1285-4771-ab5f-1f05e666424d-kube-api-access-25cjd\") pod \"perf-node-gather-daemonset-x26cx\" (UID: \"6a0d7258-1285-4771-ab5f-1f05e666424d\") " pod="openshift-must-gather-5wtbr/perf-node-gather-daemonset-x26cx"
Jul 28 05:10:27.568920 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:27.568888    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5wtbr/perf-node-gather-daemonset-x26cx"
Jul 28 05:10:27.795101 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:27.795059    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5wtbr/perf-node-gather-daemonset-x26cx"]
Jul 28 05:10:27.798890 ip-10-0-129-187 kubenswrapper[2566]: W0728 05:10:27.798831    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6a0d7258_1285_4771_ab5f_1f05e666424d.slice/crio-64d05cd9798713ac3057409878c3620699c1e20ea63dae90ec97af0ba2dedeff WatchSource:0}: Error finding container 64d05cd9798713ac3057409878c3620699c1e20ea63dae90ec97af0ba2dedeff: Status 404 returned error can't find the container with id 64d05cd9798713ac3057409878c3620699c1e20ea63dae90ec97af0ba2dedeff
Jul 28 05:10:27.854003 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:27.853923    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7bd48fb7cf-wbkqp_c0a0e084-e243-4fd1-84d0-f429ae783778/console/0.log"
Jul 28 05:10:27.892269 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:27.892233    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-rs96k_e1817115-022a-447e-9431-62a0d5c2e4c6/download-server/0.log"
Jul 28 05:10:28.194272 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:28.194089    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5wtbr/perf-node-gather-daemonset-x26cx" event={"ID":"6a0d7258-1285-4771-ab5f-1f05e666424d","Type":"ContainerStarted","Data":"d19e3478bf4308e054f334944b8536d1d0d50ff6f22bfd030b395ab2356febdf"}
Jul 28 05:10:28.194272 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:28.194164    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5wtbr/perf-node-gather-daemonset-x26cx" event={"ID":"6a0d7258-1285-4771-ab5f-1f05e666424d","Type":"ContainerStarted","Data":"64d05cd9798713ac3057409878c3620699c1e20ea63dae90ec97af0ba2dedeff"}
Jul 28 05:10:28.194691 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:28.194619    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5wtbr/perf-node-gather-daemonset-x26cx"
Jul 28 05:10:28.212193 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:28.212134    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5wtbr/perf-node-gather-daemonset-x26cx" podStartSLOduration=1.212115373 podStartE2EDuration="1.212115373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 05:10:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:10:28.210987165 +0000 UTC m=+3528.601163188" watchObservedRunningTime="2026-07-28 05:10:28.212115373 +0000 UTC m=+3528.602291395"
Jul 28 05:10:29.532597 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:29.532564    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-m9zx6_a02a02e4-e2b1-45fd-aeda-12fa4e29a996/dns/0.log"
Jul 28 05:10:29.552338 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:29.552310    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-m9zx6_a02a02e4-e2b1-45fd-aeda-12fa4e29a996/kube-rbac-proxy/0.log"
Jul 28 05:10:29.647569 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:29.647534    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tlsvx_068d70ee-9efb-49a5-94cb-2df7591a4652/dns-node-resolver/0.log"
Jul 28 05:10:30.221350 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:30.221314    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-t6nfz_8cf586fe-2e9b-474b-8fa6-3ca6317e2b31/node-ca/0.log"
Jul 28 05:10:31.324455 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:31.324424    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58d69b594d-jqh6r_baf9fb3c-ce25-4d8c-a06d-ce4a4aa666c5/kube-auth-proxy/0.log"
Jul 28 05:10:31.430363 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:31.430329    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-kncv4_fc1e8560-ba15-4aeb-8b1c-760913971991/payload-pre-processing/0.log"
Jul 28 05:10:31.483142 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:31.483116    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-95c8f4d49-5ncng_5c7f7f74-e140-4d88-bc93-c029ae1c18a7/router/0.log"
Jul 28 05:10:32.311043 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:32.311011    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-z8gxf_9cb032cd-6003-4249-a65d-95d48fea8a02/serve-healthcheck-canary/0.log"
Jul 28 05:10:32.775167 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:32.775132    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-lmrmn_f77a6b03-d007-46e3-852b-4b262feca301/insights-operator/0.log"
Jul 28 05:10:32.777703 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:32.777680    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-lmrmn_f77a6b03-d007-46e3-852b-4b262feca301/insights-operator/1.log"
Jul 28 05:10:32.865285 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:32.865255    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k8fxt_83539dc8-61dc-4d3c-aff3-9ac4a543abd9/kube-rbac-proxy/0.log"
Jul 28 05:10:32.886456 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:32.886422    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k8fxt_83539dc8-61dc-4d3c-aff3-9ac4a543abd9/exporter/0.log"
Jul 28 05:10:32.907816 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:32.907794    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-k8fxt_83539dc8-61dc-4d3c-aff3-9ac4a543abd9/extractor/0.log"
Jul 28 05:10:34.214269 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:34.214235    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5wtbr/perf-node-gather-daemonset-x26cx"
Jul 28 05:10:35.411872 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:35.411827    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7847888f58-4nr6t_15ba97a6-0f40-4e47-a51a-2e23a16d5175/manager/0.log"
Jul 28 05:10:35.697720 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:35.697635    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-69d65f9dd8-lgq4p_3eb1967e-94c7-450b-82d7-076c52b46af1/manager/0.log"
Jul 28 05:10:35.719092 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:35.719055    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-69d65f9dd8-lgq4p_3eb1967e-94c7-450b-82d7-076c52b46af1/copy-manifests/0.log"
Jul 28 05:10:37.149033 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:37.148938    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7796f8d956-tv25r_3abde1d4-6845-4a1e-8ec4-3746b5c7d5fe/manager/0.log"
Jul 28 05:10:37.868089 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:37.868051    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58d69b594d-jqh6r_baf9fb3c-ce25-4d8c-a06d-ce4a4aa666c5/kube-auth-proxy/0.log"
Jul 28 05:10:37.950604 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:37.950573    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-kncv4_fc1e8560-ba15-4aeb-8b1c-760913971991/payload-pre-processing/0.log"
Jul 28 05:10:37.983548 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:37.983519    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-95c8f4d49-5ncng_5c7f7f74-e140-4d88-bc93-c029ae1c18a7/router/0.log"
Jul 28 05:10:39.491700 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:39.491633    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-58d69b594d-jqh6r_baf9fb3c-ce25-4d8c-a06d-ce4a4aa666c5/kube-auth-proxy/0.log"
Jul 28 05:10:39.576957 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:39.576919    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-kncv4_fc1e8560-ba15-4aeb-8b1c-760913971991/payload-pre-processing/0.log"
Jul 28 05:10:39.616322 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:39.616297    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-95c8f4d49-5ncng_5c7f7f74-e140-4d88-bc93-c029ae1c18a7/router/0.log"
Jul 28 05:10:42.569618 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:42.569582    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-d2lpq_3837b644-a03e-498c-9033-1ed2e1d894d0/kube-storage-version-migrator-operator/1.log"
Jul 28 05:10:42.570485 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:42.570466    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-d2lpq_3837b644-a03e-498c-9033-1ed2e1d894d0/kube-storage-version-migrator-operator/0.log"
Jul 28 05:10:43.714434 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:43.714407    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fk4hq_93788c64-611b-40ff-be6c-8dcea894c08d/kube-multus-additional-cni-plugins/0.log"
Jul 28 05:10:43.735220 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:43.735197    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fk4hq_93788c64-611b-40ff-be6c-8dcea894c08d/egress-router-binary-copy/0.log"
Jul 28 05:10:43.757004 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:43.756986    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fk4hq_93788c64-611b-40ff-be6c-8dcea894c08d/cni-plugins/0.log"
Jul 28 05:10:43.777088 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:43.777070    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fk4hq_93788c64-611b-40ff-be6c-8dcea894c08d/bond-cni-plugin/0.log"
Jul 28 05:10:43.797467 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:43.797450    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fk4hq_93788c64-611b-40ff-be6c-8dcea894c08d/routeoverride-cni/0.log"
Jul 28 05:10:43.817724 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:43.817703    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fk4hq_93788c64-611b-40ff-be6c-8dcea894c08d/whereabouts-cni-bincopy/0.log"
Jul 28 05:10:43.839041 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:43.839025    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-fk4hq_93788c64-611b-40ff-be6c-8dcea894c08d/whereabouts-cni/0.log"
Jul 28 05:10:44.031965 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:44.031938    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jxpf8_aaaa4008-a3fc-4b57-a3f7-eeb6bf9c9a67/kube-multus/0.log"
Jul 28 05:10:44.127312 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:44.127287    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-l6p44_e6107883-62a6-4a82-9e04-ed46057a6b0e/network-metrics-daemon/0.log"
Jul 28 05:10:44.147271 ip-10-0-129-187 kubenswrapper[2566]: I0728 05:10:44.147242    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-l6p44_e6107883-62a6-4a82-9e04-ed46057a6b0e/kube-rbac-proxy/0.log"