Sep 10 15:42:02.905658 ip-10-0-139-107 systemd[1]: Starting Kubernetes Kubelet... Sep 10 15:42:03.310949 ip-10-0-139-107 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 15:42:03.310949 ip-10-0-139-107 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 10 15:42:03.310949 ip-10-0-139-107 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 15:42:03.310949 ip-10-0-139-107 kubenswrapper[2583]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 10 15:42:03.310949 ip-10-0-139-107 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 10 15:42:03.311665 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.311007 2583 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 10 15:42:03.317180 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317158 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 15:42:03.317180 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317178 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 15:42:03.317180 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317182 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 15:42:03.317275 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317188 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 15:42:03.317275 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317191 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 15:42:03.317275 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317194 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 15:42:03.317275 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317197 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 15:42:03.317275 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317201 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 15:42:03.317275 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317204 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 15:42:03.317275 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317207 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 15:42:03.317275 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317210 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 15:42:03.317275 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317213 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 15:42:03.317275 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317215 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 15:42:03.317275 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317218 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 15:42:03.317275 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317222 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 15:42:03.317275 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317225 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 15:42:03.317275 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317227 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 15:42:03.317275 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317230 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 15:42:03.317275 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317232 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 15:42:03.317275 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317235 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 15:42:03.317275 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317238 2583 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 15:42:03.317275 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317240 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 15:42:03.317275 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317243 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 15:42:03.317778 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317246 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 15:42:03.317778 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317248 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 15:42:03.317778 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317251 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 15:42:03.317778 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317253 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 15:42:03.317778 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317256 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 15:42:03.317778 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317258 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 15:42:03.317778 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317261 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 15:42:03.317778 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317263 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 15:42:03.317778 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317266 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 15:42:03.317778 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317268 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 15:42:03.317778 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317271 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 15:42:03.317778 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317273 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 15:42:03.317778 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317276 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 15:42:03.317778 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317280 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 15:42:03.317778 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317284 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 15:42:03.317778 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317288 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 15:42:03.317778 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317291 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 15:42:03.317778 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317294 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 15:42:03.317778 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317297 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 15:42:03.318241 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317299 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 15:42:03.318241 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317302 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 15:42:03.318241 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317305 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 15:42:03.318241 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317308 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 15:42:03.318241 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317310 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 15:42:03.318241 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317313 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 15:42:03.318241 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317316 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 15:42:03.318241 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317318 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 15:42:03.318241 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317321 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 15:42:03.318241 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317324 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 15:42:03.318241 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317327 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 15:42:03.318241 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317330 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 15:42:03.318241 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317333 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 15:42:03.318241 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317335 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 15:42:03.318241 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317338 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 15:42:03.318241 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317341 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 15:42:03.318241 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317344 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 15:42:03.318241 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317346 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 15:42:03.318241 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317349 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 15:42:03.318241 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317352 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 15:42:03.318241 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317355 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 15:42:03.318763 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317357 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 15:42:03.318763 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317360 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 15:42:03.318763 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317363 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 15:42:03.318763 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317366 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 15:42:03.318763 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317368 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 15:42:03.318763 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317371 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 15:42:03.318763 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317374 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 15:42:03.318763 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317389 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 15:42:03.318763 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317392 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 15:42:03.318763 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317395 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 15:42:03.318763 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317398 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 15:42:03.318763 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317400 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 15:42:03.318763 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317403 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 15:42:03.318763 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317409 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 15:42:03.318763 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317413 2583 feature_gate.go:328] unrecognized feature gate: Example Sep 10 15:42:03.318763 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317417 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 15:42:03.318763 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317421 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 15:42:03.318763 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317424 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 15:42:03.319228 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317427 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 15:42:03.319228 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317430 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 15:42:03.319228 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317433 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 15:42:03.319228 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317435 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 15:42:03.319228 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317438 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 15:42:03.319228 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317440 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 15:42:03.319228 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317443 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 15:42:03.319228 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317447 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 15:42:03.319228 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317449 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 15:42:03.319228 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317452 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 15:42:03.319228 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317455 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 15:42:03.319228 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317458 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 15:42:03.319228 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317461 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 15:42:03.319228 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317464 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 15:42:03.319228 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317466 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 15:42:03.319228 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317468 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 15:42:03.319228 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317471 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 15:42:03.319228 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.317474 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 15:42:03.321021 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321007 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 15:42:03.321021 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321019 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 15:42:03.321021 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321022 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 15:42:03.321111 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321026 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 15:42:03.321111 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321029 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 15:42:03.321111 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321032 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 15:42:03.321111 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321034 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 15:42:03.321111 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321037 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 15:42:03.321111 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321039 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 15:42:03.321111 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321044 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 15:42:03.321111 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321048 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 15:42:03.321111 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321052 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 15:42:03.321111 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321054 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 15:42:03.321111 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321057 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 15:42:03.321111 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321060 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 15:42:03.321111 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321062 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 15:42:03.321111 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321065 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 15:42:03.321111 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321067 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 15:42:03.321111 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321070 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 15:42:03.321111 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321072 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 15:42:03.321111 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321075 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 15:42:03.321111 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321078 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 15:42:03.321606 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321080 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 15:42:03.321606 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321083 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 15:42:03.321606 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321085 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 15:42:03.321606 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321088 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 15:42:03.321606 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321092 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 15:42:03.321606 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321095 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 15:42:03.321606 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321098 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 15:42:03.321606 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321100 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 15:42:03.321606 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321103 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 15:42:03.321606 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321105 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 15:42:03.321606 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321108 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 15:42:03.321606 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321110 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 15:42:03.321606 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321113 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 15:42:03.321606 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321116 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 15:42:03.321606 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321119 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 15:42:03.321606 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321122 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 15:42:03.321606 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321124 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 15:42:03.321606 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321127 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 15:42:03.321606 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321130 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 15:42:03.322068 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321134 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 15:42:03.322068 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321138 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 15:42:03.322068 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321142 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 15:42:03.322068 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321145 2583 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 15:42:03.322068 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321148 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 15:42:03.322068 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321151 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 15:42:03.322068 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321154 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 15:42:03.322068 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321157 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 15:42:03.322068 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321159 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 15:42:03.322068 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321162 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 15:42:03.322068 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321165 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 15:42:03.322068 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321167 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 15:42:03.322068 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321169 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 15:42:03.322068 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321172 2583 feature_gate.go:328] unrecognized feature gate: Example Sep 10 15:42:03.322068 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321175 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 15:42:03.322068 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321178 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 15:42:03.322068 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321180 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 15:42:03.322068 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321183 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 15:42:03.322068 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321186 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 15:42:03.322068 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321188 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 15:42:03.322569 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321191 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 15:42:03.322569 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321194 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 15:42:03.322569 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321196 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 15:42:03.322569 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321199 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 15:42:03.322569 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321201 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 15:42:03.322569 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321204 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 15:42:03.322569 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321206 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 15:42:03.322569 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321209 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 15:42:03.322569 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321211 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 15:42:03.322569 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321214 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 15:42:03.322569 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321216 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 15:42:03.322569 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321219 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 15:42:03.322569 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321222 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 15:42:03.322569 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321224 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 15:42:03.322569 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321229 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 15:42:03.322569 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321231 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 15:42:03.322569 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321234 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 15:42:03.322569 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321236 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 15:42:03.322569 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321239 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 15:42:03.322569 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321241 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 15:42:03.323057 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321244 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 15:42:03.323057 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321246 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 15:42:03.323057 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321249 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 15:42:03.323057 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321251 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 15:42:03.323057 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321254 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 15:42:03.323057 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321256 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 15:42:03.323057 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321260 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 15:42:03.323057 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321263 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 15:42:03.323057 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321266 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 15:42:03.323057 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321268 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 15:42:03.323057 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321271 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 15:42:03.323057 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321275 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 15:42:03.323057 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321278 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 15:42:03.323057 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321280 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 15:42:03.323057 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321283 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 15:42:03.323057 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321285 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 15:42:03.323057 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321288 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 15:42:03.323057 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321291 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 15:42:03.323057 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321714 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 15:42:03.323057 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321719 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 15:42:03.323549 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321722 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 15:42:03.323549 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321725 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 15:42:03.323549 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321728 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 15:42:03.323549 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321731 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 15:42:03.323549 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321734 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 15:42:03.323549 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321736 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 15:42:03.323549 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321740 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 15:42:03.323549 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321743 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 15:42:03.323549 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321746 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 15:42:03.323549 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321748 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 15:42:03.323549 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321751 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 15:42:03.323549 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321754 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 15:42:03.323549 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321757 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 15:42:03.323549 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321759 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 15:42:03.323549 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321762 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 15:42:03.323549 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321765 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 15:42:03.323549 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321769 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 15:42:03.323549 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321772 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 15:42:03.323549 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321776 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 15:42:03.324025 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321780 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 15:42:03.324025 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321783 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 15:42:03.324025 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321786 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 15:42:03.324025 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321788 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 15:42:03.324025 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321791 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 15:42:03.324025 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321794 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 15:42:03.324025 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321796 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 15:42:03.324025 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321799 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 15:42:03.324025 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321802 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 15:42:03.324025 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321805 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 15:42:03.324025 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321807 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 15:42:03.324025 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321810 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 15:42:03.324025 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321812 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 15:42:03.324025 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321815 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 15:42:03.324025 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321817 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 15:42:03.324025 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321820 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 15:42:03.324025 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321823 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 15:42:03.324025 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321826 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 15:42:03.324025 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321828 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 15:42:03.324025 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321831 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 15:42:03.324528 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321834 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 15:42:03.324528 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321836 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 15:42:03.324528 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321839 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 15:42:03.324528 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321841 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 15:42:03.324528 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321844 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 15:42:03.324528 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321846 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 15:42:03.324528 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321849 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 15:42:03.324528 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321851 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 15:42:03.324528 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321854 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 15:42:03.324528 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321856 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 15:42:03.324528 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321859 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 15:42:03.324528 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321862 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 15:42:03.324528 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321865 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 15:42:03.324528 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321867 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 15:42:03.324528 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321870 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 15:42:03.324528 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321872 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 15:42:03.324528 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321875 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 15:42:03.324528 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321877 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 15:42:03.324528 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321881 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 15:42:03.324528 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321883 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 15:42:03.325127 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321886 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 15:42:03.325127 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321888 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 15:42:03.325127 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321892 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 15:42:03.325127 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321896 2583 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 15:42:03.325127 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321899 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 15:42:03.325127 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321901 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 15:42:03.325127 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321904 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 15:42:03.325127 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321907 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 15:42:03.325127 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321910 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 15:42:03.325127 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321912 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 15:42:03.325127 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321917 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 15:42:03.325127 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321920 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 15:42:03.325127 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321923 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 15:42:03.325127 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321926 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 15:42:03.325127 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321929 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 15:42:03.325127 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321932 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 15:42:03.325127 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321935 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 15:42:03.325127 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321937 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 15:42:03.325127 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321940 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 15:42:03.325127 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321943 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 15:42:03.325693 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321945 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 15:42:03.325693 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321948 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 15:42:03.325693 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321951 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 15:42:03.325693 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321953 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 15:42:03.325693 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321956 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 15:42:03.325693 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321959 2583 feature_gate.go:328] unrecognized feature gate: Example Sep 10 15:42:03.325693 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321961 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 15:42:03.325693 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321964 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 15:42:03.325693 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321967 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 15:42:03.325693 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321969 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 15:42:03.325693 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321972 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 15:42:03.325693 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321975 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 15:42:03.325693 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321978 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 15:42:03.325693 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321981 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 15:42:03.325693 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321983 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 15:42:03.325693 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321985 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 15:42:03.325693 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321988 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 15:42:03.325693 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.321991 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 15:42:03.325693 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322076 2583 flags.go:64] FLAG: --address="0.0.0.0" Sep 10 15:42:03.325693 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322117 2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 10 15:42:03.325693 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322124 2583 flags.go:64] FLAG: --anonymous-auth="true" Sep 10 15:42:03.326206 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322129 2583 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 10 15:42:03.326206 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322133 2583 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 10 15:42:03.326206 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322137 2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 10 15:42:03.326206 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322141 2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 10 15:42:03.326206 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322146 2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 10 15:42:03.326206 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322150 2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 10 15:42:03.326206 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322153 2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 10 15:42:03.326206 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322156 2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 10 15:42:03.326206 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322160 2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 10 15:42:03.326206 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322163 2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 10 15:42:03.326206 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322166 2583 flags.go:64] FLAG: --cgroup-root="" Sep 10 15:42:03.326206 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322169 2583 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 10 15:42:03.326206 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322172 2583 flags.go:64] FLAG: --client-ca-file="" Sep 10 15:42:03.326206 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322174 2583 flags.go:64] FLAG: --cloud-provider="external" Sep 10 15:42:03.326206 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322178 2583 flags.go:64] FLAG: --cluster-dns="[]" Sep 10 15:42:03.326206 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322182 2583 flags.go:64] FLAG: --cluster-domain="" Sep 10 15:42:03.326206 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322185 2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 10 15:42:03.326206 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322188 2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 10 15:42:03.326206 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322191 2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 10 15:42:03.326206 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322194 2583 flags.go:64] FLAG: --container-log-max-files="5" Sep 10 15:42:03.326206 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322198 2583 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 10 15:42:03.326206 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322201 2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 10 15:42:03.326206 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322207 2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 10 15:42:03.326780 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322210 2583 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 10 15:42:03.326780 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322213 2583 flags.go:64] FLAG: --contention-profiling="false" Sep 10 15:42:03.326780 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322216 2583 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 10 15:42:03.326780 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322219 2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 10 15:42:03.326780 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322222 2583 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 10 15:42:03.326780 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322225 2583 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 10 15:42:03.326780 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322229 2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 10 15:42:03.326780 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322232 2583 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 10 15:42:03.326780 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322235 2583 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 10 15:42:03.326780 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322238 2583 flags.go:64] FLAG: --enable-load-reader="false" Sep 10 15:42:03.326780 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322241 2583 flags.go:64] FLAG: --enable-server="true" Sep 10 15:42:03.326780 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322244 2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 10 15:42:03.326780 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322250 2583 flags.go:64] FLAG: --event-burst="100" Sep 10 15:42:03.326780 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322252 2583 flags.go:64] FLAG: --event-qps="50" Sep 10 15:42:03.326780 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322255 2583 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 10 15:42:03.326780 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322263 2583 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 10 15:42:03.326780 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322266 2583 flags.go:64] FLAG: --eviction-hard="" Sep 10 15:42:03.326780 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322270 2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 10 15:42:03.326780 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322273 2583 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 10 15:42:03.326780 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322276 2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 10 15:42:03.326780 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322279 2583 flags.go:64] FLAG: --eviction-soft="" Sep 10 15:42:03.326780 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322282 2583 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 10 15:42:03.326780 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322285 2583 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 10 15:42:03.326780 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322288 2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 10 15:42:03.326780 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322291 2583 flags.go:64] FLAG: --experimental-mounter-path="" Sep 10 15:42:03.327412 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322294 2583 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 10 15:42:03.327412 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322297 2583 flags.go:64] FLAG: --fail-swap-on="true" Sep 10 15:42:03.327412 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322300 2583 flags.go:64] FLAG: --feature-gates="" Sep 10 15:42:03.327412 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322304 2583 flags.go:64] FLAG: --file-check-frequency="20s" Sep 10 15:42:03.327412 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322307 2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 10 15:42:03.327412 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322310 2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 10 15:42:03.327412 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322314 2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 10 15:42:03.327412 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322318 2583 flags.go:64] FLAG: --healthz-port="10248" Sep 10 15:42:03.327412 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322321 2583 flags.go:64] FLAG: --help="false" Sep 10 15:42:03.327412 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322324 2583 flags.go:64] FLAG: --hostname-override="ip-10-0-139-107.ec2.internal" Sep 10 15:42:03.327412 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322327 2583 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 10 15:42:03.327412 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322330 2583 flags.go:64] FLAG: --http-check-frequency="20s" Sep 10 15:42:03.327412 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322333 2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 10 15:42:03.327412 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322336 2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 10 15:42:03.327412 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322340 2583 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 10 15:42:03.327412 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322343 2583 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 10 15:42:03.327412 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322346 2583 flags.go:64] FLAG: --image-service-endpoint="" Sep 10 15:42:03.327412 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322348 2583 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 10 15:42:03.327412 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322351 2583 flags.go:64] FLAG: --kube-api-burst="100" Sep 10 15:42:03.327412 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322354 2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 10 15:42:03.327412 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322357 2583 flags.go:64] FLAG: --kube-api-qps="50" Sep 10 15:42:03.327412 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322360 2583 flags.go:64] FLAG: --kube-reserved="" Sep 10 15:42:03.327412 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322363 2583 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 10 15:42:03.327412 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322366 2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 10 15:42:03.328005 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322369 2583 flags.go:64] FLAG: --kubelet-cgroups="" Sep 10 15:42:03.328005 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322372 2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 10 15:42:03.328005 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322375 2583 flags.go:64] FLAG: --lock-file="" Sep 10 15:42:03.328005 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322391 2583 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 10 15:42:03.328005 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322394 2583 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 10 15:42:03.328005 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322398 2583 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 10 15:42:03.328005 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322405 2583 flags.go:64] FLAG: --log-json-split-stream="false" Sep 10 15:42:03.328005 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322408 2583 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 10 15:42:03.328005 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322411 2583 flags.go:64] FLAG: --log-text-split-stream="false" Sep 10 15:42:03.328005 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322413 2583 flags.go:64] FLAG: --logging-format="text" Sep 10 15:42:03.328005 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322417 2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 10 15:42:03.328005 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322420 2583 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 10 15:42:03.328005 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322423 2583 flags.go:64] FLAG: --manifest-url="" Sep 10 15:42:03.328005 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322426 2583 flags.go:64] FLAG: --manifest-url-header="" Sep 10 15:42:03.328005 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322431 2583 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 10 15:42:03.328005 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322434 2583 flags.go:64] FLAG: --max-open-files="1000000" Sep 10 15:42:03.328005 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322439 2583 flags.go:64] FLAG: --max-pods="110" Sep 10 15:42:03.328005 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322442 2583 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 10 15:42:03.328005 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322445 2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 10 15:42:03.328005 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322448 2583 flags.go:64] FLAG: --memory-manager-policy="None" Sep 10 15:42:03.328005 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322451 2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 10 15:42:03.328005 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322454 2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 10 15:42:03.328005 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322457 2583 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 10 15:42:03.328005 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322460 2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 10 15:42:03.328005 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322468 2583 flags.go:64] FLAG: --node-status-max-images="50" Sep 10 15:42:03.328653 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322471 2583 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 10 15:42:03.328653 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322474 2583 flags.go:64] FLAG: --oom-score-adj="-999" Sep 10 15:42:03.328653 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322477 2583 flags.go:64] FLAG: --pod-cidr="" Sep 10 15:42:03.328653 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322480 2583 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 10 15:42:03.328653 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322485 2583 flags.go:64] FLAG: --pod-manifest-path="" Sep 10 15:42:03.328653 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322488 2583 flags.go:64] FLAG: --pod-max-pids="-1" Sep 10 15:42:03.328653 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322492 2583 flags.go:64] FLAG: --pods-per-core="0" Sep 10 15:42:03.328653 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322496 2583 flags.go:64] FLAG: --port="10250" Sep 10 15:42:03.328653 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322499 2583 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 10 15:42:03.328653 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322502 2583 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09fe6d882871a3c8c" Sep 10 15:42:03.328653 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322505 2583 flags.go:64] FLAG: --qos-reserved="" Sep 10 15:42:03.328653 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322508 2583 flags.go:64] FLAG: --read-only-port="10255" Sep 10 15:42:03.328653 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322511 2583 flags.go:64] FLAG: --register-node="true" Sep 10 15:42:03.328653 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322514 2583 flags.go:64] FLAG: --register-with-taints="" Sep 10 15:42:03.328653 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322521 2583 flags.go:64] FLAG: --registry-burst="10" Sep 10 15:42:03.328653 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322524 2583 flags.go:64] FLAG: --registry-qps="5" Sep 10 15:42:03.328653 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322527 2583 flags.go:64] FLAG: --reserved-cpus="" Sep 10 15:42:03.328653 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322530 2583 flags.go:64] FLAG: --reserved-memory="" Sep 10 15:42:03.328653 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322534 2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 10 15:42:03.328653 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322537 2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 10 15:42:03.328653 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322540 2583 flags.go:64] FLAG: --rotate-certificates="false" Sep 10 15:42:03.328653 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322544 2583 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 10 15:42:03.328653 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322547 2583 flags.go:64] FLAG: --runonce="false" Sep 10 15:42:03.328653 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322550 2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 10 15:42:03.328653 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322553 2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 10 15:42:03.329309 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322556 2583 flags.go:64] FLAG: --seccomp-default="false" Sep 10 15:42:03.329309 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322559 2583 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 10 15:42:03.329309 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322562 2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 10 15:42:03.329309 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322565 2583 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 10 15:42:03.329309 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322568 2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 10 15:42:03.329309 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322571 2583 flags.go:64] FLAG: --storage-driver-password="root" Sep 10 15:42:03.329309 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322574 2583 flags.go:64] FLAG: --storage-driver-secure="false" Sep 10 15:42:03.329309 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322578 2583 flags.go:64] FLAG: --storage-driver-table="stats" Sep 10 15:42:03.329309 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322581 2583 flags.go:64] FLAG: --storage-driver-user="root" Sep 10 15:42:03.329309 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322584 2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 10 15:42:03.329309 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322587 2583 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 10 15:42:03.329309 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322590 2583 flags.go:64] FLAG: --system-cgroups="" Sep 10 15:42:03.329309 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322593 2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 10 15:42:03.329309 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322598 2583 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 10 15:42:03.329309 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322601 2583 flags.go:64] FLAG: --tls-cert-file="" Sep 10 15:42:03.329309 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322605 2583 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 10 15:42:03.329309 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322609 2583 flags.go:64] FLAG: --tls-min-version="" Sep 10 15:42:03.329309 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322611 2583 flags.go:64] FLAG: --tls-private-key-file="" Sep 10 15:42:03.329309 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322614 2583 flags.go:64] FLAG: --topology-manager-policy="none" Sep 10 15:42:03.329309 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322617 2583 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 10 15:42:03.329309 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322620 2583 flags.go:64] FLAG: --topology-manager-scope="container" Sep 10 15:42:03.329309 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322624 2583 flags.go:64] FLAG: --v="2" Sep 10 15:42:03.329309 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322628 2583 flags.go:64] FLAG: --version="false" Sep 10 15:42:03.329309 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322632 2583 flags.go:64] FLAG: --vmodule="" Sep 10 15:42:03.329309 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322635 2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 10 15:42:03.329921 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.322638 2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 10 15:42:03.329921 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322759 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 15:42:03.329921 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322762 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 15:42:03.329921 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322767 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 15:42:03.329921 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322770 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 15:42:03.329921 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322773 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 15:42:03.329921 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322775 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 15:42:03.329921 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322779 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 15:42:03.329921 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322781 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 15:42:03.329921 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322784 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 15:42:03.329921 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322787 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 15:42:03.329921 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322790 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 15:42:03.329921 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322792 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 15:42:03.329921 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322796 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 15:42:03.329921 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322800 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 15:42:03.329921 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322802 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 15:42:03.329921 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322805 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 15:42:03.329921 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322808 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 15:42:03.329921 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322810 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 15:42:03.329921 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322813 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 15:42:03.330426 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322815 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 15:42:03.330426 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322818 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 15:42:03.330426 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322821 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 15:42:03.330426 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322824 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 15:42:03.330426 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322827 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 15:42:03.330426 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322829 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 15:42:03.330426 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322831 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 15:42:03.330426 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322834 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 15:42:03.330426 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322839 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 15:42:03.330426 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322841 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 15:42:03.330426 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322844 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 15:42:03.330426 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322847 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 15:42:03.330426 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322849 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 15:42:03.330426 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322852 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 15:42:03.330426 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322854 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 15:42:03.330426 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322860 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 15:42:03.330426 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322863 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 15:42:03.330426 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322865 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 15:42:03.330426 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322868 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 15:42:03.330426 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322870 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 15:42:03.330976 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322873 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 15:42:03.330976 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322875 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 15:42:03.330976 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322878 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 15:42:03.330976 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322881 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 15:42:03.330976 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322883 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 15:42:03.330976 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322886 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 15:42:03.330976 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322888 2583 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 15:42:03.330976 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322891 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 15:42:03.330976 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322893 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 15:42:03.330976 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322896 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 15:42:03.330976 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322900 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 15:42:03.330976 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322904 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 15:42:03.330976 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322907 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 15:42:03.330976 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322909 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 15:42:03.330976 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322912 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 15:42:03.330976 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322915 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 15:42:03.330976 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322918 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 15:42:03.330976 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322922 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 15:42:03.330976 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322924 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 15:42:03.331805 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322927 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 15:42:03.331805 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322931 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 15:42:03.331805 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322934 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 15:42:03.331805 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322937 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 15:42:03.331805 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322940 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 15:42:03.331805 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322942 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 15:42:03.331805 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322945 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 15:42:03.331805 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322948 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 15:42:03.331805 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322952 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 15:42:03.331805 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322954 2583 feature_gate.go:328] unrecognized feature gate: Example Sep 10 15:42:03.331805 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322957 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 15:42:03.331805 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322959 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 15:42:03.331805 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322962 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 15:42:03.331805 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322965 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 15:42:03.331805 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322967 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 15:42:03.331805 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322970 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 15:42:03.331805 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322972 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 15:42:03.331805 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322975 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 15:42:03.331805 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322978 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 15:42:03.331805 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322980 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 15:42:03.332665 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322983 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 15:42:03.332665 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322985 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 15:42:03.332665 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322988 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 15:42:03.332665 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322990 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 15:42:03.332665 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322993 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 15:42:03.332665 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322995 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 15:42:03.332665 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.322998 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 15:42:03.332665 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.323000 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 15:42:03.332665 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.323003 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 15:42:03.332665 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.323006 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 15:42:03.332665 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.323011 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 15:42:03.332665 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.323014 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 15:42:03.332665 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.323017 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 15:42:03.332665 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.323021 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 15:42:03.332665 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.323023 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 15:42:03.332665 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.323026 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 15:42:03.332665 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.323029 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 15:42:03.332665 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.323032 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 15:42:03.332665 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.323034 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 15:42:03.332665 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.323037 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 15:42:03.333596 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.323041 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 15:42:03.333596 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.323787 2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 15:42:03.333596 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.333345 2583 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 10 15:42:03.333596 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.333364 2583 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 10 15:42:03.335513 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.335498 2583 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 10 15:42:03.335576 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.335518 2583 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 10 15:42:03.335627 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335618 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 15:42:03.335627 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335626 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 15:42:03.335714 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335631 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 15:42:03.335714 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335636 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 15:42:03.335714 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335641 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 15:42:03.335714 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335645 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 15:42:03.335714 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335650 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 15:42:03.335714 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335654 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 15:42:03.335714 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335659 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 15:42:03.335714 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335663 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 15:42:03.335714 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335667 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 15:42:03.335714 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335672 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 15:42:03.335714 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335676 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 15:42:03.335714 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335680 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 15:42:03.335714 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335684 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 15:42:03.335714 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335688 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 15:42:03.335714 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335694 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 15:42:03.335714 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335699 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 15:42:03.335714 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335703 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 15:42:03.335714 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335708 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 15:42:03.335714 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335712 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 15:42:03.335714 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335716 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 15:42:03.336580 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335720 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 15:42:03.336580 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335724 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 15:42:03.336580 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335728 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 15:42:03.336580 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335733 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 15:42:03.336580 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335736 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 15:42:03.336580 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335741 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 15:42:03.336580 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335749 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 15:42:03.336580 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335755 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 15:42:03.336580 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335759 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 15:42:03.336580 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335764 2583 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 15:42:03.336580 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335767 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 15:42:03.336580 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335772 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 15:42:03.336580 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335776 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 15:42:03.336580 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335780 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 15:42:03.336580 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335784 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 15:42:03.336580 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335788 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 15:42:03.336580 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335792 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 15:42:03.336580 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335796 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 15:42:03.336580 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335800 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 15:42:03.337284 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335804 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 15:42:03.337284 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335808 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 15:42:03.337284 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335812 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 15:42:03.337284 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335818 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 15:42:03.337284 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335824 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 15:42:03.337284 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335830 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 15:42:03.337284 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335834 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 15:42:03.337284 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335840 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 15:42:03.337284 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335851 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 15:42:03.337284 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335856 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 15:42:03.337284 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335860 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 15:42:03.337284 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335864 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 15:42:03.337284 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335868 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 15:42:03.337284 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335872 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 15:42:03.337284 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335877 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 15:42:03.337284 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335881 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 15:42:03.337284 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335886 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 15:42:03.337284 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335891 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 15:42:03.337284 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335895 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 15:42:03.337846 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335899 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 15:42:03.337846 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335903 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 15:42:03.337846 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335908 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 15:42:03.337846 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335912 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 15:42:03.337846 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335916 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 15:42:03.337846 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335921 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 15:42:03.337846 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335925 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 15:42:03.337846 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335929 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 15:42:03.337846 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335934 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 15:42:03.337846 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335938 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 15:42:03.337846 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335942 2583 feature_gate.go:328] unrecognized feature gate: Example Sep 10 15:42:03.337846 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335946 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 15:42:03.337846 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335950 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 15:42:03.337846 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335954 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 15:42:03.337846 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335959 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 15:42:03.337846 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335963 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 15:42:03.337846 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335967 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 15:42:03.337846 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335971 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 15:42:03.337846 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335978 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 15:42:03.337846 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335984 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 15:42:03.338746 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335988 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 15:42:03.338746 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335992 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 15:42:03.338746 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.335997 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 15:42:03.338746 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336001 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 15:42:03.338746 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336006 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 15:42:03.338746 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336009 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 15:42:03.338746 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336014 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 15:42:03.338746 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336019 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 15:42:03.338746 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336024 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 15:42:03.338746 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336028 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 15:42:03.338746 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336032 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 15:42:03.338746 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336037 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 15:42:03.338746 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336041 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 15:42:03.338746 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336045 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 15:42:03.338746 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336049 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 15:42:03.338746 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336054 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 15:42:03.338746 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336058 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 15:42:03.338746 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336062 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 15:42:03.338746 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336066 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 15:42:03.339506 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.336074 2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 15:42:03.339506 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336312 2583 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 10 15:42:03.339506 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336320 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 10 15:42:03.339506 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336326 2583 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 10 15:42:03.339506 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336331 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 10 15:42:03.339506 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336335 2583 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 10 15:42:03.339506 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336339 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 10 15:42:03.339506 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336344 2583 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 10 15:42:03.339506 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336348 2583 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 10 15:42:03.339506 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336352 2583 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 10 15:42:03.339506 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336356 2583 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 10 15:42:03.339506 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336360 2583 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 10 15:42:03.339506 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336364 2583 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 10 15:42:03.339506 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336368 2583 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 10 15:42:03.339506 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336372 2583 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 10 15:42:03.339887 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336395 2583 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 10 15:42:03.339887 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336408 2583 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 10 15:42:03.339887 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336413 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 10 15:42:03.339887 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336420 2583 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 10 15:42:03.339887 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336426 2583 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 10 15:42:03.339887 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336431 2583 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 10 15:42:03.339887 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336436 2583 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 10 15:42:03.339887 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336441 2583 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 10 15:42:03.339887 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336447 2583 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 10 15:42:03.339887 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336452 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 10 15:42:03.339887 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336457 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 10 15:42:03.339887 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336461 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 10 15:42:03.339887 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336465 2583 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 10 15:42:03.339887 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336470 2583 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 10 15:42:03.339887 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336474 2583 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 10 15:42:03.339887 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336479 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 10 15:42:03.339887 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336483 2583 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 10 15:42:03.339887 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336487 2583 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 10 15:42:03.339887 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336491 2583 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 10 15:42:03.340506 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336495 2583 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 10 15:42:03.340506 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336499 2583 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 10 15:42:03.340506 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336502 2583 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 10 15:42:03.340506 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336507 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 10 15:42:03.340506 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336511 2583 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 10 15:42:03.340506 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336515 2583 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 10 15:42:03.340506 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336519 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 10 15:42:03.340506 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336523 2583 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 10 15:42:03.340506 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336527 2583 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 10 15:42:03.340506 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336531 2583 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 10 15:42:03.340506 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336535 2583 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 10 15:42:03.340506 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336540 2583 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 10 15:42:03.340506 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336543 2583 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 10 15:42:03.340506 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336547 2583 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 10 15:42:03.340506 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336551 2583 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 10 15:42:03.340506 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336563 2583 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 10 15:42:03.340506 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336569 2583 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 10 15:42:03.340506 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336577 2583 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 10 15:42:03.340506 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336583 2583 feature_gate.go:328] unrecognized feature gate: Example2 Sep 10 15:42:03.340986 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336589 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 10 15:42:03.340986 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336593 2583 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 10 15:42:03.340986 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336597 2583 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 10 15:42:03.340986 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336602 2583 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 10 15:42:03.340986 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336606 2583 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 10 15:42:03.340986 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336610 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 10 15:42:03.340986 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336614 2583 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 10 15:42:03.340986 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336618 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 10 15:42:03.340986 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336623 2583 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 10 15:42:03.340986 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336627 2583 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 10 15:42:03.340986 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336631 2583 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 10 15:42:03.340986 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336635 2583 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 10 15:42:03.340986 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336639 2583 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 10 15:42:03.340986 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336643 2583 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 10 15:42:03.340986 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336648 2583 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 10 15:42:03.340986 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336652 2583 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 10 15:42:03.340986 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336656 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 10 15:42:03.340986 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336660 2583 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 10 15:42:03.340986 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336664 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 10 15:42:03.341524 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336668 2583 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 10 15:42:03.341524 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336672 2583 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 10 15:42:03.341524 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336676 2583 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 10 15:42:03.341524 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336680 2583 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 10 15:42:03.341524 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336684 2583 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 10 15:42:03.341524 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336688 2583 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 10 15:42:03.341524 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336693 2583 feature_gate.go:328] unrecognized feature gate: Example Sep 10 15:42:03.341524 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336697 2583 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 10 15:42:03.341524 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336701 2583 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 10 15:42:03.341524 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336705 2583 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 10 15:42:03.341524 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336717 2583 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 10 15:42:03.341524 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336722 2583 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 10 15:42:03.341524 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336726 2583 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 10 15:42:03.341524 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336731 2583 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 10 15:42:03.341524 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336735 2583 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 10 15:42:03.341524 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336739 2583 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 10 15:42:03.341524 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336743 2583 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 10 15:42:03.341524 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336747 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 10 15:42:03.341524 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336751 2583 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 10 15:42:03.341983 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336755 2583 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 10 15:42:03.341983 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336759 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 10 15:42:03.341983 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336764 2583 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 10 15:42:03.341983 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336768 2583 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 10 15:42:03.341983 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336772 2583 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 10 15:42:03.341983 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336776 2583 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 10 15:42:03.341983 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336781 2583 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 10 15:42:03.341983 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336785 2583 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 10 15:42:03.341983 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:03.336789 2583 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 10 15:42:03.341983 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.336798 2583 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 10 15:42:03.341983 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.336981 2583 server.go:962] "Client rotation is on, will bootstrap in background" Sep 10 15:42:03.341983 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.340045 2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 10 15:42:03.341983 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.341354 2583 server.go:1019] "Starting client certificate rotation" Sep 10 15:42:03.341983 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.341423 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 15:42:03.342442 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.342426 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 10 15:42:03.344908 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.344890 2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 15:42:03.350917 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.350892 2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 10 15:42:03.361257 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.361240 2583 log.go:25] "Validated CRI v1 runtime API" Sep 10 15:42:03.364667 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.364654 2583 log.go:25] "Validated CRI v1 image API" Sep 10 15:42:03.365569 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.365548 2583 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 10 15:42:03.369037 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.369022 2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 15:42:03.369273 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.369250 2583 fs.go:135] Filesystem UUIDs: map[4c3c07fc-59a2-443b-b160-08158cdb00b5:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 cd28870f-c860-4beb-ab2e-6623325aaf11:/dev/nvme0n1p4] Sep 10 15:42:03.369310 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.369274 2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 10 15:42:03.375138 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.375027 2583 manager.go:216] Machine: {Timestamp:2026-09-10 15:42:03.373550919 +0000 UTC m=+0.348810865 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098017 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f9127ab0f595da47e569a6e20201c SystemUUID:ec2f9127-ab0f-595d-a47e-569a6e20201c BootID:a955e700-e5a2-406f-b90e-3f825bbca1f4 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:72:f3:95:c9:97 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:72:f3:95:c9:97 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f6:d4:5d:19:1e:32 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 10 15:42:03.375138 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.375131 2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 10 15:42:03.375300 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.375213 2583 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 10 15:42:03.378835 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.378807 2583 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 10 15:42:03.378974 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.378836 2583 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-107.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 10 15:42:03.379652 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.379642 2583 topology_manager.go:138] "Creating topology manager with none policy" Sep 10 15:42:03.379683 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.379653 2583 container_manager_linux.go:309] "Creating device plugin manager" Sep 10 15:42:03.379683 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.379661 2583 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 15:42:03.381175 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.381163 2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 10 15:42:03.382883 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.382873 2583 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 10 15:42:03.386003 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.385992 2583 state_mem.go:36] "Initialized new in-memory state store" Sep 10 15:42:03.386114 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.386105 2583 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 10 15:42:03.386150 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.386133 2583 kubelet.go:486] "Attempting to sync node with API server" Sep 10 15:42:03.386150 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.386147 2583 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 10 15:42:03.386202 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.386163 2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 10 15:42:03.386202 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.386171 2583 kubelet.go:398] "Adding apiserver pod source" Sep 10 15:42:03.386202 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.386180 2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 10 15:42:03.386637 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.386623 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 15:42:03.386692 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.386647 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 10 15:42:03.389042 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.389019 2583 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 10 15:42:03.391910 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.391896 2583 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 10 15:42:03.391980 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.391925 2583 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 10 15:42:03.392088 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.392079 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 10 15:42:03.392118 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.392094 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 10 15:42:03.392118 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.392100 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 10 15:42:03.392118 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.392106 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 10 15:42:03.392118 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.392114 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 10 15:42:03.392118 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.392120 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 10 15:42:03.392749 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.392730 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 10 15:42:03.392749 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.392748 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 10 15:42:03.392824 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.392763 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 10 15:42:03.392824 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.392770 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 10 15:42:03.392824 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.392784 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 10 15:42:03.392824 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.392793 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 10 15:42:03.394414 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.394391 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 10 15:42:03.394414 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:03.394399 2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-107.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 10 15:42:03.394534 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.394416 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 10 15:42:03.394534 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:03.394525 2583 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 10 15:42:03.397508 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.397475 2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vj5z9" Sep 10 15:42:03.397717 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.397704 2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-107.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 10 15:42:03.398194 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.398183 2583 server.go:1268] "Started kubelet" Sep 10 15:42:03.398286 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.398255 2583 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 10 15:42:03.398335 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.398275 2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 10 15:42:03.398335 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.398333 2583 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 10 15:42:03.399150 ip-10-0-139-107 systemd[1]: Started Kubernetes Kubelet. Sep 10 15:42:03.403664 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.403639 2583 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 10 15:42:03.403964 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.403830 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vj5z9" Sep 10 15:42:03.405039 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:03.405001 2583 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 10 15:42:03.405448 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.405429 2583 server.go:310] "Adding debug handlers to kubelet server" Sep 10 15:42:03.407587 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:03.406508 2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-107.ec2.internal.18d3fff958cf8d00 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-107.ec2.internal,UID:ip-10-0-139-107.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-107.ec2.internal,},FirstTimestamp:2026-09-10 15:42:03.39815552 +0000 UTC m=+0.373415463,LastTimestamp:2026-09-10 15:42:03.39815552 +0000 UTC m=+0.373415463,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-107.ec2.internal,}" Sep 10 15:42:03.408960 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.408949 2583 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 10 15:42:03.409017 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.408971 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 10 15:42:03.409207 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:03.409186 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-107.ec2.internal\" not found" Sep 10 15:42:03.409695 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.409667 2583 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 10 15:42:03.409695 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.409685 2583 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 10 15:42:03.409841 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.409666 2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 10 15:42:03.409841 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.409804 2583 reconstruct.go:97] "Volume reconstruction finished" Sep 10 15:42:03.409841 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.409812 2583 reconciler.go:29] "Reconciler: start to sync state" Sep 10 15:42:03.409996 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.409921 2583 factory.go:55] Registering systemd factory Sep 10 15:42:03.409996 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.409938 2583 factory.go:223] Registration of the systemd container factory successfully Sep 10 15:42:03.410138 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.410125 2583 factory.go:153] Registering CRI-O factory Sep 10 15:42:03.410181 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.410142 2583 factory.go:223] Registration of the crio container factory successfully Sep 10 15:42:03.410218 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.410189 2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 10 15:42:03.410218 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.410205 2583 factory.go:103] Registering Raw factory Sep 10 15:42:03.410218 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.410216 2583 manager.go:1196] Started watching for new ooms in manager Sep 10 15:42:03.410576 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.410565 2583 manager.go:349] Starting recovery of all containers Sep 10 15:42:03.418399 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.418354 2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 15:42:03.421326 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.421309 2583 manager.go:354] Recovery completed Sep 10 15:42:03.422706 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:03.422681 2583 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Sep 10 15:42:03.423392 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:03.423360 2583 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-139-107.ec2.internal\" not found" node="ip-10-0-139-107.ec2.internal" Sep 10 15:42:03.425864 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.425852 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 15:42:03.427627 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.427609 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-107.ec2.internal" event="NodeHasSufficientMemory" Sep 10 15:42:03.427722 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.427636 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-107.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 15:42:03.427722 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.427648 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-107.ec2.internal" event="NodeHasSufficientPID" Sep 10 15:42:03.428208 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.428193 2583 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 10 15:42:03.428208 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.428206 2583 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 10 15:42:03.428320 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.428199 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 15:42:03.428320 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.428228 2583 state_mem.go:36] "Initialized new in-memory state store" Sep 10 15:42:03.429077 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.429060 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-107.ec2.internal" event="NodeHasSufficientMemory" Sep 10 15:42:03.429143 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.429086 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-107.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 15:42:03.429143 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.429095 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-107.ec2.internal" event="NodeHasSufficientPID" Sep 10 15:42:03.429533 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.429517 2583 policy_none.go:49] "None policy: Start" Sep 10 15:42:03.429533 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.429535 2583 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 10 15:42:03.429645 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.429545 2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 10 15:42:03.430563 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.430553 2583 policy_none.go:47] "Start" Sep 10 15:42:03.475029 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.475013 2583 manager.go:337] "Starting Device Plugin manager" Sep 10 15:42:03.478117 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:03.475056 2583 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 10 15:42:03.478117 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.475069 2583 server.go:87] "Starting device plugin registration server" Sep 10 15:42:03.478117 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.475287 2583 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 10 15:42:03.478117 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.475299 2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 10 15:42:03.478117 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.475420 2583 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 10 15:42:03.478117 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.475520 2583 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 10 15:42:03.478117 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.475546 2583 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 10 15:42:03.478117 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:03.475929 2583 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 10 15:42:03.478117 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:03.475970 2583 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-107.ec2.internal\" not found" Sep 10 15:42:03.507361 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.507336 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 10 15:42:03.508467 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.508452 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 10 15:42:03.508537 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.508494 2583 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 10 15:42:03.508537 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.508518 2583 kubelet.go:2448] "Starting kubelet main sync loop" Sep 10 15:42:03.508632 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:03.508552 2583 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 10 15:42:03.510349 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.510333 2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 15:42:03.576123 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.576085 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 15:42:03.576845 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.576831 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-107.ec2.internal" event="NodeHasSufficientMemory" Sep 10 15:42:03.576919 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.576859 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-107.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 15:42:03.576919 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.576873 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-107.ec2.internal" event="NodeHasSufficientPID" Sep 10 15:42:03.576919 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.576905 2583 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-107.ec2.internal" Sep 10 15:42:03.584772 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.584757 2583 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-107.ec2.internal" Sep 10 15:42:03.584861 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:03.584776 2583 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-107.ec2.internal\": node \"ip-10-0-139-107.ec2.internal\" not found" Sep 10 15:42:03.596366 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:03.596342 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-107.ec2.internal\" not found" Sep 10 15:42:03.608974 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.608840 2583 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-107.ec2.internal"] Sep 10 15:42:03.609031 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.609014 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 15:42:03.609757 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.609739 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-107.ec2.internal" event="NodeHasSufficientMemory" Sep 10 15:42:03.609832 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.609770 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-107.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 15:42:03.609832 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.609784 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-107.ec2.internal" event="NodeHasSufficientPID" Sep 10 15:42:03.609935 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.609920 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/35f373b97584485ab0c1f517fb075548-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal\" (UID: \"35f373b97584485ab0c1f517fb075548\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal" Sep 10 15:42:03.609976 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.609948 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/35f373b97584485ab0c1f517fb075548-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal\" (UID: \"35f373b97584485ab0c1f517fb075548\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal" Sep 10 15:42:03.611095 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.611084 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 15:42:03.611248 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.611224 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal" Sep 10 15:42:03.611309 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.611259 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 15:42:03.611736 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.611714 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-107.ec2.internal" event="NodeHasSufficientMemory" Sep 10 15:42:03.611736 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.611728 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-107.ec2.internal" event="NodeHasSufficientMemory" Sep 10 15:42:03.611853 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.611745 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-107.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 15:42:03.611853 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.611745 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-107.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 15:42:03.611853 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.611755 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-107.ec2.internal" event="NodeHasSufficientPID" Sep 10 15:42:03.611853 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.611766 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-107.ec2.internal" event="NodeHasSufficientPID" Sep 10 15:42:03.613057 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.613041 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-107.ec2.internal" Sep 10 15:42:03.613106 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.613070 2583 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 10 15:42:03.614499 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.614483 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-107.ec2.internal" event="NodeHasSufficientMemory" Sep 10 15:42:03.614582 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.614505 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-107.ec2.internal" event="NodeHasNoDiskPressure" Sep 10 15:42:03.614582 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.614517 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-107.ec2.internal" event="NodeHasSufficientPID" Sep 10 15:42:03.642884 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:03.642864 2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-107.ec2.internal\" not found" node="ip-10-0-139-107.ec2.internal" Sep 10 15:42:03.647001 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:03.646985 2583 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-107.ec2.internal\" not found" node="ip-10-0-139-107.ec2.internal" Sep 10 15:42:03.696448 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:03.696428 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-107.ec2.internal\" not found" Sep 10 15:42:03.710779 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.710759 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/35f373b97584485ab0c1f517fb075548-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal\" (UID: \"35f373b97584485ab0c1f517fb075548\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal" Sep 10 15:42:03.710864 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.710790 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/35f373b97584485ab0c1f517fb075548-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal\" (UID: \"35f373b97584485ab0c1f517fb075548\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal" Sep 10 15:42:03.710920 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.710860 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/35f373b97584485ab0c1f517fb075548-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal\" (UID: \"35f373b97584485ab0c1f517fb075548\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal" Sep 10 15:42:03.710920 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.710870 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/35f373b97584485ab0c1f517fb075548-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal\" (UID: \"35f373b97584485ab0c1f517fb075548\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal" Sep 10 15:42:03.796691 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:03.796671 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-107.ec2.internal\" not found" Sep 10 15:42:03.810922 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.810901 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef65ebb74ba3516cc857fe0f9e4cef0a-config\") pod \"kube-apiserver-proxy-ip-10-0-139-107.ec2.internal\" (UID: \"ef65ebb74ba3516cc857fe0f9e4cef0a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-107.ec2.internal" Sep 10 15:42:03.897348 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:03.897298 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-107.ec2.internal\" not found" Sep 10 15:42:03.911614 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.911596 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef65ebb74ba3516cc857fe0f9e4cef0a-config\") pod \"kube-apiserver-proxy-ip-10-0-139-107.ec2.internal\" (UID: \"ef65ebb74ba3516cc857fe0f9e4cef0a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-107.ec2.internal" Sep 10 15:42:03.911668 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.911637 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef65ebb74ba3516cc857fe0f9e4cef0a-config\") pod \"kube-apiserver-proxy-ip-10-0-139-107.ec2.internal\" (UID: \"ef65ebb74ba3516cc857fe0f9e4cef0a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-107.ec2.internal" Sep 10 15:42:03.945737 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.945718 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal" Sep 10 15:42:03.950567 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:03.950552 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-107.ec2.internal" Sep 10 15:42:03.998176 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:03.998152 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-107.ec2.internal\" not found" Sep 10 15:42:04.098684 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:04.098664 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-107.ec2.internal\" not found" Sep 10 15:42:04.199154 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:04.199097 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-107.ec2.internal\" not found" Sep 10 15:42:04.299768 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:04.299744 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-107.ec2.internal\" not found" Sep 10 15:42:04.341584 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:04.341564 2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 10 15:42:04.341912 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:04.341710 2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 15:42:04.341912 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:04.341730 2583 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 10 15:42:04.400753 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:04.400732 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-107.ec2.internal\" not found" Sep 10 15:42:04.409834 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:04.409796 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 15:37:03 +0000 UTC" deadline="2028-05-09 07:37:57.988246294 +0000 UTC" Sep 10 15:42:04.409834 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:04.409831 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14559h55m53.57842027s" Sep 10 15:42:04.501418 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:04.501342 2583 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-107.ec2.internal\" not found" Sep 10 15:42:04.545606 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:04.545573 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35f373b97584485ab0c1f517fb075548.slice/crio-0f90ac58b7a63d7320fb0c577e54a8e705fa8dbae6ee5b29b074950c5c090dfa WatchSource:0}: Error finding container 0f90ac58b7a63d7320fb0c577e54a8e705fa8dbae6ee5b29b074950c5c090dfa: Status 404 returned error can't find the container with id 0f90ac58b7a63d7320fb0c577e54a8e705fa8dbae6ee5b29b074950c5c090dfa Sep 10 15:42:04.546084 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:04.546051 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef65ebb74ba3516cc857fe0f9e4cef0a.slice/crio-6d2bc40d10963349273f3572c18fa290d910649d4862ed18712a2490d5e78e18 WatchSource:0}: Error finding container 6d2bc40d10963349273f3572c18fa290d910649d4862ed18712a2490d5e78e18: Status 404 returned error can't find the container with id 6d2bc40d10963349273f3572c18fa290d910649d4862ed18712a2490d5e78e18 Sep 10 15:42:04.550518 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:04.550503 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 15:42:04.592441 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:04.592418 2583 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 15:42:04.609623 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:04.609599 2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal" Sep 10 15:42:04.621789 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:04.621772 2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 15:42:04.623944 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:04.623931 2583 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-107.ec2.internal" Sep 10 15:42:04.633435 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:04.633419 2583 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 15:42:04.636846 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:04.636834 2583 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 10 15:42:05.387348 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.387318 2583 apiserver.go:52] "Watching apiserver" Sep 10 15:42:05.398007 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.397835 2583 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 10 15:42:05.398874 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.398838 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-wqgsp","openshift-ovn-kubernetes/ovnkube-node-t48fw","kube-system/konnectivity-agent-mcx2v","openshift-multus/multus-8q52b","openshift-multus/multus-additional-cni-plugins-qgbvz","openshift-multus/network-metrics-daemon-qztq2","openshift-network-operator/iptables-alerter-m5t8c","kube-system/global-pull-secret-syncer-swxpd","kube-system/kube-apiserver-proxy-ip-10-0-139-107.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8bwv","openshift-cluster-node-tuning-operator/tuned-zdqdf","openshift-image-registry/node-ca-5lpqq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal"] Sep 10 15:42:05.406665 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.406620 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-m5t8c" Sep 10 15:42:05.408858 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.408833 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.409730 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.409605 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 10 15:42:05.409730 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.409628 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 10 15:42:05.409730 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.409664 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wm8km\"" Sep 10 15:42:05.409730 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.409725 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 10 15:42:05.410007 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.409865 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 10 15:42:05.410920 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.410901 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-s29k5\"" Sep 10 15:42:05.411372 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.411345 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mcx2v" Sep 10 15:42:05.412378 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.411728 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 10 15:42:05.412378 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.411842 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 10 15:42:05.412378 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.411986 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 10 15:42:05.412378 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.412016 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 10 15:42:05.412378 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.412184 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 10 15:42:05.412378 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.412306 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 10 15:42:05.412741 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.412650 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.413446 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.413428 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-mr4ll\"" Sep 10 15:42:05.414771 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.414487 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qgbvz" Sep 10 15:42:05.414771 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.414584 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 10 15:42:05.414938 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.414927 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 10 15:42:05.415204 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.415184 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 10 15:42:05.415328 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.415310 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 10 15:42:05.415522 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.415504 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 10 15:42:05.416257 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.415836 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 10 15:42:05.416322 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.416312 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-nj7mv\"" Sep 10 15:42:05.417028 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.417000 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lj8xg\"" Sep 10 15:42:05.417149 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.417131 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 10 15:42:05.417212 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.417009 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 10 15:42:05.418505 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.418488 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qztq2" Sep 10 15:42:05.418599 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:05.418565 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qztq2" podUID="bcf35967-2a15-4bd4-8837-a058d81d1aff" Sep 10 15:42:05.419978 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.419954 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/621b9bdc-da4f-4833-a547-2bd2c40e83d1-ovn-node-metrics-cert\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.420077 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.419993 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/621b9bdc-da4f-4833-a547-2bd2c40e83d1-node-log\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.420077 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.420022 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/621b9bdc-da4f-4833-a547-2bd2c40e83d1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.420077 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.420048 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/621b9bdc-da4f-4833-a547-2bd2c40e83d1-ovnkube-script-lib\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.420077 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.420067 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqgsp" Sep 10 15:42:05.420077 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.420075 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bl78\" (UniqueName: \"kubernetes.io/projected/621b9bdc-da4f-4833-a547-2bd2c40e83d1-kube-api-access-6bl78\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.420333 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.420104 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d339b125-a6fe-4bff-b264-281c40b162c7-iptables-alerter-script\") pod \"iptables-alerter-m5t8c\" (UID: \"d339b125-a6fe-4bff-b264-281c40b162c7\") " pod="openshift-network-operator/iptables-alerter-m5t8c" Sep 10 15:42:05.420333 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.420130 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/621b9bdc-da4f-4833-a547-2bd2c40e83d1-host-slash\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.420333 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.420154 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/621b9bdc-da4f-4833-a547-2bd2c40e83d1-host-run-netns\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.420333 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.420178 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/621b9bdc-da4f-4833-a547-2bd2c40e83d1-run-systemd\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.420333 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:05.420154 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqgsp" podUID="d88aeb8f-b005-4deb-bc50-5f843a53699c" Sep 10 15:42:05.420333 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.420203 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/621b9bdc-da4f-4833-a547-2bd2c40e83d1-etc-openvswitch\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.420333 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.420227 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/621b9bdc-da4f-4833-a547-2bd2c40e83d1-host-cni-netd\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.420333 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.420266 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/621b9bdc-da4f-4833-a547-2bd2c40e83d1-ovnkube-config\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.420333 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.420331 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3448a418-7ccc-4a41-b2f0-0e1fa12cd592-agent-certs\") pod \"konnectivity-agent-mcx2v\" (UID: \"3448a418-7ccc-4a41-b2f0-0e1fa12cd592\") " pod="kube-system/konnectivity-agent-mcx2v" Sep 10 15:42:05.420805 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.420368 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d339b125-a6fe-4bff-b264-281c40b162c7-host-slash\") pod \"iptables-alerter-m5t8c\" (UID: \"d339b125-a6fe-4bff-b264-281c40b162c7\") " pod="openshift-network-operator/iptables-alerter-m5t8c" Sep 10 15:42:05.420805 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.420475 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k6xs\" (UniqueName: \"kubernetes.io/projected/d339b125-a6fe-4bff-b264-281c40b162c7-kube-api-access-5k6xs\") pod \"iptables-alerter-m5t8c\" (UID: \"d339b125-a6fe-4bff-b264-281c40b162c7\") " pod="openshift-network-operator/iptables-alerter-m5t8c" Sep 10 15:42:05.420805 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.420502 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/621b9bdc-da4f-4833-a547-2bd2c40e83d1-host-kubelet\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.420805 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.420541 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/621b9bdc-da4f-4833-a547-2bd2c40e83d1-run-ovn\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.420805 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.420581 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/621b9bdc-da4f-4833-a547-2bd2c40e83d1-host-run-ovn-kubernetes\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.420805 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.420632 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/621b9bdc-da4f-4833-a547-2bd2c40e83d1-env-overrides\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.420805 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.420677 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/621b9bdc-da4f-4833-a547-2bd2c40e83d1-var-lib-openvswitch\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.420805 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.420706 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3448a418-7ccc-4a41-b2f0-0e1fa12cd592-konnectivity-ca\") pod \"konnectivity-agent-mcx2v\" (UID: \"3448a418-7ccc-4a41-b2f0-0e1fa12cd592\") " pod="kube-system/konnectivity-agent-mcx2v" Sep 10 15:42:05.420805 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.420728 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/621b9bdc-da4f-4833-a547-2bd2c40e83d1-systemd-units\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.420805 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.420744 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/621b9bdc-da4f-4833-a547-2bd2c40e83d1-run-openvswitch\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.420805 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.420768 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/621b9bdc-da4f-4833-a547-2bd2c40e83d1-log-socket\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.420805 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.420800 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/621b9bdc-da4f-4833-a547-2bd2c40e83d1-host-cni-bin\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.421738 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.421715 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8bwv" Sep 10 15:42:05.423627 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.423608 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zdqdf" Sep 10 15:42:05.424521 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.424496 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 10 15:42:05.424644 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.424621 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 10 15:42:05.424991 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.424971 2583 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 10 15:42:05.425065 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.425044 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 10 15:42:05.425315 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.425295 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5tfsg\"" Sep 10 15:42:05.425584 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.425560 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5lpqq" Sep 10 15:42:05.426358 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.426330 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 10 15:42:05.426519 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.426370 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 10 15:42:05.426519 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.426394 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-f2s6q\"" Sep 10 15:42:05.427024 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.426997 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swxpd" Sep 10 15:42:05.427156 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:05.427079 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swxpd" podUID="bdf61327-2614-4068-974c-a5c0f7c8e3e0" Sep 10 15:42:05.428124 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.428104 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 10 15:42:05.428348 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.428332 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 10 15:42:05.428566 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.428542 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 10 15:42:05.429965 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.429935 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vsspx\"" Sep 10 15:42:05.451129 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.451105 2583 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-lbjg6" Sep 10 15:42:05.458276 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.458203 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-lbjg6" Sep 10 15:42:05.511056 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.511033 2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 10 15:42:05.513693 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.513647 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-107.ec2.internal" event={"ID":"ef65ebb74ba3516cc857fe0f9e4cef0a","Type":"ContainerStarted","Data":"6d2bc40d10963349273f3572c18fa290d910649d4862ed18712a2490d5e78e18"} Sep 10 15:42:05.514865 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.514840 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal" event={"ID":"35f373b97584485ab0c1f517fb075548","Type":"ContainerStarted","Data":"0f90ac58b7a63d7320fb0c577e54a8e705fa8dbae6ee5b29b074950c5c090dfa"} Sep 10 15:42:05.521875 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.521850 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d40147d-08e8-4362-b32a-8ee4c9727518-multus-cni-dir\") pod \"multus-8q52b\" (UID: \"2d40147d-08e8-4362-b32a-8ee4c9727518\") " pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.521976 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.521890 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2d40147d-08e8-4362-b32a-8ee4c9727518-host-run-k8s-cni-cncf-io\") pod \"multus-8q52b\" (UID: \"2d40147d-08e8-4362-b32a-8ee4c9727518\") " pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.521976 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.521922 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b631218e-ec48-4b74-8e2b-e6047ceba20b-socket-dir\") pod \"aws-ebs-csi-driver-node-l8bwv\" (UID: \"b631218e-ec48-4b74-8e2b-e6047ceba20b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8bwv" Sep 10 15:42:05.522141 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.521990 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b631218e-ec48-4b74-8e2b-e6047ceba20b-device-dir\") pod \"aws-ebs-csi-driver-node-l8bwv\" (UID: \"b631218e-ec48-4b74-8e2b-e6047ceba20b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8bwv" Sep 10 15:42:05.522141 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.522040 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3448a418-7ccc-4a41-b2f0-0e1fa12cd592-konnectivity-ca\") pod \"konnectivity-agent-mcx2v\" (UID: \"3448a418-7ccc-4a41-b2f0-0e1fa12cd592\") " pod="kube-system/konnectivity-agent-mcx2v" Sep 10 15:42:05.522141 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.522066 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/621b9bdc-da4f-4833-a547-2bd2c40e83d1-host-cni-bin\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.522141 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.522092 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b068e03b-cd04-497c-81bf-1a8e4546dc77-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qgbvz\" (UID: \"b068e03b-cd04-497c-81bf-1a8e4546dc77\") " pod="openshift-multus/multus-additional-cni-plugins-qgbvz" Sep 10 15:42:05.522141 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.522118 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgxp9\" (UniqueName: \"kubernetes.io/projected/c2384bdb-90cf-49a6-885a-42f96018fb04-kube-api-access-qgxp9\") pod \"tuned-zdqdf\" (UID: \"c2384bdb-90cf-49a6-885a-42f96018fb04\") " pod="openshift-cluster-node-tuning-operator/tuned-zdqdf" Sep 10 15:42:05.522400 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.522144 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d40147d-08e8-4362-b32a-8ee4c9727518-system-cni-dir\") pod \"multus-8q52b\" (UID: \"2d40147d-08e8-4362-b32a-8ee4c9727518\") " pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.522400 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.522139 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/621b9bdc-da4f-4833-a547-2bd2c40e83d1-host-cni-bin\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.522400 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.522166 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b631218e-ec48-4b74-8e2b-e6047ceba20b-sys-fs\") pod \"aws-ebs-csi-driver-node-l8bwv\" (UID: \"b631218e-ec48-4b74-8e2b-e6047ceba20b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8bwv" Sep 10 15:42:05.522400 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.522196 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bl78\" (UniqueName: \"kubernetes.io/projected/621b9bdc-da4f-4833-a547-2bd2c40e83d1-kube-api-access-6bl78\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.522400 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.522220 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b068e03b-cd04-497c-81bf-1a8e4546dc77-system-cni-dir\") pod \"multus-additional-cni-plugins-qgbvz\" (UID: \"b068e03b-cd04-497c-81bf-1a8e4546dc77\") " pod="openshift-multus/multus-additional-cni-plugins-qgbvz" Sep 10 15:42:05.522400 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.522248 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b068e03b-cd04-497c-81bf-1a8e4546dc77-cnibin\") pod \"multus-additional-cni-plugins-qgbvz\" (UID: \"b068e03b-cd04-497c-81bf-1a8e4546dc77\") " pod="openshift-multus/multus-additional-cni-plugins-qgbvz" Sep 10 15:42:05.522400 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.522288 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b068e03b-cd04-497c-81bf-1a8e4546dc77-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qgbvz\" (UID: \"b068e03b-cd04-497c-81bf-1a8e4546dc77\") " pod="openshift-multus/multus-additional-cni-plugins-qgbvz" Sep 10 15:42:05.522400 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.522314 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s62b8\" (UniqueName: \"kubernetes.io/projected/b068e03b-cd04-497c-81bf-1a8e4546dc77-kube-api-access-s62b8\") pod \"multus-additional-cni-plugins-qgbvz\" (UID: \"b068e03b-cd04-497c-81bf-1a8e4546dc77\") " pod="openshift-multus/multus-additional-cni-plugins-qgbvz" Sep 10 15:42:05.522400 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.522340 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c2384bdb-90cf-49a6-885a-42f96018fb04-etc-sysconfig\") pod \"tuned-zdqdf\" (UID: \"c2384bdb-90cf-49a6-885a-42f96018fb04\") " pod="openshift-cluster-node-tuning-operator/tuned-zdqdf" Sep 10 15:42:05.522400 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.522365 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c2384bdb-90cf-49a6-885a-42f96018fb04-etc-sysctl-conf\") pod \"tuned-zdqdf\" (UID: \"c2384bdb-90cf-49a6-885a-42f96018fb04\") " pod="openshift-cluster-node-tuning-operator/tuned-zdqdf" Sep 10 15:42:05.522400 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.522389 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bdf61327-2614-4068-974c-a5c0f7c8e3e0-dbus\") pod \"global-pull-secret-syncer-swxpd\" (UID: \"bdf61327-2614-4068-974c-a5c0f7c8e3e0\") " pod="kube-system/global-pull-secret-syncer-swxpd" Sep 10 15:42:05.522923 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.522433 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/621b9bdc-da4f-4833-a547-2bd2c40e83d1-host-slash\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.522923 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.522471 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/621b9bdc-da4f-4833-a547-2bd2c40e83d1-host-run-netns\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.522923 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.522496 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c2384bdb-90cf-49a6-885a-42f96018fb04-etc-modprobe-d\") pod \"tuned-zdqdf\" (UID: \"c2384bdb-90cf-49a6-885a-42f96018fb04\") " pod="openshift-cluster-node-tuning-operator/tuned-zdqdf" Sep 10 15:42:05.522923 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.522527 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/621b9bdc-da4f-4833-a547-2bd2c40e83d1-host-run-netns\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.522923 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.522486 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/621b9bdc-da4f-4833-a547-2bd2c40e83d1-host-slash\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.522923 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.522522 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d40147d-08e8-4362-b32a-8ee4c9727518-host-var-lib-cni-bin\") pod \"multus-8q52b\" (UID: \"2d40147d-08e8-4362-b32a-8ee4c9727518\") " pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.522923 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.522575 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b631218e-ec48-4b74-8e2b-e6047ceba20b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l8bwv\" (UID: \"b631218e-ec48-4b74-8e2b-e6047ceba20b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8bwv" Sep 10 15:42:05.522923 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.522605 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3448a418-7ccc-4a41-b2f0-0e1fa12cd592-agent-certs\") pod \"konnectivity-agent-mcx2v\" (UID: \"3448a418-7ccc-4a41-b2f0-0e1fa12cd592\") " pod="kube-system/konnectivity-agent-mcx2v" Sep 10 15:42:05.522923 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.522628 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/621b9bdc-da4f-4833-a547-2bd2c40e83d1-host-kubelet\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.522923 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.522649 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/621b9bdc-da4f-4833-a547-2bd2c40e83d1-host-run-ovn-kubernetes\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.522923 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.522675 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/621b9bdc-da4f-4833-a547-2bd2c40e83d1-ovn-node-metrics-cert\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.522923 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.522702 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2384bdb-90cf-49a6-885a-42f96018fb04-etc-kubernetes\") pod \"tuned-zdqdf\" (UID: \"c2384bdb-90cf-49a6-885a-42f96018fb04\") " pod="openshift-cluster-node-tuning-operator/tuned-zdqdf" Sep 10 15:42:05.522923 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.522705 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/621b9bdc-da4f-4833-a547-2bd2c40e83d1-host-kubelet\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.522923 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.522740 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/621b9bdc-da4f-4833-a547-2bd2c40e83d1-host-run-ovn-kubernetes\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.522923 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.522728 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c2384bdb-90cf-49a6-885a-42f96018fb04-etc-tuned\") pod \"tuned-zdqdf\" (UID: \"c2384bdb-90cf-49a6-885a-42f96018fb04\") " pod="openshift-cluster-node-tuning-operator/tuned-zdqdf" Sep 10 15:42:05.522923 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.522790 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2d40147d-08e8-4362-b32a-8ee4c9727518-host-var-lib-cni-multus\") pod \"multus-8q52b\" (UID: \"2d40147d-08e8-4362-b32a-8ee4c9727518\") " pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.522923 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.522823 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/621b9bdc-da4f-4833-a547-2bd2c40e83d1-log-socket\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.523758 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.522851 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2384bdb-90cf-49a6-885a-42f96018fb04-var-lib-kubelet\") pod \"tuned-zdqdf\" (UID: \"c2384bdb-90cf-49a6-885a-42f96018fb04\") " pod="openshift-cluster-node-tuning-operator/tuned-zdqdf" Sep 10 15:42:05.523758 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.522875 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bdf61327-2614-4068-974c-a5c0f7c8e3e0-kubelet-config\") pod \"global-pull-secret-syncer-swxpd\" (UID: \"bdf61327-2614-4068-974c-a5c0f7c8e3e0\") " pod="kube-system/global-pull-secret-syncer-swxpd" Sep 10 15:42:05.523758 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.522898 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/621b9bdc-da4f-4833-a547-2bd2c40e83d1-log-socket\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.523758 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.522900 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdf61327-2614-4068-974c-a5c0f7c8e3e0-original-pull-secret\") pod \"global-pull-secret-syncer-swxpd\" (UID: \"bdf61327-2614-4068-974c-a5c0f7c8e3e0\") " pod="kube-system/global-pull-secret-syncer-swxpd" Sep 10 15:42:05.523758 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.522948 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7jw8\" (UniqueName: \"kubernetes.io/projected/bcf35967-2a15-4bd4-8837-a058d81d1aff-kube-api-access-w7jw8\") pod \"network-metrics-daemon-qztq2\" (UID: \"bcf35967-2a15-4bd4-8837-a058d81d1aff\") " pod="openshift-multus/network-metrics-daemon-qztq2" Sep 10 15:42:05.523758 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.522991 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b631218e-ec48-4b74-8e2b-e6047ceba20b-etc-selinux\") pod \"aws-ebs-csi-driver-node-l8bwv\" (UID: \"b631218e-ec48-4b74-8e2b-e6047ceba20b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8bwv" Sep 10 15:42:05.523758 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.523038 2583 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 10 15:42:05.523758 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.523061 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7dc7aa02-57e8-4cdc-aab6-9875e1427825-serviceca\") pod \"node-ca-5lpqq\" (UID: \"7dc7aa02-57e8-4cdc-aab6-9875e1427825\") " pod="openshift-image-registry/node-ca-5lpqq" Sep 10 15:42:05.523758 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.523087 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6gx9\" (UniqueName: \"kubernetes.io/projected/7dc7aa02-57e8-4cdc-aab6-9875e1427825-kube-api-access-r6gx9\") pod \"node-ca-5lpqq\" (UID: \"7dc7aa02-57e8-4cdc-aab6-9875e1427825\") " pod="openshift-image-registry/node-ca-5lpqq" Sep 10 15:42:05.523758 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.523116 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/621b9bdc-da4f-4833-a547-2bd2c40e83d1-node-log\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.523758 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.523150 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/621b9bdc-da4f-4833-a547-2bd2c40e83d1-ovnkube-script-lib\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.523758 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.523207 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/621b9bdc-da4f-4833-a547-2bd2c40e83d1-node-log\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.523758 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.523225 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b068e03b-cd04-497c-81bf-1a8e4546dc77-cni-binary-copy\") pod \"multus-additional-cni-plugins-qgbvz\" (UID: \"b068e03b-cd04-497c-81bf-1a8e4546dc77\") " pod="openshift-multus/multus-additional-cni-plugins-qgbvz" Sep 10 15:42:05.523758 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.523410 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2384bdb-90cf-49a6-885a-42f96018fb04-sys\") pod \"tuned-zdqdf\" (UID: \"c2384bdb-90cf-49a6-885a-42f96018fb04\") " pod="openshift-cluster-node-tuning-operator/tuned-zdqdf" Sep 10 15:42:05.523758 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.523462 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/3448a418-7ccc-4a41-b2f0-0e1fa12cd592-konnectivity-ca\") pod \"konnectivity-agent-mcx2v\" (UID: \"3448a418-7ccc-4a41-b2f0-0e1fa12cd592\") " pod="kube-system/konnectivity-agent-mcx2v" Sep 10 15:42:05.523758 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.523575 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d339b125-a6fe-4bff-b264-281c40b162c7-iptables-alerter-script\") pod \"iptables-alerter-m5t8c\" (UID: \"d339b125-a6fe-4bff-b264-281c40b162c7\") " pod="openshift-network-operator/iptables-alerter-m5t8c" Sep 10 15:42:05.523758 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.523608 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/621b9bdc-da4f-4833-a547-2bd2c40e83d1-host-cni-netd\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.524511 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.523645 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/621b9bdc-da4f-4833-a547-2bd2c40e83d1-ovnkube-config\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.524511 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.523671 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c2384bdb-90cf-49a6-885a-42f96018fb04-run\") pod \"tuned-zdqdf\" (UID: \"c2384bdb-90cf-49a6-885a-42f96018fb04\") " pod="openshift-cluster-node-tuning-operator/tuned-zdqdf" Sep 10 15:42:05.524511 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.523696 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d40147d-08e8-4362-b32a-8ee4c9727518-host-run-netns\") pod \"multus-8q52b\" (UID: \"2d40147d-08e8-4362-b32a-8ee4c9727518\") " pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.524511 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.523721 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2d40147d-08e8-4362-b32a-8ee4c9727518-multus-daemon-config\") pod \"multus-8q52b\" (UID: \"2d40147d-08e8-4362-b32a-8ee4c9727518\") " pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.524511 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.523746 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2d40147d-08e8-4362-b32a-8ee4c9727518-host-run-multus-certs\") pod \"multus-8q52b\" (UID: \"2d40147d-08e8-4362-b32a-8ee4c9727518\") " pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.524511 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.523774 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/621b9bdc-da4f-4833-a547-2bd2c40e83d1-run-ovn\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.524511 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.523797 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2384bdb-90cf-49a6-885a-42f96018fb04-host\") pod \"tuned-zdqdf\" (UID: \"c2384bdb-90cf-49a6-885a-42f96018fb04\") " pod="openshift-cluster-node-tuning-operator/tuned-zdqdf" Sep 10 15:42:05.524511 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.523821 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d40147d-08e8-4362-b32a-8ee4c9727518-cni-binary-copy\") pod \"multus-8q52b\" (UID: \"2d40147d-08e8-4362-b32a-8ee4c9727518\") " pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.524511 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.523845 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7dc7aa02-57e8-4cdc-aab6-9875e1427825-host\") pod \"node-ca-5lpqq\" (UID: \"7dc7aa02-57e8-4cdc-aab6-9875e1427825\") " pod="openshift-image-registry/node-ca-5lpqq" Sep 10 15:42:05.524511 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.523850 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/621b9bdc-da4f-4833-a547-2bd2c40e83d1-ovnkube-script-lib\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.524511 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.523874 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/621b9bdc-da4f-4833-a547-2bd2c40e83d1-systemd-units\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.524511 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.523943 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/621b9bdc-da4f-4833-a547-2bd2c40e83d1-run-openvswitch\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.524511 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.523949 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/621b9bdc-da4f-4833-a547-2bd2c40e83d1-host-cni-netd\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.524511 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.523968 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2d40147d-08e8-4362-b32a-8ee4c9727518-multus-socket-dir-parent\") pod \"multus-8q52b\" (UID: \"2d40147d-08e8-4362-b32a-8ee4c9727518\") " pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.524511 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.523985 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2d40147d-08e8-4362-b32a-8ee4c9727518-hostroot\") pod \"multus-8q52b\" (UID: \"2d40147d-08e8-4362-b32a-8ee4c9727518\") " pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.524511 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.524030 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q5c4\" (UniqueName: \"kubernetes.io/projected/d88aeb8f-b005-4deb-bc50-5f843a53699c-kube-api-access-2q5c4\") pod \"network-check-target-wqgsp\" (UID: \"d88aeb8f-b005-4deb-bc50-5f843a53699c\") " pod="openshift-network-diagnostics/network-check-target-wqgsp" Sep 10 15:42:05.524511 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.524053 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b631218e-ec48-4b74-8e2b-e6047ceba20b-registration-dir\") pod \"aws-ebs-csi-driver-node-l8bwv\" (UID: \"b631218e-ec48-4b74-8e2b-e6047ceba20b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8bwv" Sep 10 15:42:05.525334 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.524067 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2384bdb-90cf-49a6-885a-42f96018fb04-tmp\") pod \"tuned-zdqdf\" (UID: \"c2384bdb-90cf-49a6-885a-42f96018fb04\") " pod="openshift-cluster-node-tuning-operator/tuned-zdqdf" Sep 10 15:42:05.525334 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.524085 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d40147d-08e8-4362-b32a-8ee4c9727518-etc-kubernetes\") pod \"multus-8q52b\" (UID: \"2d40147d-08e8-4362-b32a-8ee4c9727518\") " pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.525334 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.524094 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d339b125-a6fe-4bff-b264-281c40b162c7-iptables-alerter-script\") pod \"iptables-alerter-m5t8c\" (UID: \"d339b125-a6fe-4bff-b264-281c40b162c7\") " pod="openshift-network-operator/iptables-alerter-m5t8c" Sep 10 15:42:05.525334 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.524100 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/621b9bdc-da4f-4833-a547-2bd2c40e83d1-env-overrides\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.525334 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.524159 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/621b9bdc-da4f-4833-a547-2bd2c40e83d1-run-openvswitch\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.525334 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.524182 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/621b9bdc-da4f-4833-a547-2bd2c40e83d1-run-systemd\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.525334 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.524207 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d40147d-08e8-4362-b32a-8ee4c9727518-cnibin\") pod \"multus-8q52b\" (UID: \"2d40147d-08e8-4362-b32a-8ee4c9727518\") " pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.525334 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.524224 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcf35967-2a15-4bd4-8837-a058d81d1aff-metrics-certs\") pod \"network-metrics-daemon-qztq2\" (UID: \"bcf35967-2a15-4bd4-8837-a058d81d1aff\") " pod="openshift-multus/network-metrics-daemon-qztq2" Sep 10 15:42:05.525334 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.524240 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tggbc\" (UniqueName: \"kubernetes.io/projected/b631218e-ec48-4b74-8e2b-e6047ceba20b-kube-api-access-tggbc\") pod \"aws-ebs-csi-driver-node-l8bwv\" (UID: \"b631218e-ec48-4b74-8e2b-e6047ceba20b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8bwv" Sep 10 15:42:05.525334 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.524257 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d339b125-a6fe-4bff-b264-281c40b162c7-host-slash\") pod \"iptables-alerter-m5t8c\" (UID: \"d339b125-a6fe-4bff-b264-281c40b162c7\") " pod="openshift-network-operator/iptables-alerter-m5t8c" Sep 10 15:42:05.525334 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.524277 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k6xs\" (UniqueName: \"kubernetes.io/projected/d339b125-a6fe-4bff-b264-281c40b162c7-kube-api-access-5k6xs\") pod \"iptables-alerter-m5t8c\" (UID: \"d339b125-a6fe-4bff-b264-281c40b162c7\") " pod="openshift-network-operator/iptables-alerter-m5t8c" Sep 10 15:42:05.525334 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.524298 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b068e03b-cd04-497c-81bf-1a8e4546dc77-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qgbvz\" (UID: \"b068e03b-cd04-497c-81bf-1a8e4546dc77\") " pod="openshift-multus/multus-additional-cni-plugins-qgbvz" Sep 10 15:42:05.525334 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.524315 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c2384bdb-90cf-49a6-885a-42f96018fb04-etc-systemd\") pod \"tuned-zdqdf\" (UID: \"c2384bdb-90cf-49a6-885a-42f96018fb04\") " pod="openshift-cluster-node-tuning-operator/tuned-zdqdf" Sep 10 15:42:05.525334 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.524333 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d40147d-08e8-4362-b32a-8ee4c9727518-host-var-lib-kubelet\") pod \"multus-8q52b\" (UID: \"2d40147d-08e8-4362-b32a-8ee4c9727518\") " pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.525334 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.524349 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/621b9bdc-da4f-4833-a547-2bd2c40e83d1-var-lib-openvswitch\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.525334 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.524371 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/621b9bdc-da4f-4833-a547-2bd2c40e83d1-ovnkube-config\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.525334 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.524365 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c2384bdb-90cf-49a6-885a-42f96018fb04-etc-sysctl-d\") pod \"tuned-zdqdf\" (UID: \"c2384bdb-90cf-49a6-885a-42f96018fb04\") " pod="openshift-cluster-node-tuning-operator/tuned-zdqdf" Sep 10 15:42:05.526229 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.524432 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/621b9bdc-da4f-4833-a547-2bd2c40e83d1-run-ovn\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.526229 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.524449 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkc95\" (UniqueName: \"kubernetes.io/projected/2d40147d-08e8-4362-b32a-8ee4c9727518-kube-api-access-pkc95\") pod \"multus-8q52b\" (UID: \"2d40147d-08e8-4362-b32a-8ee4c9727518\") " pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.526229 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.524491 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/621b9bdc-da4f-4833-a547-2bd2c40e83d1-systemd-units\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.526229 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.524496 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/621b9bdc-da4f-4833-a547-2bd2c40e83d1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.526229 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.524433 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/621b9bdc-da4f-4833-a547-2bd2c40e83d1-env-overrides\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.526229 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.524527 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/621b9bdc-da4f-4833-a547-2bd2c40e83d1-run-systemd\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.526229 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.524625 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/621b9bdc-da4f-4833-a547-2bd2c40e83d1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.526229 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.524685 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d339b125-a6fe-4bff-b264-281c40b162c7-host-slash\") pod \"iptables-alerter-m5t8c\" (UID: \"d339b125-a6fe-4bff-b264-281c40b162c7\") " pod="openshift-network-operator/iptables-alerter-m5t8c" Sep 10 15:42:05.526229 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.524729 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/621b9bdc-da4f-4833-a547-2bd2c40e83d1-var-lib-openvswitch\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.526229 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.524845 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/621b9bdc-da4f-4833-a547-2bd2c40e83d1-etc-openvswitch\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.526229 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.525028 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c2384bdb-90cf-49a6-885a-42f96018fb04-lib-modules\") pod \"tuned-zdqdf\" (UID: \"c2384bdb-90cf-49a6-885a-42f96018fb04\") " pod="openshift-cluster-node-tuning-operator/tuned-zdqdf" Sep 10 15:42:05.526229 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.525073 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d40147d-08e8-4362-b32a-8ee4c9727518-multus-conf-dir\") pod \"multus-8q52b\" (UID: \"2d40147d-08e8-4362-b32a-8ee4c9727518\") " pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.526229 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.525079 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/621b9bdc-da4f-4833-a547-2bd2c40e83d1-etc-openvswitch\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.527322 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.527285 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/621b9bdc-da4f-4833-a547-2bd2c40e83d1-ovn-node-metrics-cert\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.527443 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.527386 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/3448a418-7ccc-4a41-b2f0-0e1fa12cd592-agent-certs\") pod \"konnectivity-agent-mcx2v\" (UID: \"3448a418-7ccc-4a41-b2f0-0e1fa12cd592\") " pod="kube-system/konnectivity-agent-mcx2v" Sep 10 15:42:05.533640 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.533617 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bl78\" (UniqueName: \"kubernetes.io/projected/621b9bdc-da4f-4833-a547-2bd2c40e83d1-kube-api-access-6bl78\") pod \"ovnkube-node-t48fw\" (UID: \"621b9bdc-da4f-4833-a547-2bd2c40e83d1\") " pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.534342 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.534320 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k6xs\" (UniqueName: \"kubernetes.io/projected/d339b125-a6fe-4bff-b264-281c40b162c7-kube-api-access-5k6xs\") pod \"iptables-alerter-m5t8c\" (UID: \"d339b125-a6fe-4bff-b264-281c40b162c7\") " pod="openshift-network-operator/iptables-alerter-m5t8c" Sep 10 15:42:05.626286 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.626255 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b631218e-ec48-4b74-8e2b-e6047ceba20b-socket-dir\") pod \"aws-ebs-csi-driver-node-l8bwv\" (UID: \"b631218e-ec48-4b74-8e2b-e6047ceba20b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8bwv" Sep 10 15:42:05.626286 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.626289 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b631218e-ec48-4b74-8e2b-e6047ceba20b-device-dir\") pod \"aws-ebs-csi-driver-node-l8bwv\" (UID: \"b631218e-ec48-4b74-8e2b-e6047ceba20b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8bwv" Sep 10 15:42:05.626523 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.626317 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b068e03b-cd04-497c-81bf-1a8e4546dc77-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qgbvz\" (UID: \"b068e03b-cd04-497c-81bf-1a8e4546dc77\") " pod="openshift-multus/multus-additional-cni-plugins-qgbvz" Sep 10 15:42:05.626523 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.626343 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgxp9\" (UniqueName: \"kubernetes.io/projected/c2384bdb-90cf-49a6-885a-42f96018fb04-kube-api-access-qgxp9\") pod \"tuned-zdqdf\" (UID: \"c2384bdb-90cf-49a6-885a-42f96018fb04\") " pod="openshift-cluster-node-tuning-operator/tuned-zdqdf" Sep 10 15:42:05.626523 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.626420 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b631218e-ec48-4b74-8e2b-e6047ceba20b-device-dir\") pod \"aws-ebs-csi-driver-node-l8bwv\" (UID: \"b631218e-ec48-4b74-8e2b-e6047ceba20b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8bwv" Sep 10 15:42:05.626523 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.626444 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d40147d-08e8-4362-b32a-8ee4c9727518-system-cni-dir\") pod \"multus-8q52b\" (UID: \"2d40147d-08e8-4362-b32a-8ee4c9727518\") " pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.626523 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.626488 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b631218e-ec48-4b74-8e2b-e6047ceba20b-sys-fs\") pod \"aws-ebs-csi-driver-node-l8bwv\" (UID: \"b631218e-ec48-4b74-8e2b-e6047ceba20b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8bwv" Sep 10 15:42:05.626523 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.626481 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b631218e-ec48-4b74-8e2b-e6047ceba20b-socket-dir\") pod \"aws-ebs-csi-driver-node-l8bwv\" (UID: \"b631218e-ec48-4b74-8e2b-e6047ceba20b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8bwv" Sep 10 15:42:05.626523 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.626518 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b068e03b-cd04-497c-81bf-1a8e4546dc77-system-cni-dir\") pod \"multus-additional-cni-plugins-qgbvz\" (UID: \"b068e03b-cd04-497c-81bf-1a8e4546dc77\") " pod="openshift-multus/multus-additional-cni-plugins-qgbvz" Sep 10 15:42:05.626850 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.626545 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b068e03b-cd04-497c-81bf-1a8e4546dc77-cnibin\") pod \"multus-additional-cni-plugins-qgbvz\" (UID: \"b068e03b-cd04-497c-81bf-1a8e4546dc77\") " pod="openshift-multus/multus-additional-cni-plugins-qgbvz" Sep 10 15:42:05.626850 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.626556 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d40147d-08e8-4362-b32a-8ee4c9727518-system-cni-dir\") pod \"multus-8q52b\" (UID: \"2d40147d-08e8-4362-b32a-8ee4c9727518\") " pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.626850 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.626567 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b631218e-ec48-4b74-8e2b-e6047ceba20b-sys-fs\") pod \"aws-ebs-csi-driver-node-l8bwv\" (UID: \"b631218e-ec48-4b74-8e2b-e6047ceba20b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8bwv" Sep 10 15:42:05.626850 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.626572 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b068e03b-cd04-497c-81bf-1a8e4546dc77-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qgbvz\" (UID: \"b068e03b-cd04-497c-81bf-1a8e4546dc77\") " pod="openshift-multus/multus-additional-cni-plugins-qgbvz" Sep 10 15:42:05.626850 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.626601 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s62b8\" (UniqueName: \"kubernetes.io/projected/b068e03b-cd04-497c-81bf-1a8e4546dc77-kube-api-access-s62b8\") pod \"multus-additional-cni-plugins-qgbvz\" (UID: \"b068e03b-cd04-497c-81bf-1a8e4546dc77\") " pod="openshift-multus/multus-additional-cni-plugins-qgbvz" Sep 10 15:42:05.626850 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.626609 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b068e03b-cd04-497c-81bf-1a8e4546dc77-cnibin\") pod \"multus-additional-cni-plugins-qgbvz\" (UID: \"b068e03b-cd04-497c-81bf-1a8e4546dc77\") " pod="openshift-multus/multus-additional-cni-plugins-qgbvz" Sep 10 15:42:05.626850 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.626610 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b068e03b-cd04-497c-81bf-1a8e4546dc77-system-cni-dir\") pod \"multus-additional-cni-plugins-qgbvz\" (UID: \"b068e03b-cd04-497c-81bf-1a8e4546dc77\") " pod="openshift-multus/multus-additional-cni-plugins-qgbvz" Sep 10 15:42:05.626850 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.626626 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c2384bdb-90cf-49a6-885a-42f96018fb04-etc-sysconfig\") pod \"tuned-zdqdf\" (UID: \"c2384bdb-90cf-49a6-885a-42f96018fb04\") " pod="openshift-cluster-node-tuning-operator/tuned-zdqdf" Sep 10 15:42:05.626850 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.626653 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c2384bdb-90cf-49a6-885a-42f96018fb04-etc-sysctl-conf\") pod \"tuned-zdqdf\" (UID: \"c2384bdb-90cf-49a6-885a-42f96018fb04\") " pod="openshift-cluster-node-tuning-operator/tuned-zdqdf" Sep 10 15:42:05.626850 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.626678 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bdf61327-2614-4068-974c-a5c0f7c8e3e0-dbus\") pod \"global-pull-secret-syncer-swxpd\" (UID: \"bdf61327-2614-4068-974c-a5c0f7c8e3e0\") " pod="kube-system/global-pull-secret-syncer-swxpd" Sep 10 15:42:05.626850 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.626703 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c2384bdb-90cf-49a6-885a-42f96018fb04-etc-modprobe-d\") pod \"tuned-zdqdf\" (UID: \"c2384bdb-90cf-49a6-885a-42f96018fb04\") " pod="openshift-cluster-node-tuning-operator/tuned-zdqdf" Sep 10 15:42:05.626850 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.626729 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d40147d-08e8-4362-b32a-8ee4c9727518-host-var-lib-cni-bin\") pod \"multus-8q52b\" (UID: \"2d40147d-08e8-4362-b32a-8ee4c9727518\") " pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.626850 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.626759 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b631218e-ec48-4b74-8e2b-e6047ceba20b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l8bwv\" (UID: \"b631218e-ec48-4b74-8e2b-e6047ceba20b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8bwv" Sep 10 15:42:05.626850 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.626770 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c2384bdb-90cf-49a6-885a-42f96018fb04-etc-sysconfig\") pod \"tuned-zdqdf\" (UID: \"c2384bdb-90cf-49a6-885a-42f96018fb04\") " pod="openshift-cluster-node-tuning-operator/tuned-zdqdf" Sep 10 15:42:05.626850 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.626833 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b631218e-ec48-4b74-8e2b-e6047ceba20b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l8bwv\" (UID: \"b631218e-ec48-4b74-8e2b-e6047ceba20b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8bwv" Sep 10 15:42:05.626850 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.626837 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d40147d-08e8-4362-b32a-8ee4c9727518-host-var-lib-cni-bin\") pod \"multus-8q52b\" (UID: \"2d40147d-08e8-4362-b32a-8ee4c9727518\") " pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.627719 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.626871 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2384bdb-90cf-49a6-885a-42f96018fb04-etc-kubernetes\") pod \"tuned-zdqdf\" (UID: \"c2384bdb-90cf-49a6-885a-42f96018fb04\") " pod="openshift-cluster-node-tuning-operator/tuned-zdqdf" Sep 10 15:42:05.627719 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.626891 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c2384bdb-90cf-49a6-885a-42f96018fb04-etc-modprobe-d\") pod \"tuned-zdqdf\" (UID: \"c2384bdb-90cf-49a6-885a-42f96018fb04\") " pod="openshift-cluster-node-tuning-operator/tuned-zdqdf" Sep 10 15:42:05.627719 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.626912 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/bdf61327-2614-4068-974c-a5c0f7c8e3e0-dbus\") pod \"global-pull-secret-syncer-swxpd\" (UID: \"bdf61327-2614-4068-974c-a5c0f7c8e3e0\") " pod="kube-system/global-pull-secret-syncer-swxpd" Sep 10 15:42:05.627719 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.626896 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c2384bdb-90cf-49a6-885a-42f96018fb04-etc-tuned\") pod \"tuned-zdqdf\" (UID: \"c2384bdb-90cf-49a6-885a-42f96018fb04\") " pod="openshift-cluster-node-tuning-operator/tuned-zdqdf" Sep 10 15:42:05.627719 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.626930 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c2384bdb-90cf-49a6-885a-42f96018fb04-etc-sysctl-conf\") pod \"tuned-zdqdf\" (UID: \"c2384bdb-90cf-49a6-885a-42f96018fb04\") " pod="openshift-cluster-node-tuning-operator/tuned-zdqdf" Sep 10 15:42:05.627719 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.626944 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c2384bdb-90cf-49a6-885a-42f96018fb04-etc-kubernetes\") pod \"tuned-zdqdf\" (UID: \"c2384bdb-90cf-49a6-885a-42f96018fb04\") " pod="openshift-cluster-node-tuning-operator/tuned-zdqdf" Sep 10 15:42:05.627719 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.626947 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2d40147d-08e8-4362-b32a-8ee4c9727518-host-var-lib-cni-multus\") pod \"multus-8q52b\" (UID: \"2d40147d-08e8-4362-b32a-8ee4c9727518\") " pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.627719 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.626981 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b068e03b-cd04-497c-81bf-1a8e4546dc77-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qgbvz\" (UID: \"b068e03b-cd04-497c-81bf-1a8e4546dc77\") " pod="openshift-multus/multus-additional-cni-plugins-qgbvz" Sep 10 15:42:05.627719 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.626993 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2384bdb-90cf-49a6-885a-42f96018fb04-var-lib-kubelet\") pod \"tuned-zdqdf\" (UID: \"c2384bdb-90cf-49a6-885a-42f96018fb04\") " pod="openshift-cluster-node-tuning-operator/tuned-zdqdf" Sep 10 15:42:05.627719 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.626988 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2d40147d-08e8-4362-b32a-8ee4c9727518-host-var-lib-cni-multus\") pod \"multus-8q52b\" (UID: \"2d40147d-08e8-4362-b32a-8ee4c9727518\") " pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.627719 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.627022 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bdf61327-2614-4068-974c-a5c0f7c8e3e0-kubelet-config\") pod \"global-pull-secret-syncer-swxpd\" (UID: \"bdf61327-2614-4068-974c-a5c0f7c8e3e0\") " pod="kube-system/global-pull-secret-syncer-swxpd" Sep 10 15:42:05.627719 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.627048 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdf61327-2614-4068-974c-a5c0f7c8e3e0-original-pull-secret\") pod \"global-pull-secret-syncer-swxpd\" (UID: \"bdf61327-2614-4068-974c-a5c0f7c8e3e0\") " pod="kube-system/global-pull-secret-syncer-swxpd" Sep 10 15:42:05.627719 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.627054 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c2384bdb-90cf-49a6-885a-42f96018fb04-var-lib-kubelet\") pod \"tuned-zdqdf\" (UID: \"c2384bdb-90cf-49a6-885a-42f96018fb04\") " pod="openshift-cluster-node-tuning-operator/tuned-zdqdf" Sep 10 15:42:05.627719 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.627074 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7jw8\" (UniqueName: \"kubernetes.io/projected/bcf35967-2a15-4bd4-8837-a058d81d1aff-kube-api-access-w7jw8\") pod \"network-metrics-daemon-qztq2\" (UID: \"bcf35967-2a15-4bd4-8837-a058d81d1aff\") " pod="openshift-multus/network-metrics-daemon-qztq2" Sep 10 15:42:05.627719 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.627099 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b631218e-ec48-4b74-8e2b-e6047ceba20b-etc-selinux\") pod \"aws-ebs-csi-driver-node-l8bwv\" (UID: \"b631218e-ec48-4b74-8e2b-e6047ceba20b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8bwv" Sep 10 15:42:05.627719 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.627112 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/bdf61327-2614-4068-974c-a5c0f7c8e3e0-kubelet-config\") pod \"global-pull-secret-syncer-swxpd\" (UID: \"bdf61327-2614-4068-974c-a5c0f7c8e3e0\") " pod="kube-system/global-pull-secret-syncer-swxpd" Sep 10 15:42:05.627719 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.627126 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7dc7aa02-57e8-4cdc-aab6-9875e1427825-serviceca\") pod \"node-ca-5lpqq\" (UID: \"7dc7aa02-57e8-4cdc-aab6-9875e1427825\") " pod="openshift-image-registry/node-ca-5lpqq" Sep 10 15:42:05.627719 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:05.627150 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 15:42:05.628643 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.627151 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6gx9\" (UniqueName: \"kubernetes.io/projected/7dc7aa02-57e8-4cdc-aab6-9875e1427825-kube-api-access-r6gx9\") pod \"node-ca-5lpqq\" (UID: \"7dc7aa02-57e8-4cdc-aab6-9875e1427825\") " pod="openshift-image-registry/node-ca-5lpqq" Sep 10 15:42:05.628643 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.627214 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b631218e-ec48-4b74-8e2b-e6047ceba20b-etc-selinux\") pod \"aws-ebs-csi-driver-node-l8bwv\" (UID: \"b631218e-ec48-4b74-8e2b-e6047ceba20b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8bwv" Sep 10 15:42:05.628643 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.627212 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b068e03b-cd04-497c-81bf-1a8e4546dc77-cni-binary-copy\") pod \"multus-additional-cni-plugins-qgbvz\" (UID: \"b068e03b-cd04-497c-81bf-1a8e4546dc77\") " pod="openshift-multus/multus-additional-cni-plugins-qgbvz" Sep 10 15:42:05.628643 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:05.627251 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdf61327-2614-4068-974c-a5c0f7c8e3e0-original-pull-secret podName:bdf61327-2614-4068-974c-a5c0f7c8e3e0 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:06.127209062 +0000 UTC m=+3.102469003 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bdf61327-2614-4068-974c-a5c0f7c8e3e0-original-pull-secret") pod "global-pull-secret-syncer-swxpd" (UID: "bdf61327-2614-4068-974c-a5c0f7c8e3e0") : object "kube-system"/"original-pull-secret" not registered Sep 10 15:42:05.628643 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.627342 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2384bdb-90cf-49a6-885a-42f96018fb04-sys\") pod \"tuned-zdqdf\" (UID: \"c2384bdb-90cf-49a6-885a-42f96018fb04\") " pod="openshift-cluster-node-tuning-operator/tuned-zdqdf" Sep 10 15:42:05.628643 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.627375 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c2384bdb-90cf-49a6-885a-42f96018fb04-run\") pod \"tuned-zdqdf\" (UID: \"c2384bdb-90cf-49a6-885a-42f96018fb04\") " pod="openshift-cluster-node-tuning-operator/tuned-zdqdf" Sep 10 15:42:05.628643 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.627420 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d40147d-08e8-4362-b32a-8ee4c9727518-host-run-netns\") pod \"multus-8q52b\" (UID: \"2d40147d-08e8-4362-b32a-8ee4c9727518\") " pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.628643 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.627448 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2d40147d-08e8-4362-b32a-8ee4c9727518-multus-daemon-config\") pod \"multus-8q52b\" (UID: \"2d40147d-08e8-4362-b32a-8ee4c9727518\") " pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.628643 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.627470 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c2384bdb-90cf-49a6-885a-42f96018fb04-sys\") pod \"tuned-zdqdf\" (UID: \"c2384bdb-90cf-49a6-885a-42f96018fb04\") " pod="openshift-cluster-node-tuning-operator/tuned-zdqdf" Sep 10 15:42:05.628643 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.627477 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2d40147d-08e8-4362-b32a-8ee4c9727518-host-run-multus-certs\") pod \"multus-8q52b\" (UID: \"2d40147d-08e8-4362-b32a-8ee4c9727518\") " pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.628643 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.627518 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2384bdb-90cf-49a6-885a-42f96018fb04-host\") pod \"tuned-zdqdf\" (UID: \"c2384bdb-90cf-49a6-885a-42f96018fb04\") " pod="openshift-cluster-node-tuning-operator/tuned-zdqdf" Sep 10 15:42:05.628643 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.627471 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c2384bdb-90cf-49a6-885a-42f96018fb04-run\") pod \"tuned-zdqdf\" (UID: \"c2384bdb-90cf-49a6-885a-42f96018fb04\") " pod="openshift-cluster-node-tuning-operator/tuned-zdqdf" Sep 10 15:42:05.628643 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.627526 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2d40147d-08e8-4362-b32a-8ee4c9727518-host-run-multus-certs\") pod \"multus-8q52b\" (UID: \"2d40147d-08e8-4362-b32a-8ee4c9727518\") " pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.628643 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.627522 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d40147d-08e8-4362-b32a-8ee4c9727518-host-run-netns\") pod \"multus-8q52b\" (UID: \"2d40147d-08e8-4362-b32a-8ee4c9727518\") " pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.628643 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.627650 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c2384bdb-90cf-49a6-885a-42f96018fb04-host\") pod \"tuned-zdqdf\" (UID: \"c2384bdb-90cf-49a6-885a-42f96018fb04\") " pod="openshift-cluster-node-tuning-operator/tuned-zdqdf" Sep 10 15:42:05.628643 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.627686 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d40147d-08e8-4362-b32a-8ee4c9727518-cni-binary-copy\") pod \"multus-8q52b\" (UID: \"2d40147d-08e8-4362-b32a-8ee4c9727518\") " pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.628643 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.627712 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7dc7aa02-57e8-4cdc-aab6-9875e1427825-host\") pod \"node-ca-5lpqq\" (UID: \"7dc7aa02-57e8-4cdc-aab6-9875e1427825\") " pod="openshift-image-registry/node-ca-5lpqq" Sep 10 15:42:05.629597 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.627728 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7dc7aa02-57e8-4cdc-aab6-9875e1427825-serviceca\") pod \"node-ca-5lpqq\" (UID: \"7dc7aa02-57e8-4cdc-aab6-9875e1427825\") " pod="openshift-image-registry/node-ca-5lpqq" Sep 10 15:42:05.629597 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.627742 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2d40147d-08e8-4362-b32a-8ee4c9727518-multus-socket-dir-parent\") pod \"multus-8q52b\" (UID: \"2d40147d-08e8-4362-b32a-8ee4c9727518\") " pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.629597 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.627767 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b068e03b-cd04-497c-81bf-1a8e4546dc77-cni-binary-copy\") pod \"multus-additional-cni-plugins-qgbvz\" (UID: \"b068e03b-cd04-497c-81bf-1a8e4546dc77\") " pod="openshift-multus/multus-additional-cni-plugins-qgbvz" Sep 10 15:42:05.629597 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.627782 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2d40147d-08e8-4362-b32a-8ee4c9727518-hostroot\") pod \"multus-8q52b\" (UID: \"2d40147d-08e8-4362-b32a-8ee4c9727518\") " pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.629597 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.627794 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2d40147d-08e8-4362-b32a-8ee4c9727518-multus-socket-dir-parent\") pod \"multus-8q52b\" (UID: \"2d40147d-08e8-4362-b32a-8ee4c9727518\") " pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.629597 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.627814 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q5c4\" (UniqueName: \"kubernetes.io/projected/d88aeb8f-b005-4deb-bc50-5f843a53699c-kube-api-access-2q5c4\") pod \"network-check-target-wqgsp\" (UID: \"d88aeb8f-b005-4deb-bc50-5f843a53699c\") " pod="openshift-network-diagnostics/network-check-target-wqgsp" Sep 10 15:42:05.629597 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.627821 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2d40147d-08e8-4362-b32a-8ee4c9727518-hostroot\") pod \"multus-8q52b\" (UID: \"2d40147d-08e8-4362-b32a-8ee4c9727518\") " pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.629597 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.627815 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7dc7aa02-57e8-4cdc-aab6-9875e1427825-host\") pod \"node-ca-5lpqq\" (UID: \"7dc7aa02-57e8-4cdc-aab6-9875e1427825\") " pod="openshift-image-registry/node-ca-5lpqq" Sep 10 15:42:05.629597 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.627849 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b631218e-ec48-4b74-8e2b-e6047ceba20b-registration-dir\") pod \"aws-ebs-csi-driver-node-l8bwv\" (UID: \"b631218e-ec48-4b74-8e2b-e6047ceba20b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8bwv" Sep 10 15:42:05.629597 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.627868 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2384bdb-90cf-49a6-885a-42f96018fb04-tmp\") pod \"tuned-zdqdf\" (UID: \"c2384bdb-90cf-49a6-885a-42f96018fb04\") " pod="openshift-cluster-node-tuning-operator/tuned-zdqdf" Sep 10 15:42:05.629597 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.627885 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d40147d-08e8-4362-b32a-8ee4c9727518-etc-kubernetes\") pod \"multus-8q52b\" (UID: \"2d40147d-08e8-4362-b32a-8ee4c9727518\") " pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.629597 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.627912 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d40147d-08e8-4362-b32a-8ee4c9727518-cnibin\") pod \"multus-8q52b\" (UID: \"2d40147d-08e8-4362-b32a-8ee4c9727518\") " pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.629597 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.627936 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b631218e-ec48-4b74-8e2b-e6047ceba20b-registration-dir\") pod \"aws-ebs-csi-driver-node-l8bwv\" (UID: \"b631218e-ec48-4b74-8e2b-e6047ceba20b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8bwv" Sep 10 15:42:05.629597 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.627938 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcf35967-2a15-4bd4-8837-a058d81d1aff-metrics-certs\") pod \"network-metrics-daemon-qztq2\" (UID: \"bcf35967-2a15-4bd4-8837-a058d81d1aff\") " pod="openshift-multus/network-metrics-daemon-qztq2" Sep 10 15:42:05.629597 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.627977 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tggbc\" (UniqueName: \"kubernetes.io/projected/b631218e-ec48-4b74-8e2b-e6047ceba20b-kube-api-access-tggbc\") pod \"aws-ebs-csi-driver-node-l8bwv\" (UID: \"b631218e-ec48-4b74-8e2b-e6047ceba20b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8bwv" Sep 10 15:42:05.629597 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.627984 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d40147d-08e8-4362-b32a-8ee4c9727518-etc-kubernetes\") pod \"multus-8q52b\" (UID: \"2d40147d-08e8-4362-b32a-8ee4c9727518\") " pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.629597 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.628008 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b068e03b-cd04-497c-81bf-1a8e4546dc77-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qgbvz\" (UID: \"b068e03b-cd04-497c-81bf-1a8e4546dc77\") " pod="openshift-multus/multus-additional-cni-plugins-qgbvz" Sep 10 15:42:05.629597 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.628033 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2d40147d-08e8-4362-b32a-8ee4c9727518-cnibin\") pod \"multus-8q52b\" (UID: \"2d40147d-08e8-4362-b32a-8ee4c9727518\") " pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.630297 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.628045 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c2384bdb-90cf-49a6-885a-42f96018fb04-etc-systemd\") pod \"tuned-zdqdf\" (UID: \"c2384bdb-90cf-49a6-885a-42f96018fb04\") " pod="openshift-cluster-node-tuning-operator/tuned-zdqdf" Sep 10 15:42:05.630297 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.628073 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d40147d-08e8-4362-b32a-8ee4c9727518-host-var-lib-kubelet\") pod \"multus-8q52b\" (UID: \"2d40147d-08e8-4362-b32a-8ee4c9727518\") " pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.630297 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.628074 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2d40147d-08e8-4362-b32a-8ee4c9727518-multus-daemon-config\") pod \"multus-8q52b\" (UID: \"2d40147d-08e8-4362-b32a-8ee4c9727518\") " pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.630297 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.628099 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c2384bdb-90cf-49a6-885a-42f96018fb04-etc-sysctl-d\") pod \"tuned-zdqdf\" (UID: \"c2384bdb-90cf-49a6-885a-42f96018fb04\") " pod="openshift-cluster-node-tuning-operator/tuned-zdqdf" Sep 10 15:42:05.630297 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.628149 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c2384bdb-90cf-49a6-885a-42f96018fb04-etc-systemd\") pod \"tuned-zdqdf\" (UID: \"c2384bdb-90cf-49a6-885a-42f96018fb04\") " pod="openshift-cluster-node-tuning-operator/tuned-zdqdf" Sep 10 15:42:05.630297 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.628167 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b068e03b-cd04-497c-81bf-1a8e4546dc77-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qgbvz\" (UID: \"b068e03b-cd04-497c-81bf-1a8e4546dc77\") " pod="openshift-multus/multus-additional-cni-plugins-qgbvz" Sep 10 15:42:05.630297 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.628183 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkc95\" (UniqueName: \"kubernetes.io/projected/2d40147d-08e8-4362-b32a-8ee4c9727518-kube-api-access-pkc95\") pod \"multus-8q52b\" (UID: \"2d40147d-08e8-4362-b32a-8ee4c9727518\") " pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.630297 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.628191 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2d40147d-08e8-4362-b32a-8ee4c9727518-cni-binary-copy\") pod \"multus-8q52b\" (UID: \"2d40147d-08e8-4362-b32a-8ee4c9727518\") " pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.630297 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.628210 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c2384bdb-90cf-49a6-885a-42f96018fb04-etc-sysctl-d\") pod \"tuned-zdqdf\" (UID: \"c2384bdb-90cf-49a6-885a-42f96018fb04\") " pod="openshift-cluster-node-tuning-operator/tuned-zdqdf" Sep 10 15:42:05.630297 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.628218 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d40147d-08e8-4362-b32a-8ee4c9727518-host-var-lib-kubelet\") pod \"multus-8q52b\" (UID: \"2d40147d-08e8-4362-b32a-8ee4c9727518\") " pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.630297 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.628224 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c2384bdb-90cf-49a6-885a-42f96018fb04-lib-modules\") pod \"tuned-zdqdf\" (UID: \"c2384bdb-90cf-49a6-885a-42f96018fb04\") " pod="openshift-cluster-node-tuning-operator/tuned-zdqdf" Sep 10 15:42:05.630297 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.628243 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d40147d-08e8-4362-b32a-8ee4c9727518-multus-conf-dir\") pod \"multus-8q52b\" (UID: \"2d40147d-08e8-4362-b32a-8ee4c9727518\") " pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.630297 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.628260 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d40147d-08e8-4362-b32a-8ee4c9727518-multus-cni-dir\") pod \"multus-8q52b\" (UID: \"2d40147d-08e8-4362-b32a-8ee4c9727518\") " pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.630297 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.628289 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2d40147d-08e8-4362-b32a-8ee4c9727518-host-run-k8s-cni-cncf-io\") pod \"multus-8q52b\" (UID: \"2d40147d-08e8-4362-b32a-8ee4c9727518\") " pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.630297 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.628320 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2d40147d-08e8-4362-b32a-8ee4c9727518-multus-conf-dir\") pod \"multus-8q52b\" (UID: \"2d40147d-08e8-4362-b32a-8ee4c9727518\") " pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.630297 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.628349 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2d40147d-08e8-4362-b32a-8ee4c9727518-multus-cni-dir\") pod \"multus-8q52b\" (UID: \"2d40147d-08e8-4362-b32a-8ee4c9727518\") " pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.630297 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.628360 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2d40147d-08e8-4362-b32a-8ee4c9727518-host-run-k8s-cni-cncf-io\") pod \"multus-8q52b\" (UID: \"2d40147d-08e8-4362-b32a-8ee4c9727518\") " pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.630297 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.628439 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c2384bdb-90cf-49a6-885a-42f96018fb04-lib-modules\") pod \"tuned-zdqdf\" (UID: \"c2384bdb-90cf-49a6-885a-42f96018fb04\") " pod="openshift-cluster-node-tuning-operator/tuned-zdqdf" Sep 10 15:42:05.630954 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:05.628450 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:42:05.630954 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:05.628523 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcf35967-2a15-4bd4-8837-a058d81d1aff-metrics-certs podName:bcf35967-2a15-4bd4-8837-a058d81d1aff nodeName:}" failed. No retries permitted until 2026-09-10 15:42:06.128502928 +0000 UTC m=+3.103762876 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bcf35967-2a15-4bd4-8837-a058d81d1aff-metrics-certs") pod "network-metrics-daemon-qztq2" (UID: "bcf35967-2a15-4bd4-8837-a058d81d1aff") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:42:05.630954 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.628691 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b068e03b-cd04-497c-81bf-1a8e4546dc77-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qgbvz\" (UID: \"b068e03b-cd04-497c-81bf-1a8e4546dc77\") " pod="openshift-multus/multus-additional-cni-plugins-qgbvz" Sep 10 15:42:05.630954 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.629993 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c2384bdb-90cf-49a6-885a-42f96018fb04-etc-tuned\") pod \"tuned-zdqdf\" (UID: \"c2384bdb-90cf-49a6-885a-42f96018fb04\") " pod="openshift-cluster-node-tuning-operator/tuned-zdqdf" Sep 10 15:42:05.630954 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.630469 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c2384bdb-90cf-49a6-885a-42f96018fb04-tmp\") pod \"tuned-zdqdf\" (UID: \"c2384bdb-90cf-49a6-885a-42f96018fb04\") " pod="openshift-cluster-node-tuning-operator/tuned-zdqdf" Sep 10 15:42:05.639040 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.638973 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6gx9\" (UniqueName: \"kubernetes.io/projected/7dc7aa02-57e8-4cdc-aab6-9875e1427825-kube-api-access-r6gx9\") pod \"node-ca-5lpqq\" (UID: \"7dc7aa02-57e8-4cdc-aab6-9875e1427825\") " pod="openshift-image-registry/node-ca-5lpqq" Sep 10 15:42:05.639416 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:05.639364 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 15:42:05.639416 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:05.639393 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 15:42:05.639416 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:05.639410 2583 projected.go:196] Error preparing data for projected volume kube-api-access-2q5c4 for pod openshift-network-diagnostics/network-check-target-wqgsp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:42:05.639633 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:05.639496 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d88aeb8f-b005-4deb-bc50-5f843a53699c-kube-api-access-2q5c4 podName:d88aeb8f-b005-4deb-bc50-5f843a53699c nodeName:}" failed. No retries permitted until 2026-09-10 15:42:06.139477829 +0000 UTC m=+3.114737762 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2q5c4" (UniqueName: "kubernetes.io/projected/d88aeb8f-b005-4deb-bc50-5f843a53699c-kube-api-access-2q5c4") pod "network-check-target-wqgsp" (UID: "d88aeb8f-b005-4deb-bc50-5f843a53699c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:42:05.639633 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.639518 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgxp9\" (UniqueName: \"kubernetes.io/projected/c2384bdb-90cf-49a6-885a-42f96018fb04-kube-api-access-qgxp9\") pod \"tuned-zdqdf\" (UID: \"c2384bdb-90cf-49a6-885a-42f96018fb04\") " pod="openshift-cluster-node-tuning-operator/tuned-zdqdf" Sep 10 15:42:05.639633 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.639529 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s62b8\" (UniqueName: \"kubernetes.io/projected/b068e03b-cd04-497c-81bf-1a8e4546dc77-kube-api-access-s62b8\") pod \"multus-additional-cni-plugins-qgbvz\" (UID: \"b068e03b-cd04-497c-81bf-1a8e4546dc77\") " pod="openshift-multus/multus-additional-cni-plugins-qgbvz" Sep 10 15:42:05.639766 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.639646 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7jw8\" (UniqueName: \"kubernetes.io/projected/bcf35967-2a15-4bd4-8837-a058d81d1aff-kube-api-access-w7jw8\") pod \"network-metrics-daemon-qztq2\" (UID: \"bcf35967-2a15-4bd4-8837-a058d81d1aff\") " pod="openshift-multus/network-metrics-daemon-qztq2" Sep 10 15:42:05.641991 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.641966 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkc95\" (UniqueName: \"kubernetes.io/projected/2d40147d-08e8-4362-b32a-8ee4c9727518-kube-api-access-pkc95\") pod \"multus-8q52b\" (UID: \"2d40147d-08e8-4362-b32a-8ee4c9727518\") " pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.642094 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.642049 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tggbc\" (UniqueName: \"kubernetes.io/projected/b631218e-ec48-4b74-8e2b-e6047ceba20b-kube-api-access-tggbc\") pod \"aws-ebs-csi-driver-node-l8bwv\" (UID: \"b631218e-ec48-4b74-8e2b-e6047ceba20b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8bwv" Sep 10 15:42:05.722591 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.722550 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-m5t8c" Sep 10 15:42:05.732881 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.732858 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:05.743566 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.743539 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mcx2v" Sep 10 15:42:05.749330 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.749304 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8q52b" Sep 10 15:42:05.756978 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.756960 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qgbvz" Sep 10 15:42:05.764676 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.764657 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8bwv" Sep 10 15:42:05.770610 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.770587 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-zdqdf" Sep 10 15:42:05.777475 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.777457 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-5lpqq" Sep 10 15:42:05.787113 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.787085 2583 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 15:42:05.841901 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:05.841872 2583 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 10 15:42:06.131734 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:06.131655 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdf61327-2614-4068-974c-a5c0f7c8e3e0-original-pull-secret\") pod \"global-pull-secret-syncer-swxpd\" (UID: \"bdf61327-2614-4068-974c-a5c0f7c8e3e0\") " pod="kube-system/global-pull-secret-syncer-swxpd" Sep 10 15:42:06.131734 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:06.131720 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcf35967-2a15-4bd4-8837-a058d81d1aff-metrics-certs\") pod \"network-metrics-daemon-qztq2\" (UID: \"bcf35967-2a15-4bd4-8837-a058d81d1aff\") " pod="openshift-multus/network-metrics-daemon-qztq2" Sep 10 15:42:06.131977 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:06.131837 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:42:06.131977 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:06.131842 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 15:42:06.131977 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:06.131901 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcf35967-2a15-4bd4-8837-a058d81d1aff-metrics-certs podName:bcf35967-2a15-4bd4-8837-a058d81d1aff nodeName:}" failed. No retries permitted until 2026-09-10 15:42:07.131886282 +0000 UTC m=+4.107146215 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bcf35967-2a15-4bd4-8837-a058d81d1aff-metrics-certs") pod "network-metrics-daemon-qztq2" (UID: "bcf35967-2a15-4bd4-8837-a058d81d1aff") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:42:06.131977 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:06.131918 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdf61327-2614-4068-974c-a5c0f7c8e3e0-original-pull-secret podName:bdf61327-2614-4068-974c-a5c0f7c8e3e0 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:07.131908403 +0000 UTC m=+4.107168336 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bdf61327-2614-4068-974c-a5c0f7c8e3e0-original-pull-secret") pod "global-pull-secret-syncer-swxpd" (UID: "bdf61327-2614-4068-974c-a5c0f7c8e3e0") : object "kube-system"/"original-pull-secret" not registered Sep 10 15:42:06.232307 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:06.232267 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q5c4\" (UniqueName: \"kubernetes.io/projected/d88aeb8f-b005-4deb-bc50-5f843a53699c-kube-api-access-2q5c4\") pod \"network-check-target-wqgsp\" (UID: \"d88aeb8f-b005-4deb-bc50-5f843a53699c\") " pod="openshift-network-diagnostics/network-check-target-wqgsp" Sep 10 15:42:06.232488 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:06.232466 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 15:42:06.232552 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:06.232495 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 15:42:06.232552 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:06.232506 2583 projected.go:196] Error preparing data for projected volume kube-api-access-2q5c4 for pod openshift-network-diagnostics/network-check-target-wqgsp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:42:06.232630 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:06.232574 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d88aeb8f-b005-4deb-bc50-5f843a53699c-kube-api-access-2q5c4 podName:d88aeb8f-b005-4deb-bc50-5f843a53699c nodeName:}" failed. No retries permitted until 2026-09-10 15:42:07.232556325 +0000 UTC m=+4.207816281 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2q5c4" (UniqueName: "kubernetes.io/projected/d88aeb8f-b005-4deb-bc50-5f843a53699c-kube-api-access-2q5c4") pod "network-check-target-wqgsp" (UID: "d88aeb8f-b005-4deb-bc50-5f843a53699c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:42:06.461319 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:06.461246 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 15:37:05 +0000 UTC" deadline="2028-06-27 19:57:11.672696238 +0000 UTC" Sep 10 15:42:06.461319 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:06.461278 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15748h15m5.21142106s" Sep 10 15:42:06.503817 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:06.503788 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod621b9bdc_da4f_4833_a547_2bd2c40e83d1.slice/crio-4da751e696138f8461bcc6273106a1bd678c39cab54d3f49720d6cd9979c0744 WatchSource:0}: Error finding container 4da751e696138f8461bcc6273106a1bd678c39cab54d3f49720d6cd9979c0744: Status 404 returned error can't find the container with id 4da751e696138f8461bcc6273106a1bd678c39cab54d3f49720d6cd9979c0744 Sep 10 15:42:06.505217 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:06.504764 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb068e03b_cd04_497c_81bf_1a8e4546dc77.slice/crio-bb6f6c1cec1faa38f10d43c9b32b251806dc12a0df111634fe145d82f32c5122 WatchSource:0}: Error finding container bb6f6c1cec1faa38f10d43c9b32b251806dc12a0df111634fe145d82f32c5122: Status 404 returned error can't find the container with id bb6f6c1cec1faa38f10d43c9b32b251806dc12a0df111634fe145d82f32c5122 Sep 10 15:42:06.508925 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:06.508735 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swxpd" Sep 10 15:42:06.508925 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:06.508878 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swxpd" podUID="bdf61327-2614-4068-974c-a5c0f7c8e3e0" Sep 10 15:42:06.512614 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:06.512299 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3448a418_7ccc_4a41_b2f0_0e1fa12cd592.slice/crio-dd7b918ffa10b99297450b0295e899cf4d28c5d7a60ff388addba218dbd8978e WatchSource:0}: Error finding container dd7b918ffa10b99297450b0295e899cf4d28c5d7a60ff388addba218dbd8978e: Status 404 returned error can't find the container with id dd7b918ffa10b99297450b0295e899cf4d28c5d7a60ff388addba218dbd8978e Sep 10 15:42:06.515846 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:06.515821 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2384bdb_90cf_49a6_885a_42f96018fb04.slice/crio-4d482809900f1863741e20fdb0262f5fb0e5e468a33702380035d9f7ee831d38 WatchSource:0}: Error finding container 4d482809900f1863741e20fdb0262f5fb0e5e468a33702380035d9f7ee831d38: Status 404 returned error can't find the container with id 4d482809900f1863741e20fdb0262f5fb0e5e468a33702380035d9f7ee831d38 Sep 10 15:42:06.519749 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:06.519713 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zdqdf" event={"ID":"c2384bdb-90cf-49a6-885a-42f96018fb04","Type":"ContainerStarted","Data":"4d482809900f1863741e20fdb0262f5fb0e5e468a33702380035d9f7ee831d38"} Sep 10 15:42:06.521186 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:06.521155 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mcx2v" event={"ID":"3448a418-7ccc-4a41-b2f0-0e1fa12cd592","Type":"ContainerStarted","Data":"dd7b918ffa10b99297450b0295e899cf4d28c5d7a60ff388addba218dbd8978e"} Sep 10 15:42:06.522578 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:06.522550 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5lpqq" event={"ID":"7dc7aa02-57e8-4cdc-aab6-9875e1427825","Type":"ContainerStarted","Data":"57ff932006465a3f68652aa317e0b38ed7ce131168d2c48c300868f4172862b0"} Sep 10 15:42:06.524300 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:06.524273 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-m5t8c" event={"ID":"d339b125-a6fe-4bff-b264-281c40b162c7","Type":"ContainerStarted","Data":"6f9b912b9054bc7407a068848369f9333ec2da1b546fed1373e285dd6cad1371"} Sep 10 15:42:06.525622 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:06.525601 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qgbvz" event={"ID":"b068e03b-cd04-497c-81bf-1a8e4546dc77","Type":"ContainerStarted","Data":"bb6f6c1cec1faa38f10d43c9b32b251806dc12a0df111634fe145d82f32c5122"} Sep 10 15:42:06.526881 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:06.526852 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" event={"ID":"621b9bdc-da4f-4833-a547-2bd2c40e83d1","Type":"ContainerStarted","Data":"4da751e696138f8461bcc6273106a1bd678c39cab54d3f49720d6cd9979c0744"} Sep 10 15:42:06.528047 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:06.528025 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8bwv" event={"ID":"b631218e-ec48-4b74-8e2b-e6047ceba20b","Type":"ContainerStarted","Data":"c027424688d8976e46a2a274354174f9f577112ef48dc9c69f23b99e327fb534"} Sep 10 15:42:07.139069 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:07.138731 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdf61327-2614-4068-974c-a5c0f7c8e3e0-original-pull-secret\") pod \"global-pull-secret-syncer-swxpd\" (UID: \"bdf61327-2614-4068-974c-a5c0f7c8e3e0\") " pod="kube-system/global-pull-secret-syncer-swxpd" Sep 10 15:42:07.139069 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:07.139035 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 15:42:07.139297 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:07.139087 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcf35967-2a15-4bd4-8837-a058d81d1aff-metrics-certs\") pod \"network-metrics-daemon-qztq2\" (UID: \"bcf35967-2a15-4bd4-8837-a058d81d1aff\") " pod="openshift-multus/network-metrics-daemon-qztq2" Sep 10 15:42:07.139297 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:07.139119 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdf61327-2614-4068-974c-a5c0f7c8e3e0-original-pull-secret podName:bdf61327-2614-4068-974c-a5c0f7c8e3e0 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:09.139095202 +0000 UTC m=+6.114355186 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bdf61327-2614-4068-974c-a5c0f7c8e3e0-original-pull-secret") pod "global-pull-secret-syncer-swxpd" (UID: "bdf61327-2614-4068-974c-a5c0f7c8e3e0") : object "kube-system"/"original-pull-secret" not registered Sep 10 15:42:07.139297 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:07.139190 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:42:07.139297 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:07.139246 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcf35967-2a15-4bd4-8837-a058d81d1aff-metrics-certs podName:bcf35967-2a15-4bd4-8837-a058d81d1aff nodeName:}" failed. No retries permitted until 2026-09-10 15:42:09.139228499 +0000 UTC m=+6.114488444 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bcf35967-2a15-4bd4-8837-a058d81d1aff-metrics-certs") pod "network-metrics-daemon-qztq2" (UID: "bcf35967-2a15-4bd4-8837-a058d81d1aff") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:42:07.240322 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:07.240279 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q5c4\" (UniqueName: \"kubernetes.io/projected/d88aeb8f-b005-4deb-bc50-5f843a53699c-kube-api-access-2q5c4\") pod \"network-check-target-wqgsp\" (UID: \"d88aeb8f-b005-4deb-bc50-5f843a53699c\") " pod="openshift-network-diagnostics/network-check-target-wqgsp" Sep 10 15:42:07.240530 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:07.240511 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 15:42:07.240606 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:07.240538 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 15:42:07.240606 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:07.240552 2583 projected.go:196] Error preparing data for projected volume kube-api-access-2q5c4 for pod openshift-network-diagnostics/network-check-target-wqgsp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:42:07.240714 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:07.240613 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d88aeb8f-b005-4deb-bc50-5f843a53699c-kube-api-access-2q5c4 podName:d88aeb8f-b005-4deb-bc50-5f843a53699c nodeName:}" failed. No retries permitted until 2026-09-10 15:42:09.240594077 +0000 UTC m=+6.215854026 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2q5c4" (UniqueName: "kubernetes.io/projected/d88aeb8f-b005-4deb-bc50-5f843a53699c-kube-api-access-2q5c4") pod "network-check-target-wqgsp" (UID: "d88aeb8f-b005-4deb-bc50-5f843a53699c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:42:07.461923 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:07.461776 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 15:37:05 +0000 UTC" deadline="2028-02-29 18:21:25.583152125 +0000 UTC" Sep 10 15:42:07.461923 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:07.461838 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12890h39m18.121318085s" Sep 10 15:42:07.511481 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:07.511192 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qztq2" Sep 10 15:42:07.511481 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:07.511337 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qztq2" podUID="bcf35967-2a15-4bd4-8837-a058d81d1aff" Sep 10 15:42:07.511481 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:07.511441 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqgsp" Sep 10 15:42:07.512061 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:07.511991 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqgsp" podUID="d88aeb8f-b005-4deb-bc50-5f843a53699c" Sep 10 15:42:07.539392 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:07.539360 2583 generic.go:356] "Generic (PLEG): container finished" podID="35f373b97584485ab0c1f517fb075548" containerID="9470931ac1fe88677ecb5febaa553f0e081b95c0672cd9539ab4e229e966b063" exitCode=0 Sep 10 15:42:07.539560 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:07.539446 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal" event={"ID":"35f373b97584485ab0c1f517fb075548","Type":"ContainerDied","Data":"9470931ac1fe88677ecb5febaa553f0e081b95c0672cd9539ab4e229e966b063"} Sep 10 15:42:07.546525 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:07.546491 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-107.ec2.internal" event={"ID":"ef65ebb74ba3516cc857fe0f9e4cef0a","Type":"ContainerStarted","Data":"79662230c0bd22d9bf55a85eaa96b809599d12bca12c4b0fe28cd885e4b0faff"} Sep 10 15:42:07.553869 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:07.553844 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8q52b" event={"ID":"2d40147d-08e8-4362-b32a-8ee4c9727518","Type":"ContainerStarted","Data":"84c78ba86766686ed57187f80ea3bbe8e69faf1bba5915e939d1f4e06f6790c6"} Sep 10 15:42:08.509709 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:08.509681 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swxpd" Sep 10 15:42:08.510150 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:08.509808 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swxpd" podUID="bdf61327-2614-4068-974c-a5c0f7c8e3e0" Sep 10 15:42:08.571478 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:08.571443 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal" event={"ID":"35f373b97584485ab0c1f517fb075548","Type":"ContainerStarted","Data":"fc1d93a79b83da6e1b0e5e93ccf35bfd543dc45eb7c0952e511464592914f489"} Sep 10 15:42:08.609781 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:08.603422 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-107.ec2.internal" podStartSLOduration=4.603400809 podStartE2EDuration="4.603400809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:07.56926238 +0000 UTC m=+4.544522337" watchObservedRunningTime="2026-09-10 15:42:08.603400809 +0000 UTC m=+5.578660778" Sep 10 15:42:09.160260 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:09.159515 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdf61327-2614-4068-974c-a5c0f7c8e3e0-original-pull-secret\") pod \"global-pull-secret-syncer-swxpd\" (UID: \"bdf61327-2614-4068-974c-a5c0f7c8e3e0\") " pod="kube-system/global-pull-secret-syncer-swxpd" Sep 10 15:42:09.160260 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:09.159586 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcf35967-2a15-4bd4-8837-a058d81d1aff-metrics-certs\") pod \"network-metrics-daemon-qztq2\" (UID: \"bcf35967-2a15-4bd4-8837-a058d81d1aff\") " pod="openshift-multus/network-metrics-daemon-qztq2" Sep 10 15:42:09.160260 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:09.159717 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:42:09.160260 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:09.159777 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcf35967-2a15-4bd4-8837-a058d81d1aff-metrics-certs podName:bcf35967-2a15-4bd4-8837-a058d81d1aff nodeName:}" failed. No retries permitted until 2026-09-10 15:42:13.159759788 +0000 UTC m=+10.135019731 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bcf35967-2a15-4bd4-8837-a058d81d1aff-metrics-certs") pod "network-metrics-daemon-qztq2" (UID: "bcf35967-2a15-4bd4-8837-a058d81d1aff") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:42:09.160260 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:09.160131 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 15:42:09.160260 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:09.160177 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdf61327-2614-4068-974c-a5c0f7c8e3e0-original-pull-secret podName:bdf61327-2614-4068-974c-a5c0f7c8e3e0 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:13.160163108 +0000 UTC m=+10.135423057 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bdf61327-2614-4068-974c-a5c0f7c8e3e0-original-pull-secret") pod "global-pull-secret-syncer-swxpd" (UID: "bdf61327-2614-4068-974c-a5c0f7c8e3e0") : object "kube-system"/"original-pull-secret" not registered Sep 10 15:42:09.260432 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:09.260399 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q5c4\" (UniqueName: \"kubernetes.io/projected/d88aeb8f-b005-4deb-bc50-5f843a53699c-kube-api-access-2q5c4\") pod \"network-check-target-wqgsp\" (UID: \"d88aeb8f-b005-4deb-bc50-5f843a53699c\") " pod="openshift-network-diagnostics/network-check-target-wqgsp" Sep 10 15:42:09.260617 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:09.260606 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 15:42:09.260692 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:09.260624 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 15:42:09.260692 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:09.260636 2583 projected.go:196] Error preparing data for projected volume kube-api-access-2q5c4 for pod openshift-network-diagnostics/network-check-target-wqgsp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:42:09.260798 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:09.260697 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d88aeb8f-b005-4deb-bc50-5f843a53699c-kube-api-access-2q5c4 podName:d88aeb8f-b005-4deb-bc50-5f843a53699c nodeName:}" failed. No retries permitted until 2026-09-10 15:42:13.260679757 +0000 UTC m=+10.235939694 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2q5c4" (UniqueName: "kubernetes.io/projected/d88aeb8f-b005-4deb-bc50-5f843a53699c-kube-api-access-2q5c4") pod "network-check-target-wqgsp" (UID: "d88aeb8f-b005-4deb-bc50-5f843a53699c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:42:09.511449 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:09.511361 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qztq2" Sep 10 15:42:09.511892 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:09.511517 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qztq2" podUID="bcf35967-2a15-4bd4-8837-a058d81d1aff" Sep 10 15:42:09.511892 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:09.511699 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqgsp" Sep 10 15:42:09.511892 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:09.511798 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqgsp" podUID="d88aeb8f-b005-4deb-bc50-5f843a53699c" Sep 10 15:42:10.509209 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:10.509178 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swxpd" Sep 10 15:42:10.509417 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:10.509317 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swxpd" podUID="bdf61327-2614-4068-974c-a5c0f7c8e3e0" Sep 10 15:42:11.509801 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:11.509705 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qztq2" Sep 10 15:42:11.510234 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:11.509705 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqgsp" Sep 10 15:42:11.510234 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:11.509850 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qztq2" podUID="bcf35967-2a15-4bd4-8837-a058d81d1aff" Sep 10 15:42:11.510234 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:11.509912 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqgsp" podUID="d88aeb8f-b005-4deb-bc50-5f843a53699c" Sep 10 15:42:12.509778 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:12.509698 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swxpd" Sep 10 15:42:12.509915 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:12.509826 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swxpd" podUID="bdf61327-2614-4068-974c-a5c0f7c8e3e0" Sep 10 15:42:13.192434 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:13.192242 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcf35967-2a15-4bd4-8837-a058d81d1aff-metrics-certs\") pod \"network-metrics-daemon-qztq2\" (UID: \"bcf35967-2a15-4bd4-8837-a058d81d1aff\") " pod="openshift-multus/network-metrics-daemon-qztq2" Sep 10 15:42:13.192434 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:13.192329 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdf61327-2614-4068-974c-a5c0f7c8e3e0-original-pull-secret\") pod \"global-pull-secret-syncer-swxpd\" (UID: \"bdf61327-2614-4068-974c-a5c0f7c8e3e0\") " pod="kube-system/global-pull-secret-syncer-swxpd" Sep 10 15:42:13.192657 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:13.192507 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:42:13.192657 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:13.192589 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcf35967-2a15-4bd4-8837-a058d81d1aff-metrics-certs podName:bcf35967-2a15-4bd4-8837-a058d81d1aff nodeName:}" failed. No retries permitted until 2026-09-10 15:42:21.192568612 +0000 UTC m=+18.167828562 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bcf35967-2a15-4bd4-8837-a058d81d1aff-metrics-certs") pod "network-metrics-daemon-qztq2" (UID: "bcf35967-2a15-4bd4-8837-a058d81d1aff") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:42:13.192853 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:13.192820 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 15:42:13.192940 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:13.192884 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdf61327-2614-4068-974c-a5c0f7c8e3e0-original-pull-secret podName:bdf61327-2614-4068-974c-a5c0f7c8e3e0 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:21.192867735 +0000 UTC m=+18.168127681 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bdf61327-2614-4068-974c-a5c0f7c8e3e0-original-pull-secret") pod "global-pull-secret-syncer-swxpd" (UID: "bdf61327-2614-4068-974c-a5c0f7c8e3e0") : object "kube-system"/"original-pull-secret" not registered Sep 10 15:42:13.293515 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:13.292807 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q5c4\" (UniqueName: \"kubernetes.io/projected/d88aeb8f-b005-4deb-bc50-5f843a53699c-kube-api-access-2q5c4\") pod \"network-check-target-wqgsp\" (UID: \"d88aeb8f-b005-4deb-bc50-5f843a53699c\") " pod="openshift-network-diagnostics/network-check-target-wqgsp" Sep 10 15:42:13.293515 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:13.292988 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 15:42:13.293515 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:13.293011 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 15:42:13.293515 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:13.293026 2583 projected.go:196] Error preparing data for projected volume kube-api-access-2q5c4 for pod openshift-network-diagnostics/network-check-target-wqgsp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:42:13.293515 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:13.293129 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d88aeb8f-b005-4deb-bc50-5f843a53699c-kube-api-access-2q5c4 podName:d88aeb8f-b005-4deb-bc50-5f843a53699c nodeName:}" failed. No retries permitted until 2026-09-10 15:42:21.293068343 +0000 UTC m=+18.268328282 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2q5c4" (UniqueName: "kubernetes.io/projected/d88aeb8f-b005-4deb-bc50-5f843a53699c-kube-api-access-2q5c4") pod "network-check-target-wqgsp" (UID: "d88aeb8f-b005-4deb-bc50-5f843a53699c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:42:13.509443 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:13.509335 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qztq2" Sep 10 15:42:13.510772 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:13.510263 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qztq2" podUID="bcf35967-2a15-4bd4-8837-a058d81d1aff" Sep 10 15:42:13.510772 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:13.510635 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqgsp" Sep 10 15:42:13.510772 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:13.510728 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqgsp" podUID="d88aeb8f-b005-4deb-bc50-5f843a53699c" Sep 10 15:42:14.509266 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:14.509071 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swxpd" Sep 10 15:42:14.509266 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:14.509204 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swxpd" podUID="bdf61327-2614-4068-974c-a5c0f7c8e3e0" Sep 10 15:42:15.510596 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:15.510136 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqgsp" Sep 10 15:42:15.510596 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:15.510311 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqgsp" podUID="d88aeb8f-b005-4deb-bc50-5f843a53699c" Sep 10 15:42:15.511114 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:15.511077 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qztq2" Sep 10 15:42:15.511394 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:15.511351 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qztq2" podUID="bcf35967-2a15-4bd4-8837-a058d81d1aff" Sep 10 15:42:16.509623 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:16.509596 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swxpd" Sep 10 15:42:16.509780 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:16.509702 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swxpd" podUID="bdf61327-2614-4068-974c-a5c0f7c8e3e0" Sep 10 15:42:17.509607 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:17.509574 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qztq2" Sep 10 15:42:17.510007 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:17.509783 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qztq2" podUID="bcf35967-2a15-4bd4-8837-a058d81d1aff" Sep 10 15:42:17.510007 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:17.509840 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqgsp" Sep 10 15:42:17.510007 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:17.509938 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqgsp" podUID="d88aeb8f-b005-4deb-bc50-5f843a53699c" Sep 10 15:42:18.509425 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:18.509393 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swxpd" Sep 10 15:42:18.509599 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:18.509539 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swxpd" podUID="bdf61327-2614-4068-974c-a5c0f7c8e3e0" Sep 10 15:42:19.509090 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:19.509057 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqgsp" Sep 10 15:42:19.509567 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:19.509063 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qztq2" Sep 10 15:42:19.509567 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:19.509178 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqgsp" podUID="d88aeb8f-b005-4deb-bc50-5f843a53699c" Sep 10 15:42:19.509567 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:19.509237 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qztq2" podUID="bcf35967-2a15-4bd4-8837-a058d81d1aff" Sep 10 15:42:20.508970 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:20.508881 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swxpd" Sep 10 15:42:20.509155 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:20.509003 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swxpd" podUID="bdf61327-2614-4068-974c-a5c0f7c8e3e0" Sep 10 15:42:21.255121 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:21.255074 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcf35967-2a15-4bd4-8837-a058d81d1aff-metrics-certs\") pod \"network-metrics-daemon-qztq2\" (UID: \"bcf35967-2a15-4bd4-8837-a058d81d1aff\") " pod="openshift-multus/network-metrics-daemon-qztq2" Sep 10 15:42:21.255291 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:21.255152 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdf61327-2614-4068-974c-a5c0f7c8e3e0-original-pull-secret\") pod \"global-pull-secret-syncer-swxpd\" (UID: \"bdf61327-2614-4068-974c-a5c0f7c8e3e0\") " pod="kube-system/global-pull-secret-syncer-swxpd" Sep 10 15:42:21.255291 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:21.255217 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:42:21.255291 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:21.255272 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 15:42:21.255291 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:21.255287 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcf35967-2a15-4bd4-8837-a058d81d1aff-metrics-certs podName:bcf35967-2a15-4bd4-8837-a058d81d1aff nodeName:}" failed. No retries permitted until 2026-09-10 15:42:37.255265197 +0000 UTC m=+34.230525153 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bcf35967-2a15-4bd4-8837-a058d81d1aff-metrics-certs") pod "network-metrics-daemon-qztq2" (UID: "bcf35967-2a15-4bd4-8837-a058d81d1aff") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:42:21.255523 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:21.255333 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdf61327-2614-4068-974c-a5c0f7c8e3e0-original-pull-secret podName:bdf61327-2614-4068-974c-a5c0f7c8e3e0 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:37.255313784 +0000 UTC m=+34.230573722 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bdf61327-2614-4068-974c-a5c0f7c8e3e0-original-pull-secret") pod "global-pull-secret-syncer-swxpd" (UID: "bdf61327-2614-4068-974c-a5c0f7c8e3e0") : object "kube-system"/"original-pull-secret" not registered Sep 10 15:42:21.356226 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:21.356198 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q5c4\" (UniqueName: \"kubernetes.io/projected/d88aeb8f-b005-4deb-bc50-5f843a53699c-kube-api-access-2q5c4\") pod \"network-check-target-wqgsp\" (UID: \"d88aeb8f-b005-4deb-bc50-5f843a53699c\") " pod="openshift-network-diagnostics/network-check-target-wqgsp" Sep 10 15:42:21.356406 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:21.356338 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 15:42:21.356406 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:21.356356 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 15:42:21.356406 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:21.356367 2583 projected.go:196] Error preparing data for projected volume kube-api-access-2q5c4 for pod openshift-network-diagnostics/network-check-target-wqgsp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:42:21.356558 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:21.356435 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d88aeb8f-b005-4deb-bc50-5f843a53699c-kube-api-access-2q5c4 podName:d88aeb8f-b005-4deb-bc50-5f843a53699c nodeName:}" failed. No retries permitted until 2026-09-10 15:42:37.356419479 +0000 UTC m=+34.331679426 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2q5c4" (UniqueName: "kubernetes.io/projected/d88aeb8f-b005-4deb-bc50-5f843a53699c-kube-api-access-2q5c4") pod "network-check-target-wqgsp" (UID: "d88aeb8f-b005-4deb-bc50-5f843a53699c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:42:21.509279 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:21.509202 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qztq2" Sep 10 15:42:21.509714 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:21.509207 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqgsp" Sep 10 15:42:21.509714 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:21.509344 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qztq2" podUID="bcf35967-2a15-4bd4-8837-a058d81d1aff" Sep 10 15:42:21.509714 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:21.509429 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqgsp" podUID="d88aeb8f-b005-4deb-bc50-5f843a53699c" Sep 10 15:42:22.508799 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:22.508772 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swxpd" Sep 10 15:42:22.508957 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:22.508877 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swxpd" podUID="bdf61327-2614-4068-974c-a5c0f7c8e3e0" Sep 10 15:42:23.509918 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:23.509891 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qztq2" Sep 10 15:42:23.510298 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:23.510018 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qztq2" podUID="bcf35967-2a15-4bd4-8837-a058d81d1aff" Sep 10 15:42:23.510298 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:23.510060 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqgsp" Sep 10 15:42:23.510298 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:23.510123 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqgsp" podUID="d88aeb8f-b005-4deb-bc50-5f843a53699c" Sep 10 15:42:24.509580 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:24.509130 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swxpd" Sep 10 15:42:24.509749 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:24.509700 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swxpd" podUID="bdf61327-2614-4068-974c-a5c0f7c8e3e0" Sep 10 15:42:24.601519 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:24.601360 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-5lpqq" event={"ID":"7dc7aa02-57e8-4cdc-aab6-9875e1427825","Type":"ContainerStarted","Data":"c37a439f7d0e243322296f93a98e3a7a1f4c0be6b9aea53fd92103abae64bfb6"} Sep 10 15:42:24.603299 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:24.603262 2583 generic.go:356] "Generic (PLEG): container finished" podID="b068e03b-cd04-497c-81bf-1a8e4546dc77" containerID="5f333ded4a49b9cddf54f8b68dc9044d7e3268471d5925dfdf4ccaf285c100a2" exitCode=0 Sep 10 15:42:24.603433 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:24.603313 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qgbvz" event={"ID":"b068e03b-cd04-497c-81bf-1a8e4546dc77","Type":"ContainerDied","Data":"5f333ded4a49b9cddf54f8b68dc9044d7e3268471d5925dfdf4ccaf285c100a2"} Sep 10 15:42:24.607197 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:24.607164 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" event={"ID":"621b9bdc-da4f-4833-a547-2bd2c40e83d1","Type":"ContainerStarted","Data":"f1c80955c3f811881420871d98eb32fcc37b4f48d81561b6ccb9c9772430a2a3"} Sep 10 15:42:24.607298 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:24.607254 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" event={"ID":"621b9bdc-da4f-4833-a547-2bd2c40e83d1","Type":"ContainerStarted","Data":"d574c4068ea58ba32f7dc4198692c17629c137726d8b1eefd335ca5b77a03a17"} Sep 10 15:42:24.607345 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:24.607303 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" event={"ID":"621b9bdc-da4f-4833-a547-2bd2c40e83d1","Type":"ContainerStarted","Data":"7c7b84967020affc8ecf32af084523a88ac5017459deebce15e70e87d2c5dac5"} Sep 10 15:42:24.607345 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:24.607334 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" event={"ID":"621b9bdc-da4f-4833-a547-2bd2c40e83d1","Type":"ContainerStarted","Data":"a7bda9796491bee933a46b861f062cc5b7e4dbc56ae6e632ab1e690d15e58f82"} Sep 10 15:42:24.607438 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:24.607366 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" event={"ID":"621b9bdc-da4f-4833-a547-2bd2c40e83d1","Type":"ContainerStarted","Data":"ff7330d73eda193533ff853c2aaaae152b24607859aaf508b79134fbeb142e94"} Sep 10 15:42:24.607438 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:24.607416 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" event={"ID":"621b9bdc-da4f-4833-a547-2bd2c40e83d1","Type":"ContainerStarted","Data":"06ff39f289ee2cf86a4c0a3f2c7d83faa731f940461c974f086b461af6d0a61a"} Sep 10 15:42:24.608765 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:24.608747 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8bwv" event={"ID":"b631218e-ec48-4b74-8e2b-e6047ceba20b","Type":"ContainerStarted","Data":"ab09b9b3a3a7d3c1f0e290e127ec5ac9dbf619cdb1995827ebf28026ef3779b2"} Sep 10 15:42:24.610521 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:24.610498 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-zdqdf" event={"ID":"c2384bdb-90cf-49a6-885a-42f96018fb04","Type":"ContainerStarted","Data":"605fe0547dabed22fa015c5a7df71b40adaffe10703c614c8831927cd91382b8"} Sep 10 15:42:24.612021 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:24.611998 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mcx2v" event={"ID":"3448a418-7ccc-4a41-b2f0-0e1fa12cd592","Type":"ContainerStarted","Data":"47c3e70d7f1bc246e4f4756d5745956e0740986976e269aa838b27cf4de08838"} Sep 10 15:42:24.613707 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:24.613654 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-5lpqq" podStartSLOduration=4.370563838 podStartE2EDuration="21.613636128s" totalImagesPullingTime="17.24307229s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:24.613257072 +0000 UTC m=+21.588517025" watchObservedRunningTime="2026-09-10 15:42:24.613636128 +0000 UTC m=+21.588896075" Sep 10 15:42:24.613820 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:24.613763 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-107.ec2.internal" podStartSLOduration=20.613756812 podStartE2EDuration="20.613756812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:08.604594047 +0000 UTC m=+5.579854019" watchObservedRunningTime="2026-09-10 15:42:24.613756812 +0000 UTC m=+21.589016767" Sep 10 15:42:24.613879 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:24.613697 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8q52b" event={"ID":"2d40147d-08e8-4362-b32a-8ee4c9727518","Type":"ContainerStarted","Data":"3d6307f04a177d04196786320c7053ab863c5d93bfd79b4faf0aa540efc42e04"} Sep 10 15:42:24.625599 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:24.625558 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mcx2v" podStartSLOduration=7.957224644 podStartE2EDuration="21.625541434s" totalImagesPullingTime="13.66831679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:42:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:24.625182411 +0000 UTC m=+21.600442364" watchObservedRunningTime="2026-09-10 15:42:24.625541434 +0000 UTC m=+21.600801404" Sep 10 15:42:24.675495 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:24.675455 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-zdqdf" podStartSLOduration=4.435307583 podStartE2EDuration="21.675444247s" totalImagesPullingTime="17.240136664s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:24.650731905 +0000 UTC m=+21.625991858" watchObservedRunningTime="2026-09-10 15:42:24.675444247 +0000 UTC m=+21.650704200" Sep 10 15:42:24.687408 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:24.687359 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-8q52b" podStartSLOduration=4.400126654 podStartE2EDuration="21.687349409s" totalImagesPullingTime="17.287222755s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:24.686872455 +0000 UTC m=+21.662132428" watchObservedRunningTime="2026-09-10 15:42:24.687349409 +0000 UTC m=+21.662609362" Sep 10 15:42:25.120184 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:25.120162 2583 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 10 15:42:25.488113 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:25.487984 2583 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T15:42:25.120180316Z","UUID":"2353816f-69ca-419e-9ac6-ef1ff3461ae7","Handler":null,"Name":"","Endpoint":""} Sep 10 15:42:25.489593 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:25.489571 2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 10 15:42:25.489721 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:25.489604 2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 10 15:42:25.509119 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:25.508752 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qztq2" Sep 10 15:42:25.509119 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:25.508882 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qztq2" podUID="bcf35967-2a15-4bd4-8837-a058d81d1aff" Sep 10 15:42:25.509293 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:25.509247 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqgsp" Sep 10 15:42:25.509369 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:25.509348 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqgsp" podUID="d88aeb8f-b005-4deb-bc50-5f843a53699c" Sep 10 15:42:25.617918 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:25.617885 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-m5t8c" event={"ID":"d339b125-a6fe-4bff-b264-281c40b162c7","Type":"ContainerStarted","Data":"d6b0cadaaec64501752607552830c4f2d4aaa683f1ab4d5a6f767a2f77cfc12d"} Sep 10 15:42:25.620795 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:25.620771 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8bwv" event={"ID":"b631218e-ec48-4b74-8e2b-e6047ceba20b","Type":"ContainerStarted","Data":"2a54bd914a0f4dfa046b2fd71611b7652404eb0bbbb3f0fbc589f350e0df7594"} Sep 10 15:42:25.630829 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:25.630789 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-m5t8c" podStartSLOduration=5.38642271 podStartE2EDuration="22.630772808s" totalImagesPullingTime="17.244350098s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:25.630601678 +0000 UTC m=+22.605861631" watchObservedRunningTime="2026-09-10 15:42:25.630772808 +0000 UTC m=+22.606032761" Sep 10 15:42:26.509309 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:26.509283 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swxpd" Sep 10 15:42:26.509451 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:26.509398 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swxpd" podUID="bdf61327-2614-4068-974c-a5c0f7c8e3e0" Sep 10 15:42:26.630220 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:26.630123 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" event={"ID":"621b9bdc-da4f-4833-a547-2bd2c40e83d1","Type":"ContainerStarted","Data":"830a6a5df8a4028dcc6204c4d921aa2979a2f1b2619f5b16f37f20a075711200"} Sep 10 15:42:26.632684 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:26.632647 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8bwv" event={"ID":"b631218e-ec48-4b74-8e2b-e6047ceba20b","Type":"ContainerStarted","Data":"6f440ad82a35b44d0abab3c7d35f80b01890047bf7aa604c427e8f32cf8f0941"} Sep 10 15:42:26.651679 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:26.651639 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8bwv" podStartSLOduration=4.472995694 podStartE2EDuration="23.651625984s" totalImagesPullingTime="19.17863029s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:26.651001719 +0000 UTC m=+23.626261672" watchObservedRunningTime="2026-09-10 15:42:26.651625984 +0000 UTC m=+23.626885944" Sep 10 15:42:27.509376 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:27.509348 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qztq2" Sep 10 15:42:27.509642 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:27.509407 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqgsp" Sep 10 15:42:27.509642 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:27.509502 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qztq2" podUID="bcf35967-2a15-4bd4-8837-a058d81d1aff" Sep 10 15:42:27.509642 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:27.509586 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqgsp" podUID="d88aeb8f-b005-4deb-bc50-5f843a53699c" Sep 10 15:42:28.508664 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:28.508638 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swxpd" Sep 10 15:42:28.509059 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:28.508738 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swxpd" podUID="bdf61327-2614-4068-974c-a5c0f7c8e3e0" Sep 10 15:42:28.556175 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:28.556152 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mcx2v" Sep 10 15:42:28.556637 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:28.556621 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mcx2v" Sep 10 15:42:28.637106 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:28.637085 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mcx2v" Sep 10 15:42:28.637539 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:28.637523 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mcx2v" Sep 10 15:42:29.509525 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:29.509271 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqgsp" Sep 10 15:42:29.510108 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:29.509342 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qztq2" Sep 10 15:42:29.510108 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:29.509626 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqgsp" podUID="d88aeb8f-b005-4deb-bc50-5f843a53699c" Sep 10 15:42:29.510108 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:29.509679 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qztq2" podUID="bcf35967-2a15-4bd4-8837-a058d81d1aff" Sep 10 15:42:29.640191 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:29.640162 2583 generic.go:356] "Generic (PLEG): container finished" podID="b068e03b-cd04-497c-81bf-1a8e4546dc77" containerID="edb8ab104a2a9418fce4b0418f8382ba398a1e81a3e59bdf0e6e333bee436c20" exitCode=0 Sep 10 15:42:29.640340 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:29.640240 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qgbvz" event={"ID":"b068e03b-cd04-497c-81bf-1a8e4546dc77","Type":"ContainerDied","Data":"edb8ab104a2a9418fce4b0418f8382ba398a1e81a3e59bdf0e6e333bee436c20"} Sep 10 15:42:29.647642 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:29.644469 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" event={"ID":"621b9bdc-da4f-4833-a547-2bd2c40e83d1","Type":"ContainerStarted","Data":"b613349c93616a023a6219f9542b5b5af727c4395c720011e3ea691b1a7e5f40"} Sep 10 15:42:29.647642 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:29.645922 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:29.647642 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:29.646060 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:29.662215 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:29.662193 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:29.662306 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:29.662282 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:29.690508 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:29.690474 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" podStartSLOduration=9.38233051 podStartE2EDuration="26.690462051s" totalImagesPullingTime="17.308131541s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:29.688828764 +0000 UTC m=+26.664088737" watchObservedRunningTime="2026-09-10 15:42:29.690462051 +0000 UTC m=+26.665722003" Sep 10 15:42:30.509494 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:30.509466 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swxpd" Sep 10 15:42:30.509624 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:30.509572 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swxpd" podUID="bdf61327-2614-4068-974c-a5c0f7c8e3e0" Sep 10 15:42:30.646711 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:30.646565 2583 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 10 15:42:30.794316 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:30.794249 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-swxpd"] Sep 10 15:42:30.794445 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:30.794346 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swxpd" Sep 10 15:42:30.794494 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:30.794446 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swxpd" podUID="bdf61327-2614-4068-974c-a5c0f7c8e3e0" Sep 10 15:42:30.797915 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:30.797851 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qztq2"] Sep 10 15:42:30.798036 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:30.797971 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qztq2" Sep 10 15:42:30.798120 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:30.798097 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qztq2" podUID="bcf35967-2a15-4bd4-8837-a058d81d1aff" Sep 10 15:42:30.798414 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:30.798392 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wqgsp"] Sep 10 15:42:30.798492 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:30.798480 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqgsp" Sep 10 15:42:30.798588 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:30.798567 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqgsp" podUID="d88aeb8f-b005-4deb-bc50-5f843a53699c" Sep 10 15:42:31.650946 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:31.650917 2583 generic.go:356] "Generic (PLEG): container finished" podID="b068e03b-cd04-497c-81bf-1a8e4546dc77" containerID="b81386135b8577c284d6b15519cf73ef5836dcae3f6acc9a6adf68755d2db9fe" exitCode=0 Sep 10 15:42:31.651282 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:31.651008 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qgbvz" event={"ID":"b068e03b-cd04-497c-81bf-1a8e4546dc77","Type":"ContainerDied","Data":"b81386135b8577c284d6b15519cf73ef5836dcae3f6acc9a6adf68755d2db9fe"} Sep 10 15:42:31.651282 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:31.651051 2583 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Sep 10 15:42:32.508954 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:32.508926 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqgsp" Sep 10 15:42:32.509077 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:32.509010 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swxpd" Sep 10 15:42:32.509077 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:32.509018 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qztq2" Sep 10 15:42:32.509077 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:32.509014 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqgsp" podUID="d88aeb8f-b005-4deb-bc50-5f843a53699c" Sep 10 15:42:32.509237 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:32.509081 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swxpd" podUID="bdf61327-2614-4068-974c-a5c0f7c8e3e0" Sep 10 15:42:32.509237 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:32.509184 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qztq2" podUID="bcf35967-2a15-4bd4-8837-a058d81d1aff" Sep 10 15:42:32.577841 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:32.577772 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:42:32.656618 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:32.656588 2583 generic.go:356] "Generic (PLEG): container finished" podID="b068e03b-cd04-497c-81bf-1a8e4546dc77" containerID="d700ceef868de6d614ba855b0e33aaaaeaf95b80af3d9b4a7ef302b5050d811c" exitCode=0 Sep 10 15:42:32.657091 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:32.656669 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qgbvz" event={"ID":"b068e03b-cd04-497c-81bf-1a8e4546dc77","Type":"ContainerDied","Data":"d700ceef868de6d614ba855b0e33aaaaeaf95b80af3d9b4a7ef302b5050d811c"} Sep 10 15:42:34.509544 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:34.509505 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swxpd" Sep 10 15:42:34.509969 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:34.509505 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qztq2" Sep 10 15:42:34.509969 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:34.509608 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swxpd" podUID="bdf61327-2614-4068-974c-a5c0f7c8e3e0" Sep 10 15:42:34.509969 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:34.509690 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qztq2" podUID="bcf35967-2a15-4bd4-8837-a058d81d1aff" Sep 10 15:42:34.509969 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:34.509505 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqgsp" Sep 10 15:42:34.509969 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:34.509818 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqgsp" podUID="d88aeb8f-b005-4deb-bc50-5f843a53699c" Sep 10 15:42:35.599127 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:35.599087 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:35.713957 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:35.713920 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:35.848065 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:35.847990 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-8v784"] Sep 10 15:42:35.900196 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:35.900160 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:35.901776 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:35.901757 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8v784" Sep 10 15:42:35.904152 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:35.904132 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kqss8\"" Sep 10 15:42:35.904348 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:35.904134 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 10 15:42:35.904348 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:35.904134 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 10 15:42:35.987189 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:35.987147 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:36.070366 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:36.070332 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwk2j\" (UniqueName: \"kubernetes.io/projected/cb7626aa-a9d1-4dec-aab8-b68a3e6adca3-kube-api-access-kwk2j\") pod \"node-resolver-8v784\" (UID: \"cb7626aa-a9d1-4dec-aab8-b68a3e6adca3\") " pod="openshift-dns/node-resolver-8v784" Sep 10 15:42:36.070522 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:36.070374 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb7626aa-a9d1-4dec-aab8-b68a3e6adca3-tmp-dir\") pod \"node-resolver-8v784\" (UID: \"cb7626aa-a9d1-4dec-aab8-b68a3e6adca3\") " pod="openshift-dns/node-resolver-8v784" Sep 10 15:42:36.070522 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:36.070458 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cb7626aa-a9d1-4dec-aab8-b68a3e6adca3-hosts-file\") pod \"node-resolver-8v784\" (UID: \"cb7626aa-a9d1-4dec-aab8-b68a3e6adca3\") " pod="openshift-dns/node-resolver-8v784" Sep 10 15:42:36.171326 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:36.171292 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cb7626aa-a9d1-4dec-aab8-b68a3e6adca3-hosts-file\") pod \"node-resolver-8v784\" (UID: \"cb7626aa-a9d1-4dec-aab8-b68a3e6adca3\") " pod="openshift-dns/node-resolver-8v784" Sep 10 15:42:36.171504 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:36.171408 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwk2j\" (UniqueName: \"kubernetes.io/projected/cb7626aa-a9d1-4dec-aab8-b68a3e6adca3-kube-api-access-kwk2j\") pod \"node-resolver-8v784\" (UID: \"cb7626aa-a9d1-4dec-aab8-b68a3e6adca3\") " pod="openshift-dns/node-resolver-8v784" Sep 10 15:42:36.171504 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:36.171428 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cb7626aa-a9d1-4dec-aab8-b68a3e6adca3-hosts-file\") pod \"node-resolver-8v784\" (UID: \"cb7626aa-a9d1-4dec-aab8-b68a3e6adca3\") " pod="openshift-dns/node-resolver-8v784" Sep 10 15:42:36.171504 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:36.171451 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb7626aa-a9d1-4dec-aab8-b68a3e6adca3-tmp-dir\") pod \"node-resolver-8v784\" (UID: \"cb7626aa-a9d1-4dec-aab8-b68a3e6adca3\") " pod="openshift-dns/node-resolver-8v784" Sep 10 15:42:36.171770 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:36.171754 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb7626aa-a9d1-4dec-aab8-b68a3e6adca3-tmp-dir\") pod \"node-resolver-8v784\" (UID: \"cb7626aa-a9d1-4dec-aab8-b68a3e6adca3\") " pod="openshift-dns/node-resolver-8v784" Sep 10 15:42:36.181763 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:36.181598 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwk2j\" (UniqueName: \"kubernetes.io/projected/cb7626aa-a9d1-4dec-aab8-b68a3e6adca3-kube-api-access-kwk2j\") pod \"node-resolver-8v784\" (UID: \"cb7626aa-a9d1-4dec-aab8-b68a3e6adca3\") " pod="openshift-dns/node-resolver-8v784" Sep 10 15:42:36.211219 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:36.211194 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8v784" Sep 10 15:42:36.508779 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:36.508705 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqgsp" Sep 10 15:42:36.508779 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:36.508747 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swxpd" Sep 10 15:42:36.508779 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:36.508747 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qztq2" Sep 10 15:42:36.509013 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:36.508807 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wqgsp" podUID="d88aeb8f-b005-4deb-bc50-5f843a53699c" Sep 10 15:42:36.509013 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:36.508925 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qztq2" podUID="bcf35967-2a15-4bd4-8837-a058d81d1aff" Sep 10 15:42:36.509013 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:36.508987 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-swxpd" podUID="bdf61327-2614-4068-974c-a5c0f7c8e3e0" Sep 10 15:42:36.786215 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:36.786149 2583 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-107.ec2.internal" event="NodeReady" Sep 10 15:42:36.786609 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:36.786310 2583 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 10 15:42:36.831300 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:36.831239 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-45ph9"] Sep 10 15:42:36.851102 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:36.851074 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wqwzs"] Sep 10 15:42:36.851220 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:36.851200 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-45ph9" Sep 10 15:42:36.853461 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:36.853435 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 10 15:42:36.853564 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:36.853479 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-z4knc\"" Sep 10 15:42:36.853564 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:36.853442 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 10 15:42:36.863415 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:36.863396 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wqwzs"] Sep 10 15:42:36.863495 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:36.863432 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-45ph9"] Sep 10 15:42:36.863551 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:36.863496 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wqwzs" Sep 10 15:42:36.865734 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:36.865696 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 10 15:42:36.865734 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:36.865704 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 10 15:42:36.865872 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:36.865786 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-8f2wr\"" Sep 10 15:42:36.865872 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:36.865828 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 10 15:42:36.977906 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:36.977873 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8681b681-0371-4b9d-8c1d-82a431bd3e62-metrics-tls\") pod \"dns-default-45ph9\" (UID: \"8681b681-0371-4b9d-8c1d-82a431bd3e62\") " pod="openshift-dns/dns-default-45ph9" Sep 10 15:42:36.978064 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:36.977933 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8681b681-0371-4b9d-8c1d-82a431bd3e62-tmp-dir\") pod \"dns-default-45ph9\" (UID: \"8681b681-0371-4b9d-8c1d-82a431bd3e62\") " pod="openshift-dns/dns-default-45ph9" Sep 10 15:42:36.978064 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:36.978009 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66ac0bb7-d8cd-43d7-b247-4d5e441f321a-cert\") pod \"ingress-canary-wqwzs\" (UID: \"66ac0bb7-d8cd-43d7-b247-4d5e441f321a\") " pod="openshift-ingress-canary/ingress-canary-wqwzs" Sep 10 15:42:36.978064 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:36.978041 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6k85\" (UniqueName: \"kubernetes.io/projected/66ac0bb7-d8cd-43d7-b247-4d5e441f321a-kube-api-access-q6k85\") pod \"ingress-canary-wqwzs\" (UID: \"66ac0bb7-d8cd-43d7-b247-4d5e441f321a\") " pod="openshift-ingress-canary/ingress-canary-wqwzs" Sep 10 15:42:36.978186 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:36.978108 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8681b681-0371-4b9d-8c1d-82a431bd3e62-config-volume\") pod \"dns-default-45ph9\" (UID: \"8681b681-0371-4b9d-8c1d-82a431bd3e62\") " pod="openshift-dns/dns-default-45ph9" Sep 10 15:42:36.978186 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:36.978124 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5n9v\" (UniqueName: \"kubernetes.io/projected/8681b681-0371-4b9d-8c1d-82a431bd3e62-kube-api-access-h5n9v\") pod \"dns-default-45ph9\" (UID: \"8681b681-0371-4b9d-8c1d-82a431bd3e62\") " pod="openshift-dns/dns-default-45ph9" Sep 10 15:42:37.079335 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:37.079259 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8681b681-0371-4b9d-8c1d-82a431bd3e62-metrics-tls\") pod \"dns-default-45ph9\" (UID: \"8681b681-0371-4b9d-8c1d-82a431bd3e62\") " pod="openshift-dns/dns-default-45ph9" Sep 10 15:42:37.079517 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:37.079403 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8681b681-0371-4b9d-8c1d-82a431bd3e62-tmp-dir\") pod \"dns-default-45ph9\" (UID: \"8681b681-0371-4b9d-8c1d-82a431bd3e62\") " pod="openshift-dns/dns-default-45ph9" Sep 10 15:42:37.079517 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:37.079427 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 15:42:37.079517 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:37.079454 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66ac0bb7-d8cd-43d7-b247-4d5e441f321a-cert\") pod \"ingress-canary-wqwzs\" (UID: \"66ac0bb7-d8cd-43d7-b247-4d5e441f321a\") " pod="openshift-ingress-canary/ingress-canary-wqwzs" Sep 10 15:42:37.079517 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:37.079493 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8681b681-0371-4b9d-8c1d-82a431bd3e62-metrics-tls podName:8681b681-0371-4b9d-8c1d-82a431bd3e62 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:37.579474307 +0000 UTC m=+34.554734239 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8681b681-0371-4b9d-8c1d-82a431bd3e62-metrics-tls") pod "dns-default-45ph9" (UID: "8681b681-0371-4b9d-8c1d-82a431bd3e62") : secret "dns-default-metrics-tls" not found Sep 10 15:42:37.079727 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:37.079530 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6k85\" (UniqueName: \"kubernetes.io/projected/66ac0bb7-d8cd-43d7-b247-4d5e441f321a-kube-api-access-q6k85\") pod \"ingress-canary-wqwzs\" (UID: \"66ac0bb7-d8cd-43d7-b247-4d5e441f321a\") " pod="openshift-ingress-canary/ingress-canary-wqwzs" Sep 10 15:42:37.079727 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:37.079586 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8681b681-0371-4b9d-8c1d-82a431bd3e62-config-volume\") pod \"dns-default-45ph9\" (UID: \"8681b681-0371-4b9d-8c1d-82a431bd3e62\") " pod="openshift-dns/dns-default-45ph9" Sep 10 15:42:37.079727 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:37.079610 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5n9v\" (UniqueName: \"kubernetes.io/projected/8681b681-0371-4b9d-8c1d-82a431bd3e62-kube-api-access-h5n9v\") pod \"dns-default-45ph9\" (UID: \"8681b681-0371-4b9d-8c1d-82a431bd3e62\") " pod="openshift-dns/dns-default-45ph9" Sep 10 15:42:37.079848 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:37.079776 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8681b681-0371-4b9d-8c1d-82a431bd3e62-tmp-dir\") pod \"dns-default-45ph9\" (UID: \"8681b681-0371-4b9d-8c1d-82a431bd3e62\") " pod="openshift-dns/dns-default-45ph9" Sep 10 15:42:37.080173 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:37.080133 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8681b681-0371-4b9d-8c1d-82a431bd3e62-config-volume\") pod \"dns-default-45ph9\" (UID: \"8681b681-0371-4b9d-8c1d-82a431bd3e62\") " pod="openshift-dns/dns-default-45ph9" Sep 10 15:42:37.082588 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:37.082560 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/66ac0bb7-d8cd-43d7-b247-4d5e441f321a-cert\") pod \"ingress-canary-wqwzs\" (UID: \"66ac0bb7-d8cd-43d7-b247-4d5e441f321a\") " pod="openshift-ingress-canary/ingress-canary-wqwzs" Sep 10 15:42:37.090848 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:37.090813 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5n9v\" (UniqueName: \"kubernetes.io/projected/8681b681-0371-4b9d-8c1d-82a431bd3e62-kube-api-access-h5n9v\") pod \"dns-default-45ph9\" (UID: \"8681b681-0371-4b9d-8c1d-82a431bd3e62\") " pod="openshift-dns/dns-default-45ph9" Sep 10 15:42:37.091114 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:37.091084 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6k85\" (UniqueName: \"kubernetes.io/projected/66ac0bb7-d8cd-43d7-b247-4d5e441f321a-kube-api-access-q6k85\") pod \"ingress-canary-wqwzs\" (UID: \"66ac0bb7-d8cd-43d7-b247-4d5e441f321a\") " pod="openshift-ingress-canary/ingress-canary-wqwzs" Sep 10 15:42:37.173108 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:37.173076 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wqwzs" Sep 10 15:42:37.281274 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:37.281242 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcf35967-2a15-4bd4-8837-a058d81d1aff-metrics-certs\") pod \"network-metrics-daemon-qztq2\" (UID: \"bcf35967-2a15-4bd4-8837-a058d81d1aff\") " pod="openshift-multus/network-metrics-daemon-qztq2" Sep 10 15:42:37.281432 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:37.281312 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdf61327-2614-4068-974c-a5c0f7c8e3e0-original-pull-secret\") pod \"global-pull-secret-syncer-swxpd\" (UID: \"bdf61327-2614-4068-974c-a5c0f7c8e3e0\") " pod="kube-system/global-pull-secret-syncer-swxpd" Sep 10 15:42:37.281432 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:37.281345 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:42:37.281432 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:37.281426 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 10 15:42:37.281596 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:37.281455 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcf35967-2a15-4bd4-8837-a058d81d1aff-metrics-certs podName:bcf35967-2a15-4bd4-8837-a058d81d1aff nodeName:}" failed. No retries permitted until 2026-09-10 15:43:09.281434684 +0000 UTC m=+66.256694642 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bcf35967-2a15-4bd4-8837-a058d81d1aff-metrics-certs") pod "network-metrics-daemon-qztq2" (UID: "bcf35967-2a15-4bd4-8837-a058d81d1aff") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 10 15:42:37.281596 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:37.281487 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdf61327-2614-4068-974c-a5c0f7c8e3e0-original-pull-secret podName:bdf61327-2614-4068-974c-a5c0f7c8e3e0 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:09.28146748 +0000 UTC m=+66.256727429 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/bdf61327-2614-4068-974c-a5c0f7c8e3e0-original-pull-secret") pod "global-pull-secret-syncer-swxpd" (UID: "bdf61327-2614-4068-974c-a5c0f7c8e3e0") : object "kube-system"/"original-pull-secret" not registered Sep 10 15:42:37.382680 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:37.382642 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q5c4\" (UniqueName: \"kubernetes.io/projected/d88aeb8f-b005-4deb-bc50-5f843a53699c-kube-api-access-2q5c4\") pod \"network-check-target-wqgsp\" (UID: \"d88aeb8f-b005-4deb-bc50-5f843a53699c\") " pod="openshift-network-diagnostics/network-check-target-wqgsp" Sep 10 15:42:37.382836 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:37.382800 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 10 15:42:37.382836 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:37.382826 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 10 15:42:37.382943 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:37.382839 2583 projected.go:196] Error preparing data for projected volume kube-api-access-2q5c4 for pod openshift-network-diagnostics/network-check-target-wqgsp: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:42:37.382943 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:37.382901 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d88aeb8f-b005-4deb-bc50-5f843a53699c-kube-api-access-2q5c4 podName:d88aeb8f-b005-4deb-bc50-5f843a53699c nodeName:}" failed. No retries permitted until 2026-09-10 15:43:09.38288159 +0000 UTC m=+66.358141541 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2q5c4" (UniqueName: "kubernetes.io/projected/d88aeb8f-b005-4deb-bc50-5f843a53699c-kube-api-access-2q5c4") pod "network-check-target-wqgsp" (UID: "d88aeb8f-b005-4deb-bc50-5f843a53699c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 10 15:42:37.488216 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:37.488187 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb7626aa_a9d1_4dec_aab8_b68a3e6adca3.slice/crio-3c0221dbbdfd1abdf3624cddd6f2b8d9cda146510cc68534fc1fde844ad12c7c WatchSource:0}: Error finding container 3c0221dbbdfd1abdf3624cddd6f2b8d9cda146510cc68534fc1fde844ad12c7c: Status 404 returned error can't find the container with id 3c0221dbbdfd1abdf3624cddd6f2b8d9cda146510cc68534fc1fde844ad12c7c Sep 10 15:42:37.585025 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:37.584946 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8681b681-0371-4b9d-8c1d-82a431bd3e62-metrics-tls\") pod \"dns-default-45ph9\" (UID: \"8681b681-0371-4b9d-8c1d-82a431bd3e62\") " pod="openshift-dns/dns-default-45ph9" Sep 10 15:42:37.585165 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:37.585145 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 15:42:37.585228 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:37.585211 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8681b681-0371-4b9d-8c1d-82a431bd3e62-metrics-tls podName:8681b681-0371-4b9d-8c1d-82a431bd3e62 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:38.585193257 +0000 UTC m=+35.560453191 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8681b681-0371-4b9d-8c1d-82a431bd3e62-metrics-tls") pod "dns-default-45ph9" (UID: "8681b681-0371-4b9d-8c1d-82a431bd3e62") : secret "dns-default-metrics-tls" not found Sep 10 15:42:37.667948 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:37.667890 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8v784" event={"ID":"cb7626aa-a9d1-4dec-aab8-b68a3e6adca3","Type":"ContainerStarted","Data":"f1ef4908752e9c85dec6687075c934baaac415dff0b45300f4b24d9836888eee"} Sep 10 15:42:37.668019 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:37.667952 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8v784" event={"ID":"cb7626aa-a9d1-4dec-aab8-b68a3e6adca3","Type":"ContainerStarted","Data":"3c0221dbbdfd1abdf3624cddd6f2b8d9cda146510cc68534fc1fde844ad12c7c"} Sep 10 15:42:37.683841 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:37.683801 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-8v784" podStartSLOduration=2.683787059 podStartE2EDuration="2.683787059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:37.683306695 +0000 UTC m=+34.658566648" watchObservedRunningTime="2026-09-10 15:42:37.683787059 +0000 UTC m=+34.659047006" Sep 10 15:42:37.770563 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:37.770349 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wqwzs"] Sep 10 15:42:37.774292 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:42:37.774262 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66ac0bb7_d8cd_43d7_b247_4d5e441f321a.slice/crio-b5e6587ed107f24578dc6191cabbc24730316d7b2e2202768d635f5a3b2517f9 WatchSource:0}: Error finding container b5e6587ed107f24578dc6191cabbc24730316d7b2e2202768d635f5a3b2517f9: Status 404 returned error can't find the container with id b5e6587ed107f24578dc6191cabbc24730316d7b2e2202768d635f5a3b2517f9 Sep 10 15:42:38.509090 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:38.509061 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swxpd" Sep 10 15:42:38.509090 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:38.509086 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qztq2" Sep 10 15:42:38.509927 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:38.509086 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqgsp" Sep 10 15:42:38.511395 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:38.511364 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 15:42:38.511851 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:38.511803 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 15:42:38.512406 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:38.512372 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 15:42:38.512406 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:38.512395 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-b79h8\"" Sep 10 15:42:38.512658 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:38.512636 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cthzr\"" Sep 10 15:42:38.512658 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:38.512645 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 15:42:38.590577 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:38.590546 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8681b681-0371-4b9d-8c1d-82a431bd3e62-metrics-tls\") pod \"dns-default-45ph9\" (UID: \"8681b681-0371-4b9d-8c1d-82a431bd3e62\") " pod="openshift-dns/dns-default-45ph9" Sep 10 15:42:38.590716 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:38.590694 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 15:42:38.590816 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:38.590772 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8681b681-0371-4b9d-8c1d-82a431bd3e62-metrics-tls podName:8681b681-0371-4b9d-8c1d-82a431bd3e62 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:40.590751755 +0000 UTC m=+37.566011691 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8681b681-0371-4b9d-8c1d-82a431bd3e62-metrics-tls") pod "dns-default-45ph9" (UID: "8681b681-0371-4b9d-8c1d-82a431bd3e62") : secret "dns-default-metrics-tls" not found Sep 10 15:42:38.674251 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:38.674216 2583 generic.go:356] "Generic (PLEG): container finished" podID="b068e03b-cd04-497c-81bf-1a8e4546dc77" containerID="83d1454597bf95d70ed1b4849a57ce645c0be289073c4140a2135d4bdc0afda3" exitCode=0 Sep 10 15:42:38.674358 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:38.674325 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qgbvz" event={"ID":"b068e03b-cd04-497c-81bf-1a8e4546dc77","Type":"ContainerDied","Data":"83d1454597bf95d70ed1b4849a57ce645c0be289073c4140a2135d4bdc0afda3"} Sep 10 15:42:38.675591 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:38.675566 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wqwzs" event={"ID":"66ac0bb7-d8cd-43d7-b247-4d5e441f321a","Type":"ContainerStarted","Data":"b5e6587ed107f24578dc6191cabbc24730316d7b2e2202768d635f5a3b2517f9"} Sep 10 15:42:39.682486 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:39.682453 2583 generic.go:356] "Generic (PLEG): container finished" podID="b068e03b-cd04-497c-81bf-1a8e4546dc77" containerID="a38aa13a8defd56c60c5d43a80f44ec76d07e75895ced3c293fcb8c3b05fcb8c" exitCode=0 Sep 10 15:42:39.682974 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:39.682499 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qgbvz" event={"ID":"b068e03b-cd04-497c-81bf-1a8e4546dc77","Type":"ContainerDied","Data":"a38aa13a8defd56c60c5d43a80f44ec76d07e75895ced3c293fcb8c3b05fcb8c"} Sep 10 15:42:39.972195 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:39.972170 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:40.072499 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:40.072474 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:40.606619 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:40.606542 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8681b681-0371-4b9d-8c1d-82a431bd3e62-metrics-tls\") pod \"dns-default-45ph9\" (UID: \"8681b681-0371-4b9d-8c1d-82a431bd3e62\") " pod="openshift-dns/dns-default-45ph9" Sep 10 15:42:40.606834 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:40.606660 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 15:42:40.606834 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:40.606714 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8681b681-0371-4b9d-8c1d-82a431bd3e62-metrics-tls podName:8681b681-0371-4b9d-8c1d-82a431bd3e62 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:44.606700751 +0000 UTC m=+41.581960682 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8681b681-0371-4b9d-8c1d-82a431bd3e62-metrics-tls") pod "dns-default-45ph9" (UID: "8681b681-0371-4b9d-8c1d-82a431bd3e62") : secret "dns-default-metrics-tls" not found Sep 10 15:42:40.686858 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:40.686824 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wqwzs" event={"ID":"66ac0bb7-d8cd-43d7-b247-4d5e441f321a","Type":"ContainerStarted","Data":"ebf0e12bbbbfba83ed5877fb78cb368a19a173196758110a29b63e2edff88a9c"} Sep 10 15:42:40.687228 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:40.686922 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:40.690296 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:40.690276 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qgbvz" event={"ID":"b068e03b-cd04-497c-81bf-1a8e4546dc77","Type":"ContainerStarted","Data":"f26942ce3fa1ee53e288823dfcf043c9904cb1c995a649bb329c573f083de4db"} Sep 10 15:42:40.704557 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:40.704506 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wqwzs" podStartSLOduration=2.386253212 podStartE2EDuration="4.704490467s" totalImagesPullingTime="2.318237255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:42:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:40.704184563 +0000 UTC m=+37.679444517" watchObservedRunningTime="2026-09-10 15:42:40.704490467 +0000 UTC m=+37.679750421" Sep 10 15:42:40.726043 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:40.726001 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qgbvz" podStartSLOduration=8.861508777 podStartE2EDuration="37.725990927s" totalImagesPullingTime="28.86448215s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:03 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:40.724941534 +0000 UTC m=+37.700201487" watchObservedRunningTime="2026-09-10 15:42:40.725990927 +0000 UTC m=+37.701250880" Sep 10 15:42:40.787828 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:40.787803 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:40.887956 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:40.887930 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:40.950928 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:40.950892 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:41.606762 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:41.606732 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:41.707986 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:41.707957 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:44.630733 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:44.630700 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8681b681-0371-4b9d-8c1d-82a431bd3e62-metrics-tls\") pod \"dns-default-45ph9\" (UID: \"8681b681-0371-4b9d-8c1d-82a431bd3e62\") " pod="openshift-dns/dns-default-45ph9" Sep 10 15:42:44.631218 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:44.630832 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 15:42:44.631218 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:44.630904 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8681b681-0371-4b9d-8c1d-82a431bd3e62-metrics-tls podName:8681b681-0371-4b9d-8c1d-82a431bd3e62 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:52.630884624 +0000 UTC m=+49.606144571 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8681b681-0371-4b9d-8c1d-82a431bd3e62-metrics-tls") pod "dns-default-45ph9" (UID: "8681b681-0371-4b9d-8c1d-82a431bd3e62") : secret "dns-default-metrics-tls" not found Sep 10 15:42:47.668226 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:47.668194 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:47.727103 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:47.727069 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:49.474417 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:49.474368 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:49.536046 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:49.536015 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:42:52.682340 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:42:52.682304 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8681b681-0371-4b9d-8c1d-82a431bd3e62-metrics-tls\") pod \"dns-default-45ph9\" (UID: \"8681b681-0371-4b9d-8c1d-82a431bd3e62\") " pod="openshift-dns/dns-default-45ph9" Sep 10 15:42:52.682736 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:52.682435 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 15:42:52.682736 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:42:52.682494 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8681b681-0371-4b9d-8c1d-82a431bd3e62-metrics-tls podName:8681b681-0371-4b9d-8c1d-82a431bd3e62 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:08.682479042 +0000 UTC m=+65.657738973 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8681b681-0371-4b9d-8c1d-82a431bd3e62-metrics-tls") pod "dns-default-45ph9" (UID: "8681b681-0371-4b9d-8c1d-82a431bd3e62") : secret "dns-default-metrics-tls" not found Sep 10 15:43:00.962078 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:00.962034 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:01.033436 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:01.033409 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:02.420333 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:02.420295 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:02.521020 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:02.520991 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:02.667522 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:02.667499 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t48fw" Sep 10 15:43:02.800478 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:02.800375 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:02.902231 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:02.902190 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:08.684842 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:08.684804 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8681b681-0371-4b9d-8c1d-82a431bd3e62-metrics-tls\") pod \"dns-default-45ph9\" (UID: \"8681b681-0371-4b9d-8c1d-82a431bd3e62\") " pod="openshift-dns/dns-default-45ph9" Sep 10 15:43:08.685211 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:08.684918 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 10 15:43:08.685211 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:08.684978 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8681b681-0371-4b9d-8c1d-82a431bd3e62-metrics-tls podName:8681b681-0371-4b9d-8c1d-82a431bd3e62 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:40.684963702 +0000 UTC m=+97.660223637 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8681b681-0371-4b9d-8c1d-82a431bd3e62-metrics-tls") pod "dns-default-45ph9" (UID: "8681b681-0371-4b9d-8c1d-82a431bd3e62") : secret "dns-default-metrics-tls" not found Sep 10 15:43:08.789544 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:08.789515 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-xrvfj"] Sep 10 15:43:08.792323 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:08.792305 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-xrvfj" Sep 10 15:43:08.794260 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:08.794241 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 10 15:43:08.794725 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:08.794699 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 10 15:43:08.794845 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:08.794705 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 10 15:43:08.794990 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:08.794975 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-5lk57\"" Sep 10 15:43:08.794990 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:08.794987 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 10 15:43:08.810800 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:08.809144 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-xrvfj"] Sep 10 15:43:08.886324 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:08.886296 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6d7bbb7b-e101-4a09-9211-b2ee0dadd7c6-config\") pod \"service-ca-operator-859f94c4b-xrvfj\" (UID: \"6d7bbb7b-e101-4a09-9211-b2ee0dadd7c6\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-xrvfj" Sep 10 15:43:08.886427 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:08.886340 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2h5p\" (UniqueName: \"kubernetes.io/projected/6d7bbb7b-e101-4a09-9211-b2ee0dadd7c6-kube-api-access-n2h5p\") pod \"service-ca-operator-859f94c4b-xrvfj\" (UID: \"6d7bbb7b-e101-4a09-9211-b2ee0dadd7c6\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-xrvfj" Sep 10 15:43:08.886427 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:08.886395 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6d7bbb7b-e101-4a09-9211-b2ee0dadd7c6-serving-cert\") pod \"service-ca-operator-859f94c4b-xrvfj\" (UID: \"6d7bbb7b-e101-4a09-9211-b2ee0dadd7c6\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-xrvfj" Sep 10 15:43:08.892127 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:08.892101 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-b4f467c4b-xjkv7"] Sep 10 15:43:08.894094 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:08.894081 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-84ff49d5c7-bh94t"] Sep 10 15:43:08.894214 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:08.894198 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-b4f467c4b-xjkv7" Sep 10 15:43:08.896040 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:08.896019 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84ff49d5c7-bh94t" Sep 10 15:43:08.896522 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:08.896507 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-lxsvl\"" Sep 10 15:43:08.896522 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:08.896516 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 10 15:43:08.896727 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:08.896715 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Sep 10 15:43:08.896770 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:08.896714 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Sep 10 15:43:08.896869 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:08.896853 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 10 15:43:08.897268 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:08.897249 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Sep 10 15:43:08.897345 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:08.897335 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Sep 10 15:43:08.897699 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:08.897686 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 10 15:43:08.898033 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:08.898019 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 10 15:43:08.898156 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:08.898141 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 10 15:43:08.898352 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:08.898337 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-whst8\"" Sep 10 15:43:08.904993 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:08.904975 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 10 15:43:08.919614 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:08.919587 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-b4f467c4b-xjkv7"] Sep 10 15:43:08.920195 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:08.920178 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84ff49d5c7-bh94t"] Sep 10 15:43:08.987617 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:08.987548 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6d7bbb7b-e101-4a09-9211-b2ee0dadd7c6-config\") pod \"service-ca-operator-859f94c4b-xrvfj\" (UID: \"6d7bbb7b-e101-4a09-9211-b2ee0dadd7c6\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-xrvfj" Sep 10 15:43:08.987617 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:08.987593 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b69645e0-943e-4301-982a-9d1532d25e56-registry-certificates\") pod \"image-registry-84ff49d5c7-bh94t\" (UID: \"b69645e0-943e-4301-982a-9d1532d25e56\") " pod="openshift-image-registry/image-registry-84ff49d5c7-bh94t" Sep 10 15:43:08.987617 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:08.987615 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e6af5593-a983-4c35-95e9-3c166bdbf385-stats-auth\") pod \"router-default-b4f467c4b-xjkv7\" (UID: \"e6af5593-a983-4c35-95e9-3c166bdbf385\") " pod="openshift-ingress/router-default-b4f467c4b-xjkv7" Sep 10 15:43:08.987825 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:08.987672 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2h5p\" (UniqueName: \"kubernetes.io/projected/6d7bbb7b-e101-4a09-9211-b2ee0dadd7c6-kube-api-access-n2h5p\") pod \"service-ca-operator-859f94c4b-xrvfj\" (UID: \"6d7bbb7b-e101-4a09-9211-b2ee0dadd7c6\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-xrvfj" Sep 10 15:43:08.987825 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:08.987691 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rdw8\" (UniqueName: \"kubernetes.io/projected/e6af5593-a983-4c35-95e9-3c166bdbf385-kube-api-access-9rdw8\") pod \"router-default-b4f467c4b-xjkv7\" (UID: \"e6af5593-a983-4c35-95e9-3c166bdbf385\") " pod="openshift-ingress/router-default-b4f467c4b-xjkv7" Sep 10 15:43:08.987825 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:08.987733 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e6af5593-a983-4c35-95e9-3c166bdbf385-default-certificate\") pod \"router-default-b4f467c4b-xjkv7\" (UID: \"e6af5593-a983-4c35-95e9-3c166bdbf385\") " pod="openshift-ingress/router-default-b4f467c4b-xjkv7" Sep 10 15:43:08.987825 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:08.987749 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6af5593-a983-4c35-95e9-3c166bdbf385-metrics-certs\") pod \"router-default-b4f467c4b-xjkv7\" (UID: \"e6af5593-a983-4c35-95e9-3c166bdbf385\") " pod="openshift-ingress/router-default-b4f467c4b-xjkv7" Sep 10 15:43:08.987825 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:08.987784 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b69645e0-943e-4301-982a-9d1532d25e56-image-registry-private-configuration\") pod \"image-registry-84ff49d5c7-bh94t\" (UID: \"b69645e0-943e-4301-982a-9d1532d25e56\") " pod="openshift-image-registry/image-registry-84ff49d5c7-bh94t" Sep 10 15:43:08.987825 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:08.987811 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6d7bbb7b-e101-4a09-9211-b2ee0dadd7c6-serving-cert\") pod \"service-ca-operator-859f94c4b-xrvfj\" (UID: \"6d7bbb7b-e101-4a09-9211-b2ee0dadd7c6\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-xrvfj" Sep 10 15:43:08.988066 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:08.987849 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b69645e0-943e-4301-982a-9d1532d25e56-registry-tls\") pod \"image-registry-84ff49d5c7-bh94t\" (UID: \"b69645e0-943e-4301-982a-9d1532d25e56\") " pod="openshift-image-registry/image-registry-84ff49d5c7-bh94t" Sep 10 15:43:08.988066 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:08.987905 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b69645e0-943e-4301-982a-9d1532d25e56-trusted-ca\") pod \"image-registry-84ff49d5c7-bh94t\" (UID: \"b69645e0-943e-4301-982a-9d1532d25e56\") " pod="openshift-image-registry/image-registry-84ff49d5c7-bh94t" Sep 10 15:43:08.988066 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:08.987933 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b69645e0-943e-4301-982a-9d1532d25e56-installation-pull-secrets\") pod \"image-registry-84ff49d5c7-bh94t\" (UID: \"b69645e0-943e-4301-982a-9d1532d25e56\") " pod="openshift-image-registry/image-registry-84ff49d5c7-bh94t" Sep 10 15:43:08.988066 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:08.987953 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qw9q\" (UniqueName: \"kubernetes.io/projected/b69645e0-943e-4301-982a-9d1532d25e56-kube-api-access-5qw9q\") pod \"image-registry-84ff49d5c7-bh94t\" (UID: \"b69645e0-943e-4301-982a-9d1532d25e56\") " pod="openshift-image-registry/image-registry-84ff49d5c7-bh94t" Sep 10 15:43:08.988066 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:08.988014 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b69645e0-943e-4301-982a-9d1532d25e56-bound-sa-token\") pod \"image-registry-84ff49d5c7-bh94t\" (UID: \"b69645e0-943e-4301-982a-9d1532d25e56\") " pod="openshift-image-registry/image-registry-84ff49d5c7-bh94t" Sep 10 15:43:08.988066 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:08.988042 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6d7bbb7b-e101-4a09-9211-b2ee0dadd7c6-config\") pod \"service-ca-operator-859f94c4b-xrvfj\" (UID: \"6d7bbb7b-e101-4a09-9211-b2ee0dadd7c6\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-xrvfj" Sep 10 15:43:08.988066 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:08.988055 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6af5593-a983-4c35-95e9-3c166bdbf385-service-ca-bundle\") pod \"router-default-b4f467c4b-xjkv7\" (UID: \"e6af5593-a983-4c35-95e9-3c166bdbf385\") " pod="openshift-ingress/router-default-b4f467c4b-xjkv7" Sep 10 15:43:08.988272 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:08.988082 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b69645e0-943e-4301-982a-9d1532d25e56-ca-trust-extracted\") pod \"image-registry-84ff49d5c7-bh94t\" (UID: \"b69645e0-943e-4301-982a-9d1532d25e56\") " pod="openshift-image-registry/image-registry-84ff49d5c7-bh94t" Sep 10 15:43:08.991062 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:08.991035 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6d7bbb7b-e101-4a09-9211-b2ee0dadd7c6-serving-cert\") pod \"service-ca-operator-859f94c4b-xrvfj\" (UID: \"6d7bbb7b-e101-4a09-9211-b2ee0dadd7c6\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-xrvfj" Sep 10 15:43:08.997715 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:08.997694 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2h5p\" (UniqueName: \"kubernetes.io/projected/6d7bbb7b-e101-4a09-9211-b2ee0dadd7c6-kube-api-access-n2h5p\") pod \"service-ca-operator-859f94c4b-xrvfj\" (UID: \"6d7bbb7b-e101-4a09-9211-b2ee0dadd7c6\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-xrvfj" Sep 10 15:43:09.057594 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:09.057568 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:09.088741 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:09.088723 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e6af5593-a983-4c35-95e9-3c166bdbf385-default-certificate\") pod \"router-default-b4f467c4b-xjkv7\" (UID: \"e6af5593-a983-4c35-95e9-3c166bdbf385\") " pod="openshift-ingress/router-default-b4f467c4b-xjkv7" Sep 10 15:43:09.088807 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:09.088744 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6af5593-a983-4c35-95e9-3c166bdbf385-metrics-certs\") pod \"router-default-b4f467c4b-xjkv7\" (UID: \"e6af5593-a983-4c35-95e9-3c166bdbf385\") " pod="openshift-ingress/router-default-b4f467c4b-xjkv7" Sep 10 15:43:09.088807 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:09.088776 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b69645e0-943e-4301-982a-9d1532d25e56-image-registry-private-configuration\") pod \"image-registry-84ff49d5c7-bh94t\" (UID: \"b69645e0-943e-4301-982a-9d1532d25e56\") " pod="openshift-image-registry/image-registry-84ff49d5c7-bh94t" Sep 10 15:43:09.088807 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:09.088792 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b69645e0-943e-4301-982a-9d1532d25e56-registry-tls\") pod \"image-registry-84ff49d5c7-bh94t\" (UID: \"b69645e0-943e-4301-982a-9d1532d25e56\") " pod="openshift-image-registry/image-registry-84ff49d5c7-bh94t" Sep 10 15:43:09.088960 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:09.088816 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b69645e0-943e-4301-982a-9d1532d25e56-trusted-ca\") pod \"image-registry-84ff49d5c7-bh94t\" (UID: \"b69645e0-943e-4301-982a-9d1532d25e56\") " pod="openshift-image-registry/image-registry-84ff49d5c7-bh94t" Sep 10 15:43:09.088960 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:09.088840 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b69645e0-943e-4301-982a-9d1532d25e56-installation-pull-secrets\") pod \"image-registry-84ff49d5c7-bh94t\" (UID: \"b69645e0-943e-4301-982a-9d1532d25e56\") " pod="openshift-image-registry/image-registry-84ff49d5c7-bh94t" Sep 10 15:43:09.088960 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:09.088864 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qw9q\" (UniqueName: \"kubernetes.io/projected/b69645e0-943e-4301-982a-9d1532d25e56-kube-api-access-5qw9q\") pod \"image-registry-84ff49d5c7-bh94t\" (UID: \"b69645e0-943e-4301-982a-9d1532d25e56\") " pod="openshift-image-registry/image-registry-84ff49d5c7-bh94t" Sep 10 15:43:09.088960 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:09.088890 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b69645e0-943e-4301-982a-9d1532d25e56-bound-sa-token\") pod \"image-registry-84ff49d5c7-bh94t\" (UID: \"b69645e0-943e-4301-982a-9d1532d25e56\") " pod="openshift-image-registry/image-registry-84ff49d5c7-bh94t" Sep 10 15:43:09.088960 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:09.088864 2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 15:43:09.089213 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:09.088983 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6af5593-a983-4c35-95e9-3c166bdbf385-metrics-certs podName:e6af5593-a983-4c35-95e9-3c166bdbf385 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:09.588963185 +0000 UTC m=+66.564223131 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e6af5593-a983-4c35-95e9-3c166bdbf385-metrics-certs") pod "router-default-b4f467c4b-xjkv7" (UID: "e6af5593-a983-4c35-95e9-3c166bdbf385") : secret "router-metrics-certs-default" not found Sep 10 15:43:09.089213 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:09.089003 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e6af5593-a983-4c35-95e9-3c166bdbf385-service-ca-bundle podName:e6af5593-a983-4c35-95e9-3c166bdbf385 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:09.588992345 +0000 UTC m=+66.564252277 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e6af5593-a983-4c35-95e9-3c166bdbf385-service-ca-bundle") pod "router-default-b4f467c4b-xjkv7" (UID: "e6af5593-a983-4c35-95e9-3c166bdbf385") : configmap references non-existent config key: service-ca.crt Sep 10 15:43:09.089213 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:09.088916 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6af5593-a983-4c35-95e9-3c166bdbf385-service-ca-bundle\") pod \"router-default-b4f467c4b-xjkv7\" (UID: \"e6af5593-a983-4c35-95e9-3c166bdbf385\") " pod="openshift-ingress/router-default-b4f467c4b-xjkv7" Sep 10 15:43:09.089213 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:09.088907 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 15:43:09.089213 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:09.089112 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b69645e0-943e-4301-982a-9d1532d25e56-ca-trust-extracted\") pod \"image-registry-84ff49d5c7-bh94t\" (UID: \"b69645e0-943e-4301-982a-9d1532d25e56\") " pod="openshift-image-registry/image-registry-84ff49d5c7-bh94t" Sep 10 15:43:09.089213 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:09.089123 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84ff49d5c7-bh94t: secret "image-registry-tls" not found Sep 10 15:43:09.089213 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:09.089156 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b69645e0-943e-4301-982a-9d1532d25e56-registry-tls podName:b69645e0-943e-4301-982a-9d1532d25e56 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:09.58914542 +0000 UTC m=+66.564405352 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b69645e0-943e-4301-982a-9d1532d25e56-registry-tls") pod "image-registry-84ff49d5c7-bh94t" (UID: "b69645e0-943e-4301-982a-9d1532d25e56") : secret "image-registry-tls" not found Sep 10 15:43:09.089213 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:09.089186 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b69645e0-943e-4301-982a-9d1532d25e56-registry-certificates\") pod \"image-registry-84ff49d5c7-bh94t\" (UID: \"b69645e0-943e-4301-982a-9d1532d25e56\") " pod="openshift-image-registry/image-registry-84ff49d5c7-bh94t" Sep 10 15:43:09.089213 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:09.089218 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e6af5593-a983-4c35-95e9-3c166bdbf385-stats-auth\") pod \"router-default-b4f467c4b-xjkv7\" (UID: \"e6af5593-a983-4c35-95e9-3c166bdbf385\") " pod="openshift-ingress/router-default-b4f467c4b-xjkv7" Sep 10 15:43:09.089726 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:09.089265 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rdw8\" (UniqueName: \"kubernetes.io/projected/e6af5593-a983-4c35-95e9-3c166bdbf385-kube-api-access-9rdw8\") pod \"router-default-b4f467c4b-xjkv7\" (UID: \"e6af5593-a983-4c35-95e9-3c166bdbf385\") " pod="openshift-ingress/router-default-b4f467c4b-xjkv7" Sep 10 15:43:09.089726 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:09.089673 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b69645e0-943e-4301-982a-9d1532d25e56-ca-trust-extracted\") pod \"image-registry-84ff49d5c7-bh94t\" (UID: \"b69645e0-943e-4301-982a-9d1532d25e56\") " pod="openshift-image-registry/image-registry-84ff49d5c7-bh94t" Sep 10 15:43:09.089842 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:09.089739 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b69645e0-943e-4301-982a-9d1532d25e56-registry-certificates\") pod \"image-registry-84ff49d5c7-bh94t\" (UID: \"b69645e0-943e-4301-982a-9d1532d25e56\") " pod="openshift-image-registry/image-registry-84ff49d5c7-bh94t" Sep 10 15:43:09.089943 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:09.089919 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b69645e0-943e-4301-982a-9d1532d25e56-trusted-ca\") pod \"image-registry-84ff49d5c7-bh94t\" (UID: \"b69645e0-943e-4301-982a-9d1532d25e56\") " pod="openshift-image-registry/image-registry-84ff49d5c7-bh94t" Sep 10 15:43:09.091215 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:09.091193 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b69645e0-943e-4301-982a-9d1532d25e56-image-registry-private-configuration\") pod \"image-registry-84ff49d5c7-bh94t\" (UID: \"b69645e0-943e-4301-982a-9d1532d25e56\") " pod="openshift-image-registry/image-registry-84ff49d5c7-bh94t" Sep 10 15:43:09.091327 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:09.091307 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e6af5593-a983-4c35-95e9-3c166bdbf385-default-certificate\") pod \"router-default-b4f467c4b-xjkv7\" (UID: \"e6af5593-a983-4c35-95e9-3c166bdbf385\") " pod="openshift-ingress/router-default-b4f467c4b-xjkv7" Sep 10 15:43:09.091608 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:09.091589 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b69645e0-943e-4301-982a-9d1532d25e56-installation-pull-secrets\") pod \"image-registry-84ff49d5c7-bh94t\" (UID: \"b69645e0-943e-4301-982a-9d1532d25e56\") " pod="openshift-image-registry/image-registry-84ff49d5c7-bh94t" Sep 10 15:43:09.091670 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:09.091654 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e6af5593-a983-4c35-95e9-3c166bdbf385-stats-auth\") pod \"router-default-b4f467c4b-xjkv7\" (UID: \"e6af5593-a983-4c35-95e9-3c166bdbf385\") " pod="openshift-ingress/router-default-b4f467c4b-xjkv7" Sep 10 15:43:09.099683 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:09.099658 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rdw8\" (UniqueName: \"kubernetes.io/projected/e6af5593-a983-4c35-95e9-3c166bdbf385-kube-api-access-9rdw8\") pod \"router-default-b4f467c4b-xjkv7\" (UID: \"e6af5593-a983-4c35-95e9-3c166bdbf385\") " pod="openshift-ingress/router-default-b4f467c4b-xjkv7" Sep 10 15:43:09.099921 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:09.099903 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b69645e0-943e-4301-982a-9d1532d25e56-bound-sa-token\") pod \"image-registry-84ff49d5c7-bh94t\" (UID: \"b69645e0-943e-4301-982a-9d1532d25e56\") " pod="openshift-image-registry/image-registry-84ff49d5c7-bh94t" Sep 10 15:43:09.100074 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:09.100059 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qw9q\" (UniqueName: \"kubernetes.io/projected/b69645e0-943e-4301-982a-9d1532d25e56-kube-api-access-5qw9q\") pod \"image-registry-84ff49d5c7-bh94t\" (UID: \"b69645e0-943e-4301-982a-9d1532d25e56\") " pod="openshift-image-registry/image-registry-84ff49d5c7-bh94t" Sep 10 15:43:09.100817 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:09.100800 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-xrvfj" Sep 10 15:43:09.166026 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:09.165994 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:09.290608 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:09.290540 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdf61327-2614-4068-974c-a5c0f7c8e3e0-original-pull-secret\") pod \"global-pull-secret-syncer-swxpd\" (UID: \"bdf61327-2614-4068-974c-a5c0f7c8e3e0\") " pod="kube-system/global-pull-secret-syncer-swxpd" Sep 10 15:43:09.290727 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:09.290645 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcf35967-2a15-4bd4-8837-a058d81d1aff-metrics-certs\") pod \"network-metrics-daemon-qztq2\" (UID: \"bcf35967-2a15-4bd4-8837-a058d81d1aff\") " pod="openshift-multus/network-metrics-daemon-qztq2" Sep 10 15:43:09.292876 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:09.292856 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 10 15:43:09.292968 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:09.292863 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 10 15:43:09.301634 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:09.301616 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 10 15:43:09.301713 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:09.301687 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcf35967-2a15-4bd4-8837-a058d81d1aff-metrics-certs podName:bcf35967-2a15-4bd4-8837-a058d81d1aff nodeName:}" failed. No retries permitted until 2026-09-10 15:44:13.301666833 +0000 UTC m=+130.276926780 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bcf35967-2a15-4bd4-8837-a058d81d1aff-metrics-certs") pod "network-metrics-daemon-qztq2" (UID: "bcf35967-2a15-4bd4-8837-a058d81d1aff") : secret "metrics-daemon-secret" not found Sep 10 15:43:09.303627 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:09.303608 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/bdf61327-2614-4068-974c-a5c0f7c8e3e0-original-pull-secret\") pod \"global-pull-secret-syncer-swxpd\" (UID: \"bdf61327-2614-4068-974c-a5c0f7c8e3e0\") " pod="kube-system/global-pull-secret-syncer-swxpd" Sep 10 15:43:09.368785 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:09.368765 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-xrvfj"] Sep 10 15:43:09.371921 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:43:09.371894 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d7bbb7b_e101_4a09_9211_b2ee0dadd7c6.slice/crio-064b5248092e82830f7285553f9b23b263069c161cc40d9760f8531fadd6686b WatchSource:0}: Error finding container 064b5248092e82830f7285553f9b23b263069c161cc40d9760f8531fadd6686b: Status 404 returned error can't find the container with id 064b5248092e82830f7285553f9b23b263069c161cc40d9760f8531fadd6686b Sep 10 15:43:09.391171 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:09.391150 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q5c4\" (UniqueName: \"kubernetes.io/projected/d88aeb8f-b005-4deb-bc50-5f843a53699c-kube-api-access-2q5c4\") pod \"network-check-target-wqgsp\" (UID: \"d88aeb8f-b005-4deb-bc50-5f843a53699c\") " pod="openshift-network-diagnostics/network-check-target-wqgsp" Sep 10 15:43:09.393262 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:09.393245 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 10 15:43:09.403932 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:09.403915 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 10 15:43:09.414974 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:09.414948 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q5c4\" (UniqueName: \"kubernetes.io/projected/d88aeb8f-b005-4deb-bc50-5f843a53699c-kube-api-access-2q5c4\") pod \"network-check-target-wqgsp\" (UID: \"d88aeb8f-b005-4deb-bc50-5f843a53699c\") " pod="openshift-network-diagnostics/network-check-target-wqgsp" Sep 10 15:43:09.419696 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:09.419681 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-swxpd" Sep 10 15:43:09.434801 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:09.434784 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cthzr\"" Sep 10 15:43:09.443468 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:09.443442 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wqgsp" Sep 10 15:43:09.592456 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:09.592396 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6af5593-a983-4c35-95e9-3c166bdbf385-service-ca-bundle\") pod \"router-default-b4f467c4b-xjkv7\" (UID: \"e6af5593-a983-4c35-95e9-3c166bdbf385\") " pod="openshift-ingress/router-default-b4f467c4b-xjkv7" Sep 10 15:43:09.592562 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:09.592461 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6af5593-a983-4c35-95e9-3c166bdbf385-metrics-certs\") pod \"router-default-b4f467c4b-xjkv7\" (UID: \"e6af5593-a983-4c35-95e9-3c166bdbf385\") " pod="openshift-ingress/router-default-b4f467c4b-xjkv7" Sep 10 15:43:09.592562 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:09.592493 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b69645e0-943e-4301-982a-9d1532d25e56-registry-tls\") pod \"image-registry-84ff49d5c7-bh94t\" (UID: \"b69645e0-943e-4301-982a-9d1532d25e56\") " pod="openshift-image-registry/image-registry-84ff49d5c7-bh94t" Sep 10 15:43:09.592639 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:09.592563 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e6af5593-a983-4c35-95e9-3c166bdbf385-service-ca-bundle podName:e6af5593-a983-4c35-95e9-3c166bdbf385 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:10.592544171 +0000 UTC m=+67.567804121 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e6af5593-a983-4c35-95e9-3c166bdbf385-service-ca-bundle") pod "router-default-b4f467c4b-xjkv7" (UID: "e6af5593-a983-4c35-95e9-3c166bdbf385") : configmap references non-existent config key: service-ca.crt Sep 10 15:43:09.592639 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:09.592594 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 15:43:09.592639 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:09.592607 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84ff49d5c7-bh94t: secret "image-registry-tls" not found Sep 10 15:43:09.592639 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:09.592609 2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 15:43:09.592639 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:09.592642 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b69645e0-943e-4301-982a-9d1532d25e56-registry-tls podName:b69645e0-943e-4301-982a-9d1532d25e56 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:10.592631788 +0000 UTC m=+67.567891719 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b69645e0-943e-4301-982a-9d1532d25e56-registry-tls") pod "image-registry-84ff49d5c7-bh94t" (UID: "b69645e0-943e-4301-982a-9d1532d25e56") : secret "image-registry-tls" not found Sep 10 15:43:09.592814 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:09.592657 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6af5593-a983-4c35-95e9-3c166bdbf385-metrics-certs podName:e6af5593-a983-4c35-95e9-3c166bdbf385 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:10.592646432 +0000 UTC m=+67.567906363 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e6af5593-a983-4c35-95e9-3c166bdbf385-metrics-certs") pod "router-default-b4f467c4b-xjkv7" (UID: "e6af5593-a983-4c35-95e9-3c166bdbf385") : secret "router-metrics-certs-default" not found Sep 10 15:43:09.687328 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:09.687301 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-swxpd"] Sep 10 15:43:09.690055 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:43:09.690031 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdf61327_2614_4068_974c_a5c0f7c8e3e0.slice/crio-b67b44091605ed61749447c89665cf3d0438e2f688acea5a8a5a91fca1963b0e WatchSource:0}: Error finding container b67b44091605ed61749447c89665cf3d0438e2f688acea5a8a5a91fca1963b0e: Status 404 returned error can't find the container with id b67b44091605ed61749447c89665cf3d0438e2f688acea5a8a5a91fca1963b0e Sep 10 15:43:09.716157 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:09.716126 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wqgsp"] Sep 10 15:43:09.719679 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:43:09.719656 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd88aeb8f_b005_4deb_bc50_5f843a53699c.slice/crio-c244432bc3fcb6199b2e64795535c56edc5593d6a31b879fe99399172dbf8902 WatchSource:0}: Error finding container c244432bc3fcb6199b2e64795535c56edc5593d6a31b879fe99399172dbf8902: Status 404 returned error can't find the container with id c244432bc3fcb6199b2e64795535c56edc5593d6a31b879fe99399172dbf8902 Sep 10 15:43:09.757902 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:09.757874 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-xrvfj" event={"ID":"6d7bbb7b-e101-4a09-9211-b2ee0dadd7c6","Type":"ContainerStarted","Data":"064b5248092e82830f7285553f9b23b263069c161cc40d9760f8531fadd6686b"} Sep 10 15:43:09.758952 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:09.758930 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wqgsp" event={"ID":"d88aeb8f-b005-4deb-bc50-5f843a53699c","Type":"ContainerStarted","Data":"c244432bc3fcb6199b2e64795535c56edc5593d6a31b879fe99399172dbf8902"} Sep 10 15:43:09.759992 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:09.759971 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-swxpd" event={"ID":"bdf61327-2614-4068-974c-a5c0f7c8e3e0","Type":"ContainerStarted","Data":"b67b44091605ed61749447c89665cf3d0438e2f688acea5a8a5a91fca1963b0e"} Sep 10 15:43:10.601176 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:10.601126 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6af5593-a983-4c35-95e9-3c166bdbf385-metrics-certs\") pod \"router-default-b4f467c4b-xjkv7\" (UID: \"e6af5593-a983-4c35-95e9-3c166bdbf385\") " pod="openshift-ingress/router-default-b4f467c4b-xjkv7" Sep 10 15:43:10.601338 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:10.601194 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b69645e0-943e-4301-982a-9d1532d25e56-registry-tls\") pod \"image-registry-84ff49d5c7-bh94t\" (UID: \"b69645e0-943e-4301-982a-9d1532d25e56\") " pod="openshift-image-registry/image-registry-84ff49d5c7-bh94t" Sep 10 15:43:10.601338 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:10.601227 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6af5593-a983-4c35-95e9-3c166bdbf385-service-ca-bundle\") pod \"router-default-b4f467c4b-xjkv7\" (UID: \"e6af5593-a983-4c35-95e9-3c166bdbf385\") " pod="openshift-ingress/router-default-b4f467c4b-xjkv7" Sep 10 15:43:10.601338 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:10.601314 2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 15:43:10.601540 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:10.601367 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 15:43:10.601540 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:10.601395 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6af5593-a983-4c35-95e9-3c166bdbf385-metrics-certs podName:e6af5593-a983-4c35-95e9-3c166bdbf385 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:12.601365431 +0000 UTC m=+69.576625376 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e6af5593-a983-4c35-95e9-3c166bdbf385-metrics-certs") pod "router-default-b4f467c4b-xjkv7" (UID: "e6af5593-a983-4c35-95e9-3c166bdbf385") : secret "router-metrics-certs-default" not found Sep 10 15:43:10.601540 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:10.601399 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84ff49d5c7-bh94t: secret "image-registry-tls" not found Sep 10 15:43:10.601540 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:10.601415 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e6af5593-a983-4c35-95e9-3c166bdbf385-service-ca-bundle podName:e6af5593-a983-4c35-95e9-3c166bdbf385 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:12.601406447 +0000 UTC m=+69.576666386 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e6af5593-a983-4c35-95e9-3c166bdbf385-service-ca-bundle") pod "router-default-b4f467c4b-xjkv7" (UID: "e6af5593-a983-4c35-95e9-3c166bdbf385") : configmap references non-existent config key: service-ca.crt Sep 10 15:43:10.601540 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:10.601443 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b69645e0-943e-4301-982a-9d1532d25e56-registry-tls podName:b69645e0-943e-4301-982a-9d1532d25e56 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:12.601427544 +0000 UTC m=+69.576687494 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b69645e0-943e-4301-982a-9d1532d25e56-registry-tls") pod "image-registry-84ff49d5c7-bh94t" (UID: "b69645e0-943e-4301-982a-9d1532d25e56") : secret "image-registry-tls" not found Sep 10 15:43:12.618442 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:12.618404 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6af5593-a983-4c35-95e9-3c166bdbf385-service-ca-bundle\") pod \"router-default-b4f467c4b-xjkv7\" (UID: \"e6af5593-a983-4c35-95e9-3c166bdbf385\") " pod="openshift-ingress/router-default-b4f467c4b-xjkv7" Sep 10 15:43:12.618995 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:12.618507 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6af5593-a983-4c35-95e9-3c166bdbf385-metrics-certs\") pod \"router-default-b4f467c4b-xjkv7\" (UID: \"e6af5593-a983-4c35-95e9-3c166bdbf385\") " pod="openshift-ingress/router-default-b4f467c4b-xjkv7" Sep 10 15:43:12.618995 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:12.618548 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b69645e0-943e-4301-982a-9d1532d25e56-registry-tls\") pod \"image-registry-84ff49d5c7-bh94t\" (UID: \"b69645e0-943e-4301-982a-9d1532d25e56\") " pod="openshift-image-registry/image-registry-84ff49d5c7-bh94t" Sep 10 15:43:12.618995 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:12.618604 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e6af5593-a983-4c35-95e9-3c166bdbf385-service-ca-bundle podName:e6af5593-a983-4c35-95e9-3c166bdbf385 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:16.618579216 +0000 UTC m=+73.593839153 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e6af5593-a983-4c35-95e9-3c166bdbf385-service-ca-bundle") pod "router-default-b4f467c4b-xjkv7" (UID: "e6af5593-a983-4c35-95e9-3c166bdbf385") : configmap references non-existent config key: service-ca.crt Sep 10 15:43:12.618995 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:12.618662 2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 15:43:12.618995 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:12.618720 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6af5593-a983-4c35-95e9-3c166bdbf385-metrics-certs podName:e6af5593-a983-4c35-95e9-3c166bdbf385 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:16.618703597 +0000 UTC m=+73.593963546 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e6af5593-a983-4c35-95e9-3c166bdbf385-metrics-certs") pod "router-default-b4f467c4b-xjkv7" (UID: "e6af5593-a983-4c35-95e9-3c166bdbf385") : secret "router-metrics-certs-default" not found Sep 10 15:43:12.618995 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:12.618665 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 15:43:12.618995 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:12.618746 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84ff49d5c7-bh94t: secret "image-registry-tls" not found Sep 10 15:43:12.618995 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:12.618792 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b69645e0-943e-4301-982a-9d1532d25e56-registry-tls podName:b69645e0-943e-4301-982a-9d1532d25e56 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:16.618779087 +0000 UTC m=+73.594039030 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b69645e0-943e-4301-982a-9d1532d25e56-registry-tls") pod "image-registry-84ff49d5c7-bh94t" (UID: "b69645e0-943e-4301-982a-9d1532d25e56") : secret "image-registry-tls" not found Sep 10 15:43:14.776083 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:14.776049 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-xrvfj" event={"ID":"6d7bbb7b-e101-4a09-9211-b2ee0dadd7c6","Type":"ContainerStarted","Data":"2aee758d7f068d0b5f9e1f978e16a1c98cc1718ab2fa625a071026c0e067b97b"} Sep 10 15:43:14.777595 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:14.777571 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wqgsp" event={"ID":"d88aeb8f-b005-4deb-bc50-5f843a53699c","Type":"ContainerStarted","Data":"1a4eccb9a009122d95935155a34015e132f0e943e973d31b159008fa3d572fc2"} Sep 10 15:43:14.777718 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:14.777619 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wqgsp" Sep 10 15:43:14.779024 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:14.779002 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-swxpd" event={"ID":"bdf61327-2614-4068-974c-a5c0f7c8e3e0","Type":"ContainerStarted","Data":"1f47311cf50759ca5631f1efb75986bd1774651494683a6c7835d36c11f8cc7c"} Sep 10 15:43:14.791914 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:14.791876 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-xrvfj" podStartSLOduration=2.088990045 podStartE2EDuration="6.791863773s" totalImagesPullingTime="4.702873728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:43:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:14.791199428 +0000 UTC m=+71.766459380" watchObservedRunningTime="2026-09-10 15:43:14.791863773 +0000 UTC m=+71.767123744" Sep 10 15:43:14.804334 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:14.804292 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-swxpd" podStartSLOduration=66.407372273 podStartE2EDuration="1m10.804275924s" totalImagesPullingTime="4.396903651s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:14.804122283 +0000 UTC m=+71.779382238" watchObservedRunningTime="2026-09-10 15:43:14.804275924 +0000 UTC m=+71.779535878" Sep 10 15:43:14.819018 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:14.818974 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wqgsp" podStartSLOduration=67.454470147 podStartE2EDuration="1m11.818958776s" totalImagesPullingTime="4.364488629s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:42:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:14.818715616 +0000 UTC m=+71.793975573" watchObservedRunningTime="2026-09-10 15:43:14.818958776 +0000 UTC m=+71.794218730" Sep 10 15:43:15.227929 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:15.227899 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:15.331958 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:15.331934 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:15.669928 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:15.669899 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:15.736265 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:15.736242 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:15.954134 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:15.954064 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:16.032937 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:16.032907 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:16.133176 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:16.133142 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:16.649836 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:16.649800 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6af5593-a983-4c35-95e9-3c166bdbf385-service-ca-bundle\") pod \"router-default-b4f467c4b-xjkv7\" (UID: \"e6af5593-a983-4c35-95e9-3c166bdbf385\") " pod="openshift-ingress/router-default-b4f467c4b-xjkv7" Sep 10 15:43:16.649994 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:16.649856 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6af5593-a983-4c35-95e9-3c166bdbf385-metrics-certs\") pod \"router-default-b4f467c4b-xjkv7\" (UID: \"e6af5593-a983-4c35-95e9-3c166bdbf385\") " pod="openshift-ingress/router-default-b4f467c4b-xjkv7" Sep 10 15:43:16.649994 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:16.649940 2583 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Sep 10 15:43:16.650065 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:16.649988 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b69645e0-943e-4301-982a-9d1532d25e56-registry-tls\") pod \"image-registry-84ff49d5c7-bh94t\" (UID: \"b69645e0-943e-4301-982a-9d1532d25e56\") " pod="openshift-image-registry/image-registry-84ff49d5c7-bh94t" Sep 10 15:43:16.650065 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:16.649996 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6af5593-a983-4c35-95e9-3c166bdbf385-metrics-certs podName:e6af5593-a983-4c35-95e9-3c166bdbf385 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:24.649978728 +0000 UTC m=+81.625238675 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e6af5593-a983-4c35-95e9-3c166bdbf385-metrics-certs") pod "router-default-b4f467c4b-xjkv7" (UID: "e6af5593-a983-4c35-95e9-3c166bdbf385") : secret "router-metrics-certs-default" not found Sep 10 15:43:16.650065 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:16.650036 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e6af5593-a983-4c35-95e9-3c166bdbf385-service-ca-bundle podName:e6af5593-a983-4c35-95e9-3c166bdbf385 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:24.650017994 +0000 UTC m=+81.625277924 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e6af5593-a983-4c35-95e9-3c166bdbf385-service-ca-bundle") pod "router-default-b4f467c4b-xjkv7" (UID: "e6af5593-a983-4c35-95e9-3c166bdbf385") : configmap references non-existent config key: service-ca.crt Sep 10 15:43:16.650065 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:16.650049 2583 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 10 15:43:16.650065 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:16.650058 2583 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-84ff49d5c7-bh94t: secret "image-registry-tls" not found Sep 10 15:43:16.650221 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:16.650081 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b69645e0-943e-4301-982a-9d1532d25e56-registry-tls podName:b69645e0-943e-4301-982a-9d1532d25e56 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:24.650073851 +0000 UTC m=+81.625333782 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b69645e0-943e-4301-982a-9d1532d25e56-registry-tls") pod "image-registry-84ff49d5c7-bh94t" (UID: "b69645e0-943e-4301-982a-9d1532d25e56") : secret "image-registry-tls" not found Sep 10 15:43:16.751001 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:16.750961 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:16.772468 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:16.772441 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-8v6zg"] Sep 10 15:43:16.774430 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:16.774416 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-8v6zg" Sep 10 15:43:16.776265 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:16.776245 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-6k9fj\"" Sep 10 15:43:16.776917 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:16.776902 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 10 15:43:16.776968 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:16.776934 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 10 15:43:16.783340 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:16.783323 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-8v6zg"] Sep 10 15:43:16.821841 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:16.821815 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:16.851168 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:16.851147 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8jvp\" (UniqueName: \"kubernetes.io/projected/815141e9-52d6-47c9-9c59-71e6f3fc385c-kube-api-access-k8jvp\") pod \"migrator-fdbb8fdff-8v6zg\" (UID: \"815141e9-52d6-47c9-9c59-71e6f3fc385c\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-8v6zg" Sep 10 15:43:16.914644 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:16.914591 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:16.926589 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:16.926568 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8wtqm"] Sep 10 15:43:16.928812 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:16.928797 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8wtqm" Sep 10 15:43:16.930670 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:16.930648 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 10 15:43:16.930985 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:16.930969 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 10 15:43:16.930985 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:16.930978 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-89tcq\"" Sep 10 15:43:16.931087 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:16.931043 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 10 15:43:16.931248 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:16.931232 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 10 15:43:16.938665 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:16.938645 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8wtqm"] Sep 10 15:43:16.952062 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:16.952043 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8jvp\" (UniqueName: \"kubernetes.io/projected/815141e9-52d6-47c9-9c59-71e6f3fc385c-kube-api-access-k8jvp\") pod \"migrator-fdbb8fdff-8v6zg\" (UID: \"815141e9-52d6-47c9-9c59-71e6f3fc385c\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-8v6zg" Sep 10 15:43:16.952154 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:16.952071 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1949768-9a66-4404-b7f1-c71ae6520f36-data-volume\") pod \"insights-runtime-extractor-8wtqm\" (UID: \"b1949768-9a66-4404-b7f1-c71ae6520f36\") " pod="openshift-insights/insights-runtime-extractor-8wtqm" Sep 10 15:43:16.952154 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:16.952089 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1949768-9a66-4404-b7f1-c71ae6520f36-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8wtqm\" (UID: \"b1949768-9a66-4404-b7f1-c71ae6520f36\") " pod="openshift-insights/insights-runtime-extractor-8wtqm" Sep 10 15:43:16.952154 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:16.952109 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1949768-9a66-4404-b7f1-c71ae6520f36-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8wtqm\" (UID: \"b1949768-9a66-4404-b7f1-c71ae6520f36\") " pod="openshift-insights/insights-runtime-extractor-8wtqm" Sep 10 15:43:16.952154 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:16.952131 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b1949768-9a66-4404-b7f1-c71ae6520f36-crio-socket\") pod \"insights-runtime-extractor-8wtqm\" (UID: \"b1949768-9a66-4404-b7f1-c71ae6520f36\") " pod="openshift-insights/insights-runtime-extractor-8wtqm" Sep 10 15:43:16.952295 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:16.952169 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrtzg\" (UniqueName: \"kubernetes.io/projected/b1949768-9a66-4404-b7f1-c71ae6520f36-kube-api-access-hrtzg\") pod \"insights-runtime-extractor-8wtqm\" (UID: \"b1949768-9a66-4404-b7f1-c71ae6520f36\") " pod="openshift-insights/insights-runtime-extractor-8wtqm" Sep 10 15:43:16.961932 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:16.961912 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8jvp\" (UniqueName: \"kubernetes.io/projected/815141e9-52d6-47c9-9c59-71e6f3fc385c-kube-api-access-k8jvp\") pod \"migrator-fdbb8fdff-8v6zg\" (UID: \"815141e9-52d6-47c9-9c59-71e6f3fc385c\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-8v6zg" Sep 10 15:43:16.974589 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:16.974568 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:17.053042 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:17.053017 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1949768-9a66-4404-b7f1-c71ae6520f36-data-volume\") pod \"insights-runtime-extractor-8wtqm\" (UID: \"b1949768-9a66-4404-b7f1-c71ae6520f36\") " pod="openshift-insights/insights-runtime-extractor-8wtqm" Sep 10 15:43:17.053139 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:17.053047 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1949768-9a66-4404-b7f1-c71ae6520f36-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8wtqm\" (UID: \"b1949768-9a66-4404-b7f1-c71ae6520f36\") " pod="openshift-insights/insights-runtime-extractor-8wtqm" Sep 10 15:43:17.053139 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:17.053069 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1949768-9a66-4404-b7f1-c71ae6520f36-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8wtqm\" (UID: \"b1949768-9a66-4404-b7f1-c71ae6520f36\") " pod="openshift-insights/insights-runtime-extractor-8wtqm" Sep 10 15:43:17.053139 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:17.053085 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b1949768-9a66-4404-b7f1-c71ae6520f36-crio-socket\") pod \"insights-runtime-extractor-8wtqm\" (UID: \"b1949768-9a66-4404-b7f1-c71ae6520f36\") " pod="openshift-insights/insights-runtime-extractor-8wtqm" Sep 10 15:43:17.053288 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:17.053151 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b1949768-9a66-4404-b7f1-c71ae6520f36-crio-socket\") pod \"insights-runtime-extractor-8wtqm\" (UID: \"b1949768-9a66-4404-b7f1-c71ae6520f36\") " pod="openshift-insights/insights-runtime-extractor-8wtqm" Sep 10 15:43:17.053288 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:17.053163 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 15:43:17.053288 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:17.053195 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrtzg\" (UniqueName: \"kubernetes.io/projected/b1949768-9a66-4404-b7f1-c71ae6520f36-kube-api-access-hrtzg\") pod \"insights-runtime-extractor-8wtqm\" (UID: \"b1949768-9a66-4404-b7f1-c71ae6520f36\") " pod="openshift-insights/insights-runtime-extractor-8wtqm" Sep 10 15:43:17.053288 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:17.053221 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b1949768-9a66-4404-b7f1-c71ae6520f36-kube-rbac-proxy-cm podName:b1949768-9a66-4404-b7f1-c71ae6520f36 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:17.553207917 +0000 UTC m=+74.528467853 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b1949768-9a66-4404-b7f1-c71ae6520f36-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8wtqm" (UID: "b1949768-9a66-4404-b7f1-c71ae6520f36") : configmap "kube-rbac-proxy" not found Sep 10 15:43:17.053288 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:17.053164 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 15:43:17.053519 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:17.053301 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1949768-9a66-4404-b7f1-c71ae6520f36-insights-runtime-extractor-tls podName:b1949768-9a66-4404-b7f1-c71ae6520f36 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:17.553285586 +0000 UTC m=+74.528545536 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b1949768-9a66-4404-b7f1-c71ae6520f36-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8wtqm" (UID: "b1949768-9a66-4404-b7f1-c71ae6520f36") : secret "insights-runtime-extractor-tls" not found Sep 10 15:43:17.053519 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:17.053373 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1949768-9a66-4404-b7f1-c71ae6520f36-data-volume\") pod \"insights-runtime-extractor-8wtqm\" (UID: \"b1949768-9a66-4404-b7f1-c71ae6520f36\") " pod="openshift-insights/insights-runtime-extractor-8wtqm" Sep 10 15:43:17.061460 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:17.061435 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrtzg\" (UniqueName: \"kubernetes.io/projected/b1949768-9a66-4404-b7f1-c71ae6520f36-kube-api-access-hrtzg\") pod \"insights-runtime-extractor-8wtqm\" (UID: \"b1949768-9a66-4404-b7f1-c71ae6520f36\") " pod="openshift-insights/insights-runtime-extractor-8wtqm" Sep 10 15:43:17.083306 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:17.083284 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-8v6zg" Sep 10 15:43:17.350168 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:17.350090 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-8v6zg"] Sep 10 15:43:17.353699 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:43:17.353654 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod815141e9_52d6_47c9_9c59_71e6f3fc385c.slice/crio-05a4b58ee22c9fbff58037c856d8c7217289e24a33b77cf4c40fc8314b4bcc3f WatchSource:0}: Error finding container 05a4b58ee22c9fbff58037c856d8c7217289e24a33b77cf4c40fc8314b4bcc3f: Status 404 returned error can't find the container with id 05a4b58ee22c9fbff58037c856d8c7217289e24a33b77cf4c40fc8314b4bcc3f Sep 10 15:43:17.531576 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:17.531547 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-hghdb"] Sep 10 15:43:17.534069 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:17.534052 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-hghdb" Sep 10 15:43:17.536269 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:17.536247 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Sep 10 15:43:17.536396 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:17.536323 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-6frbp\"" Sep 10 15:43:17.536396 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:17.536339 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Sep 10 15:43:17.536396 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:17.536351 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Sep 10 15:43:17.537203 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:17.537180 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Sep 10 15:43:17.545004 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:17.544984 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-hghdb"] Sep 10 15:43:17.556267 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:17.556246 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg46w\" (UniqueName: \"kubernetes.io/projected/9ca7faee-85cb-47b3-99a1-574afc795a17-kube-api-access-sg46w\") pod \"service-ca-75d99c484-hghdb\" (UID: \"9ca7faee-85cb-47b3-99a1-574afc795a17\") " pod="openshift-service-ca/service-ca-75d99c484-hghdb" Sep 10 15:43:17.556351 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:17.556273 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9ca7faee-85cb-47b3-99a1-574afc795a17-signing-cabundle\") pod \"service-ca-75d99c484-hghdb\" (UID: \"9ca7faee-85cb-47b3-99a1-574afc795a17\") " pod="openshift-service-ca/service-ca-75d99c484-hghdb" Sep 10 15:43:17.556351 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:17.556304 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1949768-9a66-4404-b7f1-c71ae6520f36-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8wtqm\" (UID: \"b1949768-9a66-4404-b7f1-c71ae6520f36\") " pod="openshift-insights/insights-runtime-extractor-8wtqm" Sep 10 15:43:17.556539 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:17.556403 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 15:43:17.556539 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:17.556470 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b1949768-9a66-4404-b7f1-c71ae6520f36-kube-rbac-proxy-cm podName:b1949768-9a66-4404-b7f1-c71ae6520f36 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:18.556448927 +0000 UTC m=+75.531708875 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b1949768-9a66-4404-b7f1-c71ae6520f36-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8wtqm" (UID: "b1949768-9a66-4404-b7f1-c71ae6520f36") : configmap "kube-rbac-proxy" not found Sep 10 15:43:17.556539 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:17.556494 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1949768-9a66-4404-b7f1-c71ae6520f36-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8wtqm\" (UID: \"b1949768-9a66-4404-b7f1-c71ae6520f36\") " pod="openshift-insights/insights-runtime-extractor-8wtqm" Sep 10 15:43:17.556668 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:17.556537 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9ca7faee-85cb-47b3-99a1-574afc795a17-signing-key\") pod \"service-ca-75d99c484-hghdb\" (UID: \"9ca7faee-85cb-47b3-99a1-574afc795a17\") " pod="openshift-service-ca/service-ca-75d99c484-hghdb" Sep 10 15:43:17.557183 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:17.557080 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 15:43:17.557183 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:17.557153 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1949768-9a66-4404-b7f1-c71ae6520f36-insights-runtime-extractor-tls podName:b1949768-9a66-4404-b7f1-c71ae6520f36 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:18.557135602 +0000 UTC m=+75.532395532 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b1949768-9a66-4404-b7f1-c71ae6520f36-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8wtqm" (UID: "b1949768-9a66-4404-b7f1-c71ae6520f36") : secret "insights-runtime-extractor-tls" not found Sep 10 15:43:17.589300 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:17.589272 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:17.657824 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:17.657800 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9ca7faee-85cb-47b3-99a1-574afc795a17-signing-key\") pod \"service-ca-75d99c484-hghdb\" (UID: \"9ca7faee-85cb-47b3-99a1-574afc795a17\") " pod="openshift-service-ca/service-ca-75d99c484-hghdb" Sep 10 15:43:17.657913 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:17.657839 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg46w\" (UniqueName: \"kubernetes.io/projected/9ca7faee-85cb-47b3-99a1-574afc795a17-kube-api-access-sg46w\") pod \"service-ca-75d99c484-hghdb\" (UID: \"9ca7faee-85cb-47b3-99a1-574afc795a17\") " pod="openshift-service-ca/service-ca-75d99c484-hghdb" Sep 10 15:43:17.657913 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:17.657859 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9ca7faee-85cb-47b3-99a1-574afc795a17-signing-cabundle\") pod \"service-ca-75d99c484-hghdb\" (UID: \"9ca7faee-85cb-47b3-99a1-574afc795a17\") " pod="openshift-service-ca/service-ca-75d99c484-hghdb" Sep 10 15:43:17.658521 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:17.658502 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/9ca7faee-85cb-47b3-99a1-574afc795a17-signing-cabundle\") pod \"service-ca-75d99c484-hghdb\" (UID: \"9ca7faee-85cb-47b3-99a1-574afc795a17\") " pod="openshift-service-ca/service-ca-75d99c484-hghdb" Sep 10 15:43:17.660097 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:17.660079 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/9ca7faee-85cb-47b3-99a1-574afc795a17-signing-key\") pod \"service-ca-75d99c484-hghdb\" (UID: \"9ca7faee-85cb-47b3-99a1-574afc795a17\") " pod="openshift-service-ca/service-ca-75d99c484-hghdb" Sep 10 15:43:17.668327 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:17.668309 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg46w\" (UniqueName: \"kubernetes.io/projected/9ca7faee-85cb-47b3-99a1-574afc795a17-kube-api-access-sg46w\") pod \"service-ca-75d99c484-hghdb\" (UID: \"9ca7faee-85cb-47b3-99a1-574afc795a17\") " pod="openshift-service-ca/service-ca-75d99c484-hghdb" Sep 10 15:43:17.691959 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:17.691934 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:17.787270 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:17.787241 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-8v6zg" event={"ID":"815141e9-52d6-47c9-9c59-71e6f3fc385c","Type":"ContainerStarted","Data":"05a4b58ee22c9fbff58037c856d8c7217289e24a33b77cf4c40fc8314b4bcc3f"} Sep 10 15:43:17.842654 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:17.842635 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-hghdb" Sep 10 15:43:18.114648 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:18.114571 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-hghdb"] Sep 10 15:43:18.118048 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:43:18.118020 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ca7faee_85cb_47b3_99a1_574afc795a17.slice/crio-6ef62e7e34e46edd6013acdc29e8b6866b423ba719e5d228f61e5c034806ab26 WatchSource:0}: Error finding container 6ef62e7e34e46edd6013acdc29e8b6866b423ba719e5d228f61e5c034806ab26: Status 404 returned error can't find the container with id 6ef62e7e34e46edd6013acdc29e8b6866b423ba719e5d228f61e5c034806ab26 Sep 10 15:43:18.564475 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:18.564450 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1949768-9a66-4404-b7f1-c71ae6520f36-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8wtqm\" (UID: \"b1949768-9a66-4404-b7f1-c71ae6520f36\") " pod="openshift-insights/insights-runtime-extractor-8wtqm" Sep 10 15:43:18.564580 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:18.564526 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1949768-9a66-4404-b7f1-c71ae6520f36-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8wtqm\" (UID: \"b1949768-9a66-4404-b7f1-c71ae6520f36\") " pod="openshift-insights/insights-runtime-extractor-8wtqm" Sep 10 15:43:18.564630 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:18.564584 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 15:43:18.564671 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:18.564663 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b1949768-9a66-4404-b7f1-c71ae6520f36-kube-rbac-proxy-cm podName:b1949768-9a66-4404-b7f1-c71ae6520f36 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:20.564642404 +0000 UTC m=+77.539902357 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b1949768-9a66-4404-b7f1-c71ae6520f36-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8wtqm" (UID: "b1949768-9a66-4404-b7f1-c71ae6520f36") : configmap "kube-rbac-proxy" not found Sep 10 15:43:18.564728 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:18.564663 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 15:43:18.564728 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:18.564716 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1949768-9a66-4404-b7f1-c71ae6520f36-insights-runtime-extractor-tls podName:b1949768-9a66-4404-b7f1-c71ae6520f36 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:20.564704977 +0000 UTC m=+77.539964932 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b1949768-9a66-4404-b7f1-c71ae6520f36-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8wtqm" (UID: "b1949768-9a66-4404-b7f1-c71ae6520f36") : secret "insights-runtime-extractor-tls" not found Sep 10 15:43:18.791916 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:18.791838 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-hghdb" event={"ID":"9ca7faee-85cb-47b3-99a1-574afc795a17","Type":"ContainerStarted","Data":"10dd9dac45956d0b822af9db646fe2dd6e5f5d5d5a983adce608bcb8482eabcc"} Sep 10 15:43:18.791916 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:18.791894 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-hghdb" event={"ID":"9ca7faee-85cb-47b3-99a1-574afc795a17","Type":"ContainerStarted","Data":"6ef62e7e34e46edd6013acdc29e8b6866b423ba719e5d228f61e5c034806ab26"} Sep 10 15:43:18.793797 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:18.793768 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-8v6zg" event={"ID":"815141e9-52d6-47c9-9c59-71e6f3fc385c","Type":"ContainerStarted","Data":"0fac1bbc7c3c34d4f29d6bd61710431d12b70fd400e86f2de830d642b6695223"} Sep 10 15:43:18.793906 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:18.793818 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-8v6zg" event={"ID":"815141e9-52d6-47c9-9c59-71e6f3fc385c","Type":"ContainerStarted","Data":"0104dd76d5c02e66adc37420f4777784ac62b19a3a956a6043f849cddd551844"} Sep 10 15:43:18.809650 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:18.809607 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-hghdb" podStartSLOduration=1.809594398 podStartE2EDuration="1.809594398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:43:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:18.808134929 +0000 UTC m=+75.783394882" watchObservedRunningTime="2026-09-10 15:43:18.809594398 +0000 UTC m=+75.784854350" Sep 10 15:43:18.832090 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:18.832048 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-8v6zg" podStartSLOduration=1.749643795 podStartE2EDuration="2.832035359s" totalImagesPullingTime="1.082391564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:43:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:18.831310757 +0000 UTC m=+75.806570711" watchObservedRunningTime="2026-09-10 15:43:18.832035359 +0000 UTC m=+75.807295311" Sep 10 15:43:18.860174 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:18.860146 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:19.684083 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:19.684060 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8v784_cb7626aa-a9d1-4dec-aab8-b68a3e6adca3/dns-node-resolver/0.log" Sep 10 15:43:19.769343 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:19.769303 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:20.024372 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:20.024290 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:20.119061 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:20.119025 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:20.235428 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:20.235400 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:20.336253 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:20.336171 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:20.467223 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:20.467191 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:20.484522 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:20.484499 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5lpqq_7dc7aa02-57e8-4cdc-aab6-9875e1427825/node-ca/0.log" Sep 10 15:43:20.582099 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:20.582066 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1949768-9a66-4404-b7f1-c71ae6520f36-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8wtqm\" (UID: \"b1949768-9a66-4404-b7f1-c71ae6520f36\") " pod="openshift-insights/insights-runtime-extractor-8wtqm" Sep 10 15:43:20.582256 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:20.582139 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1949768-9a66-4404-b7f1-c71ae6520f36-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8wtqm\" (UID: \"b1949768-9a66-4404-b7f1-c71ae6520f36\") " pod="openshift-insights/insights-runtime-extractor-8wtqm" Sep 10 15:43:20.582256 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:20.582198 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 15:43:20.582256 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:20.582236 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 15:43:20.582428 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:20.582259 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b1949768-9a66-4404-b7f1-c71ae6520f36-kube-rbac-proxy-cm podName:b1949768-9a66-4404-b7f1-c71ae6520f36 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:24.58224285 +0000 UTC m=+81.557502804 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b1949768-9a66-4404-b7f1-c71ae6520f36-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8wtqm" (UID: "b1949768-9a66-4404-b7f1-c71ae6520f36") : configmap "kube-rbac-proxy" not found Sep 10 15:43:20.582428 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:20.582276 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1949768-9a66-4404-b7f1-c71ae6520f36-insights-runtime-extractor-tls podName:b1949768-9a66-4404-b7f1-c71ae6520f36 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:24.582265271 +0000 UTC m=+81.557525202 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b1949768-9a66-4404-b7f1-c71ae6520f36-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8wtqm" (UID: "b1949768-9a66-4404-b7f1-c71ae6520f36") : secret "insights-runtime-extractor-tls" not found Sep 10 15:43:21.665493 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:21.665460 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:21.684714 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:21.684688 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wqwzs_66ac0bb7-d8cd-43d7-b247-4d5e441f321a/serve-healthcheck-canary/0.log" Sep 10 15:43:21.865014 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:21.864979 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:21.884785 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:21.884760 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-8v6zg_815141e9-52d6-47c9-9c59-71e6f3fc385c/migrator/0.log" Sep 10 15:43:22.071949 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:22.071883 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:22.084356 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:22.084337 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-8v6zg_815141e9-52d6-47c9-9c59-71e6f3fc385c/graceful-termination/0.log" Sep 10 15:43:22.268962 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:22.268926 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:22.382974 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:22.382948 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:22.473109 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:22.473089 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:23.181509 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:23.181476 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:23.458909 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:23.458831 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:23.660273 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:23.660247 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:23.733211 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:23.733152 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:24.062073 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:24.061999 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:24.161214 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:24.161185 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:24.262235 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:24.262205 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:24.614249 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:24.614224 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1949768-9a66-4404-b7f1-c71ae6520f36-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8wtqm\" (UID: \"b1949768-9a66-4404-b7f1-c71ae6520f36\") " pod="openshift-insights/insights-runtime-extractor-8wtqm" Sep 10 15:43:24.614405 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:24.614291 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1949768-9a66-4404-b7f1-c71ae6520f36-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8wtqm\" (UID: \"b1949768-9a66-4404-b7f1-c71ae6520f36\") " pod="openshift-insights/insights-runtime-extractor-8wtqm" Sep 10 15:43:24.614405 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:24.614343 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 10 15:43:24.614405 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:24.614400 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 10 15:43:24.614511 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:24.614434 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b1949768-9a66-4404-b7f1-c71ae6520f36-kube-rbac-proxy-cm podName:b1949768-9a66-4404-b7f1-c71ae6520f36 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:32.614418181 +0000 UTC m=+89.589678112 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b1949768-9a66-4404-b7f1-c71ae6520f36-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8wtqm" (UID: "b1949768-9a66-4404-b7f1-c71ae6520f36") : configmap "kube-rbac-proxy" not found Sep 10 15:43:24.614511 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:24.614450 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1949768-9a66-4404-b7f1-c71ae6520f36-insights-runtime-extractor-tls podName:b1949768-9a66-4404-b7f1-c71ae6520f36 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:32.614443712 +0000 UTC m=+89.589703643 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b1949768-9a66-4404-b7f1-c71ae6520f36-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8wtqm" (UID: "b1949768-9a66-4404-b7f1-c71ae6520f36") : secret "insights-runtime-extractor-tls" not found Sep 10 15:43:24.714987 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:24.714962 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b69645e0-943e-4301-982a-9d1532d25e56-registry-tls\") pod \"image-registry-84ff49d5c7-bh94t\" (UID: \"b69645e0-943e-4301-982a-9d1532d25e56\") " pod="openshift-image-registry/image-registry-84ff49d5c7-bh94t" Sep 10 15:43:24.715102 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:24.714995 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6af5593-a983-4c35-95e9-3c166bdbf385-service-ca-bundle\") pod \"router-default-b4f467c4b-xjkv7\" (UID: \"e6af5593-a983-4c35-95e9-3c166bdbf385\") " pod="openshift-ingress/router-default-b4f467c4b-xjkv7" Sep 10 15:43:24.715102 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:24.715030 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6af5593-a983-4c35-95e9-3c166bdbf385-metrics-certs\") pod \"router-default-b4f467c4b-xjkv7\" (UID: \"e6af5593-a983-4c35-95e9-3c166bdbf385\") " pod="openshift-ingress/router-default-b4f467c4b-xjkv7" Sep 10 15:43:24.715174 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:24.715133 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e6af5593-a983-4c35-95e9-3c166bdbf385-service-ca-bundle podName:e6af5593-a983-4c35-95e9-3c166bdbf385 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:40.715119479 +0000 UTC m=+97.690379410 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e6af5593-a983-4c35-95e9-3c166bdbf385-service-ca-bundle") pod "router-default-b4f467c4b-xjkv7" (UID: "e6af5593-a983-4c35-95e9-3c166bdbf385") : configmap references non-existent config key: service-ca.crt Sep 10 15:43:24.717439 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:24.717410 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b69645e0-943e-4301-982a-9d1532d25e56-registry-tls\") pod \"image-registry-84ff49d5c7-bh94t\" (UID: \"b69645e0-943e-4301-982a-9d1532d25e56\") " pod="openshift-image-registry/image-registry-84ff49d5c7-bh94t" Sep 10 15:43:24.717528 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:24.717433 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6af5593-a983-4c35-95e9-3c166bdbf385-metrics-certs\") pod \"router-default-b4f467c4b-xjkv7\" (UID: \"e6af5593-a983-4c35-95e9-3c166bdbf385\") " pod="openshift-ingress/router-default-b4f467c4b-xjkv7" Sep 10 15:43:24.809915 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:24.809886 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84ff49d5c7-bh94t" Sep 10 15:43:24.863286 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:24.862691 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:25.057310 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:25.057279 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:25.085356 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:25.085332 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-84ff49d5c7-bh94t"] Sep 10 15:43:25.088768 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:43:25.088734 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb69645e0_943e_4301_982a_9d1532d25e56.slice/crio-65fde700fcec4175428c1a4b92164731f7f151996d2ade89b34ab1581dbf53ef WatchSource:0}: Error finding container 65fde700fcec4175428c1a4b92164731f7f151996d2ade89b34ab1581dbf53ef: Status 404 returned error can't find the container with id 65fde700fcec4175428c1a4b92164731f7f151996d2ade89b34ab1581dbf53ef Sep 10 15:43:25.408172 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:25.408134 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:25.676065 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:25.675986 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:25.820025 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:25.819994 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84ff49d5c7-bh94t" event={"ID":"b69645e0-943e-4301-982a-9d1532d25e56","Type":"ContainerStarted","Data":"68af216535d6e37ae68e175e0586a236f5699f538f764b7c7178148b5d43504e"} Sep 10 15:43:25.820140 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:25.820041 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84ff49d5c7-bh94t" event={"ID":"b69645e0-943e-4301-982a-9d1532d25e56","Type":"ContainerStarted","Data":"65fde700fcec4175428c1a4b92164731f7f151996d2ade89b34ab1581dbf53ef"} Sep 10 15:43:25.820140 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:25.820130 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-84ff49d5c7-bh94t" Sep 10 15:43:25.841086 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:25.841045 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-84ff49d5c7-bh94t" podStartSLOduration=17.841030588 podStartE2EDuration="17.841030588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:43:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:25.839852518 +0000 UTC m=+82.815112471" watchObservedRunningTime="2026-09-10 15:43:25.841030588 +0000 UTC m=+82.816290542" Sep 10 15:43:26.070606 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:26.070477 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:26.271513 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:26.271482 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:26.492687 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:26.492662 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:26.670504 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:26.670471 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:27.063257 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:27.063219 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:27.667319 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:27.667278 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:28.261339 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:28.261311 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:28.462892 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:28.462862 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:28.816486 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:28.816450 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:28.917245 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:28.917213 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:29.066131 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:29.066104 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:29.459253 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:29.459224 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:29.860708 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:29.860623 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:30.064595 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:30.064565 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:30.198803 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:30.198767 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:30.299123 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:30.299095 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:30.358348 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:30.358319 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:30.465539 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:30.465471 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:30.539116 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:30.539091 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:30.705966 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:30.705939 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:30.880196 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:30.880174 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:31.093512 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:31.093485 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:31.157886 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:31.157821 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:31.685950 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:31.685921 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:32.090983 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:32.090906 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:32.215392 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:32.215351 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:32.348281 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:32.348210 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:32.676142 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:32.676113 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1949768-9a66-4404-b7f1-c71ae6520f36-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8wtqm\" (UID: \"b1949768-9a66-4404-b7f1-c71ae6520f36\") " pod="openshift-insights/insights-runtime-extractor-8wtqm" Sep 10 15:43:32.676362 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:32.676166 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1949768-9a66-4404-b7f1-c71ae6520f36-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8wtqm\" (UID: \"b1949768-9a66-4404-b7f1-c71ae6520f36\") " pod="openshift-insights/insights-runtime-extractor-8wtqm" Sep 10 15:43:32.676663 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:32.676645 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1949768-9a66-4404-b7f1-c71ae6520f36-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8wtqm\" (UID: \"b1949768-9a66-4404-b7f1-c71ae6520f36\") " pod="openshift-insights/insights-runtime-extractor-8wtqm" Sep 10 15:43:32.678343 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:32.678322 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1949768-9a66-4404-b7f1-c71ae6520f36-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8wtqm\" (UID: \"b1949768-9a66-4404-b7f1-c71ae6520f36\") " pod="openshift-insights/insights-runtime-extractor-8wtqm" Sep 10 15:43:32.707308 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:32.707278 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:32.794229 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:32.794201 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:32.837718 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:32.837697 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8wtqm" Sep 10 15:43:33.113194 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:33.113125 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8wtqm"] Sep 10 15:43:33.116303 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:43:33.116275 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1949768_9a66_4404_b7f1_c71ae6520f36.slice/crio-f033a3588fbb222d6d7403fe9e35b19c4b695edb160ca79980ab53bd4bf61d92 WatchSource:0}: Error finding container f033a3588fbb222d6d7403fe9e35b19c4b695edb160ca79980ab53bd4bf61d92: Status 404 returned error can't find the container with id f033a3588fbb222d6d7403fe9e35b19c4b695edb160ca79980ab53bd4bf61d92 Sep 10 15:43:33.849360 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:33.849333 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8wtqm" event={"ID":"b1949768-9a66-4404-b7f1-c71ae6520f36","Type":"ContainerStarted","Data":"f73f12c67a82e74e5c9ff8ee3940bdaffb491f57352f4772fcedcdeeea59184b"} Sep 10 15:43:33.849467 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:33.849423 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8wtqm" event={"ID":"b1949768-9a66-4404-b7f1-c71ae6520f36","Type":"ContainerStarted","Data":"f033a3588fbb222d6d7403fe9e35b19c4b695edb160ca79980ab53bd4bf61d92"} Sep 10 15:43:34.855069 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:34.855033 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8wtqm" event={"ID":"b1949768-9a66-4404-b7f1-c71ae6520f36","Type":"ContainerStarted","Data":"6c2635db111b1542bfc5d62ee29307a211ebf763c18e7fa406abb011195839f9"} Sep 10 15:43:35.861695 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:35.861663 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8wtqm" event={"ID":"b1949768-9a66-4404-b7f1-c71ae6520f36","Type":"ContainerStarted","Data":"da96bf6dca035f96bce72e2622081ab031bd7a6137d2895195701f39f62cd7e5"} Sep 10 15:43:35.878116 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:35.878069 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8wtqm" podStartSLOduration=17.776726441 podStartE2EDuration="19.878054883s" totalImagesPullingTime="2.101328442s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:43:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:35.877475564 +0000 UTC m=+92.852735519" watchObservedRunningTime="2026-09-10 15:43:35.878054883 +0000 UTC m=+92.853314835" Sep 10 15:43:40.734193 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:40.734163 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8681b681-0371-4b9d-8c1d-82a431bd3e62-metrics-tls\") pod \"dns-default-45ph9\" (UID: \"8681b681-0371-4b9d-8c1d-82a431bd3e62\") " pod="openshift-dns/dns-default-45ph9" Sep 10 15:43:40.734590 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:40.734203 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6af5593-a983-4c35-95e9-3c166bdbf385-service-ca-bundle\") pod \"router-default-b4f467c4b-xjkv7\" (UID: \"e6af5593-a983-4c35-95e9-3c166bdbf385\") " pod="openshift-ingress/router-default-b4f467c4b-xjkv7" Sep 10 15:43:40.734745 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:40.734728 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e6af5593-a983-4c35-95e9-3c166bdbf385-service-ca-bundle\") pod \"router-default-b4f467c4b-xjkv7\" (UID: \"e6af5593-a983-4c35-95e9-3c166bdbf385\") " pod="openshift-ingress/router-default-b4f467c4b-xjkv7" Sep 10 15:43:40.736312 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:40.736294 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8681b681-0371-4b9d-8c1d-82a431bd3e62-metrics-tls\") pod \"dns-default-45ph9\" (UID: \"8681b681-0371-4b9d-8c1d-82a431bd3e62\") " pod="openshift-dns/dns-default-45ph9" Sep 10 15:43:40.764573 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:40.764553 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-z4knc\"" Sep 10 15:43:40.772318 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:40.772300 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-45ph9" Sep 10 15:43:40.920991 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:40.920959 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:41.004310 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:41.004238 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-b4f467c4b-xjkv7" Sep 10 15:43:41.022197 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:41.022173 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:41.042300 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:41.042278 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-45ph9"] Sep 10 15:43:41.044948 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:43:41.044924 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8681b681_0371_4b9d_8c1d_82a431bd3e62.slice/crio-1d614570589ba59059b7abcec678f7d01f77b925673d73853da657acae9bceb7 WatchSource:0}: Error finding container 1d614570589ba59059b7abcec678f7d01f77b925673d73853da657acae9bceb7: Status 404 returned error can't find the container with id 1d614570589ba59059b7abcec678f7d01f77b925673d73853da657acae9bceb7 Sep 10 15:43:41.275129 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:41.275054 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-b4f467c4b-xjkv7"] Sep 10 15:43:41.277658 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:43:41.277624 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6af5593_a983_4c35_95e9_3c166bdbf385.slice/crio-432f02efe057461d12fbd9ee62fae8fd396cade7f6c6755960dc12fd005f838e WatchSource:0}: Error finding container 432f02efe057461d12fbd9ee62fae8fd396cade7f6c6755960dc12fd005f838e: Status 404 returned error can't find the container with id 432f02efe057461d12fbd9ee62fae8fd396cade7f6c6755960dc12fd005f838e Sep 10 15:43:41.880832 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:41.880791 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-45ph9" event={"ID":"8681b681-0371-4b9d-8c1d-82a431bd3e62","Type":"ContainerStarted","Data":"1d614570589ba59059b7abcec678f7d01f77b925673d73853da657acae9bceb7"} Sep 10 15:43:41.882567 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:41.882537 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-b4f467c4b-xjkv7" event={"ID":"e6af5593-a983-4c35-95e9-3c166bdbf385","Type":"ContainerStarted","Data":"66df7006869a2015866ca358bfd738dfc3779ab0ad4b39a10e28885d081b37b7"} Sep 10 15:43:41.882692 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:41.882586 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-b4f467c4b-xjkv7" event={"ID":"e6af5593-a983-4c35-95e9-3c166bdbf385","Type":"ContainerStarted","Data":"432f02efe057461d12fbd9ee62fae8fd396cade7f6c6755960dc12fd005f838e"} Sep 10 15:43:41.902494 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:41.902451 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-b4f467c4b-xjkv7" podStartSLOduration=33.90243722 podStartE2EDuration="33.90243722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:43:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:41.901279936 +0000 UTC m=+98.876539892" watchObservedRunningTime="2026-09-10 15:43:41.90243722 +0000 UTC m=+98.877697172" Sep 10 15:43:42.005444 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:42.005410 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-b4f467c4b-xjkv7" Sep 10 15:43:42.008245 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:42.008222 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-b4f467c4b-xjkv7" Sep 10 15:43:42.483575 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:42.483543 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84ff49d5c7-bh94t"] Sep 10 15:43:42.488885 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:42.488852 2583 patch_prober.go:28] interesting pod/image-registry-84ff49d5c7-bh94t container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 10 15:43:42.489085 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:42.488917 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-84ff49d5c7-bh94t" podUID="b69645e0-943e-4301-982a-9d1532d25e56" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 15:43:42.560984 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:42.560902 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:42.639762 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:42.639737 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:42.739700 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:42.739638 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:42.888216 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:42.888180 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-45ph9" event={"ID":"8681b681-0371-4b9d-8c1d-82a431bd3e62","Type":"ContainerStarted","Data":"0f9be83996f2300bb1f5bbb62c5643edd03bebb01135b22e72ee8e6a13252ea9"} Sep 10 15:43:42.888582 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:42.888241 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-45ph9" event={"ID":"8681b681-0371-4b9d-8c1d-82a431bd3e62","Type":"ContainerStarted","Data":"4eee4338e8f46360b18a998b6db6fb04068cbfa702a1e7db95c1201615cf7a91"} Sep 10 15:43:42.888582 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:42.888500 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-b4f467c4b-xjkv7" Sep 10 15:43:42.889657 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:42.889637 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-b4f467c4b-xjkv7" Sep 10 15:43:42.905682 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:42.905644 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-45ph9" podStartSLOduration=65.315473944 podStartE2EDuration="1m6.905631419s" totalImagesPullingTime="1.590157475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:42:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:42.904552804 +0000 UTC m=+99.879812771" watchObservedRunningTime="2026-09-10 15:43:42.905631419 +0000 UTC m=+99.880891434" Sep 10 15:43:43.673858 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:43.673816 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:43.774294 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:43.774269 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:43.854893 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:43.854858 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:43.891648 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:43.891619 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-45ph9" Sep 10 15:43:44.116673 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:44.116587 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:44.217399 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:44.217353 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:44.772701 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:44.772673 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:45.709195 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:45.709162 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:45.769895 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:45.769868 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:45.783754 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:45.783729 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wqgsp" Sep 10 15:43:49.708055 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:49.708026 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:49.766859 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:49.766836 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:49.869295 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:49.869269 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:49.908309 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:49.908278 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-b8ntr"] Sep 10 15:43:49.911672 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:49.911655 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-b8ntr" Sep 10 15:43:49.913914 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:49.913894 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 10 15:43:49.914046 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:49.914024 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 10 15:43:49.914907 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:49.914862 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 10 15:43:49.914971 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:49.914867 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 10 15:43:49.915034 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:49.914880 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-kcdfk\"" Sep 10 15:43:49.915034 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:49.914916 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 10 15:43:49.920479 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:49.920451 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-b8ntr"] Sep 10 15:43:49.946400 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:49.946358 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:49.994875 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:49.994822 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1fe37db8-3301-406d-b0d8-241dec30c632-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-b8ntr\" (UID: \"1fe37db8-3301-406d-b0d8-241dec30c632\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-b8ntr" Sep 10 15:43:49.994875 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:49.994850 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1fe37db8-3301-406d-b0d8-241dec30c632-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-b8ntr\" (UID: \"1fe37db8-3301-406d-b0d8-241dec30c632\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-b8ntr" Sep 10 15:43:49.995021 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:49.994876 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1fe37db8-3301-406d-b0d8-241dec30c632-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-b8ntr\" (UID: \"1fe37db8-3301-406d-b0d8-241dec30c632\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-b8ntr" Sep 10 15:43:49.995021 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:49.994994 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2q8s\" (UniqueName: \"kubernetes.io/projected/1fe37db8-3301-406d-b0d8-241dec30c632-kube-api-access-z2q8s\") pod \"prometheus-operator-6f4946db56-b8ntr\" (UID: \"1fe37db8-3301-406d-b0d8-241dec30c632\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-b8ntr" Sep 10 15:43:50.077204 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:50.077182 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:50.096233 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:50.096209 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1fe37db8-3301-406d-b0d8-241dec30c632-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-b8ntr\" (UID: \"1fe37db8-3301-406d-b0d8-241dec30c632\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-b8ntr" Sep 10 15:43:50.096297 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:50.096238 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1fe37db8-3301-406d-b0d8-241dec30c632-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-b8ntr\" (UID: \"1fe37db8-3301-406d-b0d8-241dec30c632\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-b8ntr" Sep 10 15:43:50.096297 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:50.096261 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1fe37db8-3301-406d-b0d8-241dec30c632-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-b8ntr\" (UID: \"1fe37db8-3301-406d-b0d8-241dec30c632\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-b8ntr" Sep 10 15:43:50.096368 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:50.096302 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2q8s\" (UniqueName: \"kubernetes.io/projected/1fe37db8-3301-406d-b0d8-241dec30c632-kube-api-access-z2q8s\") pod \"prometheus-operator-6f4946db56-b8ntr\" (UID: \"1fe37db8-3301-406d-b0d8-241dec30c632\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-b8ntr" Sep 10 15:43:50.096833 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:50.096812 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1fe37db8-3301-406d-b0d8-241dec30c632-metrics-client-ca\") pod \"prometheus-operator-6f4946db56-b8ntr\" (UID: \"1fe37db8-3301-406d-b0d8-241dec30c632\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-b8ntr" Sep 10 15:43:50.098616 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:50.098598 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1fe37db8-3301-406d-b0d8-241dec30c632-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-6f4946db56-b8ntr\" (UID: \"1fe37db8-3301-406d-b0d8-241dec30c632\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-b8ntr" Sep 10 15:43:50.098698 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:50.098668 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/1fe37db8-3301-406d-b0d8-241dec30c632-prometheus-operator-tls\") pod \"prometheus-operator-6f4946db56-b8ntr\" (UID: \"1fe37db8-3301-406d-b0d8-241dec30c632\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-b8ntr" Sep 10 15:43:50.104711 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:50.104687 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2q8s\" (UniqueName: \"kubernetes.io/projected/1fe37db8-3301-406d-b0d8-241dec30c632-kube-api-access-z2q8s\") pod \"prometheus-operator-6f4946db56-b8ntr\" (UID: \"1fe37db8-3301-406d-b0d8-241dec30c632\") " pod="openshift-monitoring/prometheus-operator-6f4946db56-b8ntr" Sep 10 15:43:50.220973 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:50.220955 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-6f4946db56-b8ntr" Sep 10 15:43:50.486718 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:50.486688 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-6f4946db56-b8ntr"] Sep 10 15:43:50.489711 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:43:50.489684 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fe37db8_3301_406d_b0d8_241dec30c632.slice/crio-776da739e9b88548c35d9d3e930247bdd9ecc6669cb33bf5cb16892f02cf9287 WatchSource:0}: Error finding container 776da739e9b88548c35d9d3e930247bdd9ecc6669cb33bf5cb16892f02cf9287: Status 404 returned error can't find the container with id 776da739e9b88548c35d9d3e930247bdd9ecc6669cb33bf5cb16892f02cf9287 Sep 10 15:43:50.914765 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:50.914728 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-b8ntr" event={"ID":"1fe37db8-3301-406d-b0d8-241dec30c632","Type":"ContainerStarted","Data":"776da739e9b88548c35d9d3e930247bdd9ecc6669cb33bf5cb16892f02cf9287"} Sep 10 15:43:51.919869 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:51.919828 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-b8ntr" event={"ID":"1fe37db8-3301-406d-b0d8-241dec30c632","Type":"ContainerStarted","Data":"56bfe696b255e19da2bed2a2b13c17a0b0cbd0ef77a6aca8d52b8fe8a406fe18"} Sep 10 15:43:51.920338 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:51.919892 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-6f4946db56-b8ntr" event={"ID":"1fe37db8-3301-406d-b0d8-241dec30c632","Type":"ContainerStarted","Data":"9a33ded9d8becc4055e530f1fd720bf9e00b2c33272141a21af6bbef01484624"} Sep 10 15:43:51.937272 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:51.937184 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-6f4946db56-b8ntr" podStartSLOduration=1.744617973 podStartE2EDuration="2.93716983s" totalImagesPullingTime="1.192551857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:43:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:51.935737936 +0000 UTC m=+108.910997888" watchObservedRunningTime="2026-09-10 15:43:51.93716983 +0000 UTC m=+108.912429783" Sep 10 15:43:51.941209 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:51.941186 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:52.114366 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:52.114336 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:52.182922 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:52.182903 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:52.487993 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:52.487967 2583 patch_prober.go:28] interesting pod/image-registry-84ff49d5c7-bh94t container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 10 15:43:52.488097 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:52.488011 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-84ff49d5c7-bh94t" podUID="b69645e0-943e-4301-982a-9d1532d25e56" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 15:43:53.897535 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:53.897506 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-45ph9" Sep 10 15:43:54.078469 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.078436 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:54.205412 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.205325 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:54.315733 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.315698 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:54.315973 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.315956 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-kzbh5"] Sep 10 15:43:54.318068 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.318051 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-kzbh5" Sep 10 15:43:54.320068 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.320044 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 10 15:43:54.320190 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.320111 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 10 15:43:54.320348 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.320333 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-69q6r\"" Sep 10 15:43:54.328768 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.328747 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-kzbh5"] Sep 10 15:43:54.345695 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.345674 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8hcsc"] Sep 10 15:43:54.348087 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.348071 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8hcsc" Sep 10 15:43:54.350041 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.350021 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 10 15:43:54.350370 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.350354 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 10 15:43:54.350504 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.350439 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-67ndx\"" Sep 10 15:43:54.350567 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.350452 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 10 15:43:54.398983 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.398962 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:54.425990 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.425970 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8b0d03d5-0328-428f-94cf-8f57dba68534-node-exporter-tls\") pod \"node-exporter-8hcsc\" (UID: \"8b0d03d5-0328-428f-94cf-8f57dba68534\") " pod="openshift-monitoring/node-exporter-8hcsc" Sep 10 15:43:54.426097 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.426001 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vcm2\" (UniqueName: \"kubernetes.io/projected/29dc562d-f7be-4903-bb3a-23ee7b84d3cf-kube-api-access-6vcm2\") pod \"openshift-state-metrics-657fd6c54-kzbh5\" (UID: \"29dc562d-f7be-4903-bb3a-23ee7b84d3cf\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-kzbh5" Sep 10 15:43:54.426097 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.426025 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8b0d03d5-0328-428f-94cf-8f57dba68534-root\") pod \"node-exporter-8hcsc\" (UID: \"8b0d03d5-0328-428f-94cf-8f57dba68534\") " pod="openshift-monitoring/node-exporter-8hcsc" Sep 10 15:43:54.426097 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.426050 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8b0d03d5-0328-428f-94cf-8f57dba68534-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8hcsc\" (UID: \"8b0d03d5-0328-428f-94cf-8f57dba68534\") " pod="openshift-monitoring/node-exporter-8hcsc" Sep 10 15:43:54.426252 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.426110 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/29dc562d-f7be-4903-bb3a-23ee7b84d3cf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-kzbh5\" (UID: \"29dc562d-f7be-4903-bb3a-23ee7b84d3cf\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-kzbh5" Sep 10 15:43:54.426252 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.426154 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/29dc562d-f7be-4903-bb3a-23ee7b84d3cf-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-kzbh5\" (UID: \"29dc562d-f7be-4903-bb3a-23ee7b84d3cf\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-kzbh5" Sep 10 15:43:54.426252 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.426234 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b0d03d5-0328-428f-94cf-8f57dba68534-metrics-client-ca\") pod \"node-exporter-8hcsc\" (UID: \"8b0d03d5-0328-428f-94cf-8f57dba68534\") " pod="openshift-monitoring/node-exporter-8hcsc" Sep 10 15:43:54.426410 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.426261 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29dc562d-f7be-4903-bb3a-23ee7b84d3cf-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-kzbh5\" (UID: \"29dc562d-f7be-4903-bb3a-23ee7b84d3cf\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-kzbh5" Sep 10 15:43:54.426410 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.426291 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpqgt\" (UniqueName: \"kubernetes.io/projected/8b0d03d5-0328-428f-94cf-8f57dba68534-kube-api-access-dpqgt\") pod \"node-exporter-8hcsc\" (UID: \"8b0d03d5-0328-428f-94cf-8f57dba68534\") " pod="openshift-monitoring/node-exporter-8hcsc" Sep 10 15:43:54.426410 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.426314 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8b0d03d5-0328-428f-94cf-8f57dba68534-node-exporter-wtmp\") pod \"node-exporter-8hcsc\" (UID: \"8b0d03d5-0328-428f-94cf-8f57dba68534\") " pod="openshift-monitoring/node-exporter-8hcsc" Sep 10 15:43:54.426410 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.426329 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b0d03d5-0328-428f-94cf-8f57dba68534-sys\") pod \"node-exporter-8hcsc\" (UID: \"8b0d03d5-0328-428f-94cf-8f57dba68534\") " pod="openshift-monitoring/node-exporter-8hcsc" Sep 10 15:43:54.426410 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.426348 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8b0d03d5-0328-428f-94cf-8f57dba68534-node-exporter-textfile\") pod \"node-exporter-8hcsc\" (UID: \"8b0d03d5-0328-428f-94cf-8f57dba68534\") " pod="openshift-monitoring/node-exporter-8hcsc" Sep 10 15:43:54.426410 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.426393 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8b0d03d5-0328-428f-94cf-8f57dba68534-node-exporter-accelerators-collector-config\") pod \"node-exporter-8hcsc\" (UID: \"8b0d03d5-0328-428f-94cf-8f57dba68534\") " pod="openshift-monitoring/node-exporter-8hcsc" Sep 10 15:43:54.526831 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.526775 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8b0d03d5-0328-428f-94cf-8f57dba68534-root\") pod \"node-exporter-8hcsc\" (UID: \"8b0d03d5-0328-428f-94cf-8f57dba68534\") " pod="openshift-monitoring/node-exporter-8hcsc" Sep 10 15:43:54.526831 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.526802 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8b0d03d5-0328-428f-94cf-8f57dba68534-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8hcsc\" (UID: \"8b0d03d5-0328-428f-94cf-8f57dba68534\") " pod="openshift-monitoring/node-exporter-8hcsc" Sep 10 15:43:54.526981 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.526849 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/29dc562d-f7be-4903-bb3a-23ee7b84d3cf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-kzbh5\" (UID: \"29dc562d-f7be-4903-bb3a-23ee7b84d3cf\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-kzbh5" Sep 10 15:43:54.526981 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.526869 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/29dc562d-f7be-4903-bb3a-23ee7b84d3cf-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-kzbh5\" (UID: \"29dc562d-f7be-4903-bb3a-23ee7b84d3cf\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-kzbh5" Sep 10 15:43:54.526981 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.526887 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8b0d03d5-0328-428f-94cf-8f57dba68534-root\") pod \"node-exporter-8hcsc\" (UID: \"8b0d03d5-0328-428f-94cf-8f57dba68534\") " pod="openshift-monitoring/node-exporter-8hcsc" Sep 10 15:43:54.526981 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.526901 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b0d03d5-0328-428f-94cf-8f57dba68534-metrics-client-ca\") pod \"node-exporter-8hcsc\" (UID: \"8b0d03d5-0328-428f-94cf-8f57dba68534\") " pod="openshift-monitoring/node-exporter-8hcsc" Sep 10 15:43:54.526981 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:54.526955 2583 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Sep 10 15:43:54.527233 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.526951 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29dc562d-f7be-4903-bb3a-23ee7b84d3cf-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-kzbh5\" (UID: \"29dc562d-f7be-4903-bb3a-23ee7b84d3cf\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-kzbh5" Sep 10 15:43:54.527233 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:54.527058 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/29dc562d-f7be-4903-bb3a-23ee7b84d3cf-openshift-state-metrics-tls podName:29dc562d-f7be-4903-bb3a-23ee7b84d3cf nodeName:}" failed. No retries permitted until 2026-09-10 15:43:55.027011308 +0000 UTC m=+112.002271249 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/29dc562d-f7be-4903-bb3a-23ee7b84d3cf-openshift-state-metrics-tls") pod "openshift-state-metrics-657fd6c54-kzbh5" (UID: "29dc562d-f7be-4903-bb3a-23ee7b84d3cf") : secret "openshift-state-metrics-tls" not found Sep 10 15:43:54.527233 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.527092 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpqgt\" (UniqueName: \"kubernetes.io/projected/8b0d03d5-0328-428f-94cf-8f57dba68534-kube-api-access-dpqgt\") pod \"node-exporter-8hcsc\" (UID: \"8b0d03d5-0328-428f-94cf-8f57dba68534\") " pod="openshift-monitoring/node-exporter-8hcsc" Sep 10 15:43:54.527233 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.527147 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8b0d03d5-0328-428f-94cf-8f57dba68534-node-exporter-wtmp\") pod \"node-exporter-8hcsc\" (UID: \"8b0d03d5-0328-428f-94cf-8f57dba68534\") " pod="openshift-monitoring/node-exporter-8hcsc" Sep 10 15:43:54.527465 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.527316 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8b0d03d5-0328-428f-94cf-8f57dba68534-node-exporter-wtmp\") pod \"node-exporter-8hcsc\" (UID: \"8b0d03d5-0328-428f-94cf-8f57dba68534\") " pod="openshift-monitoring/node-exporter-8hcsc" Sep 10 15:43:54.527465 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.527341 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b0d03d5-0328-428f-94cf-8f57dba68534-sys\") pod \"node-exporter-8hcsc\" (UID: \"8b0d03d5-0328-428f-94cf-8f57dba68534\") " pod="openshift-monitoring/node-exporter-8hcsc" Sep 10 15:43:54.527465 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.527405 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8b0d03d5-0328-428f-94cf-8f57dba68534-node-exporter-textfile\") pod \"node-exporter-8hcsc\" (UID: \"8b0d03d5-0328-428f-94cf-8f57dba68534\") " pod="openshift-monitoring/node-exporter-8hcsc" Sep 10 15:43:54.527465 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.527434 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8b0d03d5-0328-428f-94cf-8f57dba68534-node-exporter-accelerators-collector-config\") pod \"node-exporter-8hcsc\" (UID: \"8b0d03d5-0328-428f-94cf-8f57dba68534\") " pod="openshift-monitoring/node-exporter-8hcsc" Sep 10 15:43:54.527465 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.527439 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b0d03d5-0328-428f-94cf-8f57dba68534-sys\") pod \"node-exporter-8hcsc\" (UID: \"8b0d03d5-0328-428f-94cf-8f57dba68534\") " pod="openshift-monitoring/node-exporter-8hcsc" Sep 10 15:43:54.527705 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.527482 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8b0d03d5-0328-428f-94cf-8f57dba68534-node-exporter-tls\") pod \"node-exporter-8hcsc\" (UID: \"8b0d03d5-0328-428f-94cf-8f57dba68534\") " pod="openshift-monitoring/node-exporter-8hcsc" Sep 10 15:43:54.527705 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.527679 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vcm2\" (UniqueName: \"kubernetes.io/projected/29dc562d-f7be-4903-bb3a-23ee7b84d3cf-kube-api-access-6vcm2\") pod \"openshift-state-metrics-657fd6c54-kzbh5\" (UID: \"29dc562d-f7be-4903-bb3a-23ee7b84d3cf\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-kzbh5" Sep 10 15:43:54.527801 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.527707 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29dc562d-f7be-4903-bb3a-23ee7b84d3cf-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-kzbh5\" (UID: \"29dc562d-f7be-4903-bb3a-23ee7b84d3cf\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-kzbh5" Sep 10 15:43:54.528141 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.528097 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8b0d03d5-0328-428f-94cf-8f57dba68534-node-exporter-textfile\") pod \"node-exporter-8hcsc\" (UID: \"8b0d03d5-0328-428f-94cf-8f57dba68534\") " pod="openshift-monitoring/node-exporter-8hcsc" Sep 10 15:43:54.528302 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.528272 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8b0d03d5-0328-428f-94cf-8f57dba68534-node-exporter-accelerators-collector-config\") pod \"node-exporter-8hcsc\" (UID: \"8b0d03d5-0328-428f-94cf-8f57dba68534\") " pod="openshift-monitoring/node-exporter-8hcsc" Sep 10 15:43:54.528302 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.528291 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b0d03d5-0328-428f-94cf-8f57dba68534-metrics-client-ca\") pod \"node-exporter-8hcsc\" (UID: \"8b0d03d5-0328-428f-94cf-8f57dba68534\") " pod="openshift-monitoring/node-exporter-8hcsc" Sep 10 15:43:54.529609 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.529584 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8b0d03d5-0328-428f-94cf-8f57dba68534-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8hcsc\" (UID: \"8b0d03d5-0328-428f-94cf-8f57dba68534\") " pod="openshift-monitoring/node-exporter-8hcsc" Sep 10 15:43:54.529808 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.529785 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/29dc562d-f7be-4903-bb3a-23ee7b84d3cf-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-kzbh5\" (UID: \"29dc562d-f7be-4903-bb3a-23ee7b84d3cf\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-kzbh5" Sep 10 15:43:54.530056 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.530037 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8b0d03d5-0328-428f-94cf-8f57dba68534-node-exporter-tls\") pod \"node-exporter-8hcsc\" (UID: \"8b0d03d5-0328-428f-94cf-8f57dba68534\") " pod="openshift-monitoring/node-exporter-8hcsc" Sep 10 15:43:54.538939 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.538364 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpqgt\" (UniqueName: \"kubernetes.io/projected/8b0d03d5-0328-428f-94cf-8f57dba68534-kube-api-access-dpqgt\") pod \"node-exporter-8hcsc\" (UID: \"8b0d03d5-0328-428f-94cf-8f57dba68534\") " pod="openshift-monitoring/node-exporter-8hcsc" Sep 10 15:43:54.538939 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.538881 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vcm2\" (UniqueName: \"kubernetes.io/projected/29dc562d-f7be-4903-bb3a-23ee7b84d3cf-kube-api-access-6vcm2\") pod \"openshift-state-metrics-657fd6c54-kzbh5\" (UID: \"29dc562d-f7be-4903-bb3a-23ee7b84d3cf\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-kzbh5" Sep 10 15:43:54.540989 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.540964 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:54.638098 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.638071 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:54.656964 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.656938 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8hcsc" Sep 10 15:43:54.668679 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:43:54.668658 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b0d03d5_0328_428f_94cf_8f57dba68534.slice/crio-e84aaa3be095ce345b7d91b549c097aa3c75a1f93e5d083264e07512d0572f9b WatchSource:0}: Error finding container e84aaa3be095ce345b7d91b549c097aa3c75a1f93e5d083264e07512d0572f9b: Status 404 returned error can't find the container with id e84aaa3be095ce345b7d91b549c097aa3c75a1f93e5d083264e07512d0572f9b Sep 10 15:43:54.741865 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.741842 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:54.920651 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.920603 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:54.932439 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:54.932415 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8hcsc" event={"ID":"8b0d03d5-0328-428f-94cf-8f57dba68534","Type":"ContainerStarted","Data":"e84aaa3be095ce345b7d91b549c097aa3c75a1f93e5d083264e07512d0572f9b"} Sep 10 15:43:55.020473 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.020450 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:55.031622 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.031595 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/29dc562d-f7be-4903-bb3a-23ee7b84d3cf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-kzbh5\" (UID: \"29dc562d-f7be-4903-bb3a-23ee7b84d3cf\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-kzbh5" Sep 10 15:43:55.033836 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.033816 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/29dc562d-f7be-4903-bb3a-23ee7b84d3cf-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-kzbh5\" (UID: \"29dc562d-f7be-4903-bb3a-23ee7b84d3cf\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-kzbh5" Sep 10 15:43:55.229646 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.229580 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-kzbh5" Sep 10 15:43:55.371772 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.371733 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:55.511260 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:43:55.511232 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29dc562d_f7be_4903_bb3a_23ee7b84d3cf.slice/crio-5de35107cebee6126ea9e6045f1df1d00b383a8838dbadf53be430ef1307ad83 WatchSource:0}: Error finding container 5de35107cebee6126ea9e6045f1df1d00b383a8838dbadf53be430ef1307ad83: Status 404 returned error can't find the container with id 5de35107cebee6126ea9e6045f1df1d00b383a8838dbadf53be430ef1307ad83 Sep 10 15:43:55.517967 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.517612 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 15:43:55.522543 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.522519 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-kzbh5"] Sep 10 15:43:55.522681 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.522659 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:55.524835 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.524818 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 10 15:43:55.524899 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.524855 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 10 15:43:55.525072 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.525054 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 10 15:43:55.525233 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.525218 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 10 15:43:55.525293 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.525231 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 10 15:43:55.525569 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.525550 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 10 15:43:55.525771 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.525747 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 10 15:43:55.525873 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.525798 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-nhdfm\"" Sep 10 15:43:55.525942 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.525752 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 10 15:43:55.526044 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.526024 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 10 15:43:55.540784 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.539514 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:55.540784 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.540704 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 15:43:55.635143 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.635100 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/590890d6-a6d7-4868-b14e-f02eb06f758a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"590890d6-a6d7-4868-b14e-f02eb06f758a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:55.635302 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.635159 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/590890d6-a6d7-4868-b14e-f02eb06f758a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"590890d6-a6d7-4868-b14e-f02eb06f758a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:55.635302 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.635178 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/590890d6-a6d7-4868-b14e-f02eb06f758a-web-config\") pod \"alertmanager-main-0\" (UID: \"590890d6-a6d7-4868-b14e-f02eb06f758a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:55.635302 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.635244 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/590890d6-a6d7-4868-b14e-f02eb06f758a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"590890d6-a6d7-4868-b14e-f02eb06f758a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:55.635302 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.635287 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5shzk\" (UniqueName: \"kubernetes.io/projected/590890d6-a6d7-4868-b14e-f02eb06f758a-kube-api-access-5shzk\") pod \"alertmanager-main-0\" (UID: \"590890d6-a6d7-4868-b14e-f02eb06f758a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:55.635508 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.635308 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/590890d6-a6d7-4868-b14e-f02eb06f758a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"590890d6-a6d7-4868-b14e-f02eb06f758a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:55.635508 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.635337 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/590890d6-a6d7-4868-b14e-f02eb06f758a-config-volume\") pod \"alertmanager-main-0\" (UID: \"590890d6-a6d7-4868-b14e-f02eb06f758a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:55.635508 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.635352 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/590890d6-a6d7-4868-b14e-f02eb06f758a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"590890d6-a6d7-4868-b14e-f02eb06f758a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:55.635508 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.635440 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/590890d6-a6d7-4868-b14e-f02eb06f758a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"590890d6-a6d7-4868-b14e-f02eb06f758a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:55.635508 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.635467 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/590890d6-a6d7-4868-b14e-f02eb06f758a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"590890d6-a6d7-4868-b14e-f02eb06f758a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:55.635715 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.635542 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/590890d6-a6d7-4868-b14e-f02eb06f758a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"590890d6-a6d7-4868-b14e-f02eb06f758a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:55.635715 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.635587 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/590890d6-a6d7-4868-b14e-f02eb06f758a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"590890d6-a6d7-4868-b14e-f02eb06f758a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:55.635715 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.635632 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/590890d6-a6d7-4868-b14e-f02eb06f758a-config-out\") pod \"alertmanager-main-0\" (UID: \"590890d6-a6d7-4868-b14e-f02eb06f758a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:55.664451 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.664422 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:55.736762 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.736726 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/590890d6-a6d7-4868-b14e-f02eb06f758a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"590890d6-a6d7-4868-b14e-f02eb06f758a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:55.736762 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.736762 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/590890d6-a6d7-4868-b14e-f02eb06f758a-web-config\") pod \"alertmanager-main-0\" (UID: \"590890d6-a6d7-4868-b14e-f02eb06f758a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:55.736980 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.736785 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/590890d6-a6d7-4868-b14e-f02eb06f758a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"590890d6-a6d7-4868-b14e-f02eb06f758a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:55.736980 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.736804 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5shzk\" (UniqueName: \"kubernetes.io/projected/590890d6-a6d7-4868-b14e-f02eb06f758a-kube-api-access-5shzk\") pod \"alertmanager-main-0\" (UID: \"590890d6-a6d7-4868-b14e-f02eb06f758a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:55.736980 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.736874 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/590890d6-a6d7-4868-b14e-f02eb06f758a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"590890d6-a6d7-4868-b14e-f02eb06f758a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:55.736980 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.736934 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/590890d6-a6d7-4868-b14e-f02eb06f758a-config-volume\") pod \"alertmanager-main-0\" (UID: \"590890d6-a6d7-4868-b14e-f02eb06f758a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:55.736980 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.736964 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/590890d6-a6d7-4868-b14e-f02eb06f758a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"590890d6-a6d7-4868-b14e-f02eb06f758a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:55.737215 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.737019 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/590890d6-a6d7-4868-b14e-f02eb06f758a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"590890d6-a6d7-4868-b14e-f02eb06f758a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:55.737215 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.737055 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/590890d6-a6d7-4868-b14e-f02eb06f758a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"590890d6-a6d7-4868-b14e-f02eb06f758a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:55.737215 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.737086 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/590890d6-a6d7-4868-b14e-f02eb06f758a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"590890d6-a6d7-4868-b14e-f02eb06f758a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:55.737215 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.737113 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/590890d6-a6d7-4868-b14e-f02eb06f758a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"590890d6-a6d7-4868-b14e-f02eb06f758a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:55.737215 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.737148 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/590890d6-a6d7-4868-b14e-f02eb06f758a-config-out\") pod \"alertmanager-main-0\" (UID: \"590890d6-a6d7-4868-b14e-f02eb06f758a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:55.737215 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.737197 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/590890d6-a6d7-4868-b14e-f02eb06f758a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"590890d6-a6d7-4868-b14e-f02eb06f758a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:55.737507 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.737428 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/590890d6-a6d7-4868-b14e-f02eb06f758a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"590890d6-a6d7-4868-b14e-f02eb06f758a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:55.738318 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.738291 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/590890d6-a6d7-4868-b14e-f02eb06f758a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"590890d6-a6d7-4868-b14e-f02eb06f758a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:55.739349 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.739320 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/590890d6-a6d7-4868-b14e-f02eb06f758a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"590890d6-a6d7-4868-b14e-f02eb06f758a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:55.740045 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.740019 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/590890d6-a6d7-4868-b14e-f02eb06f758a-web-config\") pod \"alertmanager-main-0\" (UID: \"590890d6-a6d7-4868-b14e-f02eb06f758a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:55.740251 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.740225 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/590890d6-a6d7-4868-b14e-f02eb06f758a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"590890d6-a6d7-4868-b14e-f02eb06f758a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:55.741546 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.741257 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/590890d6-a6d7-4868-b14e-f02eb06f758a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"590890d6-a6d7-4868-b14e-f02eb06f758a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:55.741546 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.741332 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/590890d6-a6d7-4868-b14e-f02eb06f758a-config-out\") pod \"alertmanager-main-0\" (UID: \"590890d6-a6d7-4868-b14e-f02eb06f758a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:55.741546 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.741438 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/590890d6-a6d7-4868-b14e-f02eb06f758a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"590890d6-a6d7-4868-b14e-f02eb06f758a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:55.741546 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.741512 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/590890d6-a6d7-4868-b14e-f02eb06f758a-config-volume\") pod \"alertmanager-main-0\" (UID: \"590890d6-a6d7-4868-b14e-f02eb06f758a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:55.742346 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.742320 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/590890d6-a6d7-4868-b14e-f02eb06f758a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"590890d6-a6d7-4868-b14e-f02eb06f758a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:55.742749 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.742726 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/590890d6-a6d7-4868-b14e-f02eb06f758a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"590890d6-a6d7-4868-b14e-f02eb06f758a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:55.742920 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.742898 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/590890d6-a6d7-4868-b14e-f02eb06f758a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"590890d6-a6d7-4868-b14e-f02eb06f758a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:55.745074 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.745052 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5shzk\" (UniqueName: \"kubernetes.io/projected/590890d6-a6d7-4868-b14e-f02eb06f758a-kube-api-access-5shzk\") pod \"alertmanager-main-0\" (UID: \"590890d6-a6d7-4868-b14e-f02eb06f758a\") " pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:55.865631 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.865604 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 10 15:43:55.937946 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.937914 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8hcsc" event={"ID":"8b0d03d5-0328-428f-94cf-8f57dba68534","Type":"ContainerStarted","Data":"b8f2322ace3948ea87d7928e311969a57cc845313deabd1766424e9e4d7b349b"} Sep 10 15:43:55.940059 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.940029 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-kzbh5" event={"ID":"29dc562d-f7be-4903-bb3a-23ee7b84d3cf","Type":"ContainerStarted","Data":"3409184ce651d9b58f2cde2a432d78e357d7d87a18e8ac0edd9145a57d12e90f"} Sep 10 15:43:55.940146 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.940091 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-kzbh5" event={"ID":"29dc562d-f7be-4903-bb3a-23ee7b84d3cf","Type":"ContainerStarted","Data":"64f83b2cc51625bb9930988df0329f18ad80bbf69d5d584e7531e3bd177d8d9f"} Sep 10 15:43:55.940146 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:55.940122 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-kzbh5" event={"ID":"29dc562d-f7be-4903-bb3a-23ee7b84d3cf","Type":"ContainerStarted","Data":"5de35107cebee6126ea9e6045f1df1d00b383a8838dbadf53be430ef1307ad83"} Sep 10 15:43:56.144322 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:56.144301 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 10 15:43:56.146696 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:43:56.146672 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod590890d6_a6d7_4868_b14e_f02eb06f758a.slice/crio-48f468619fe1b45a1bcc2ebd28752bb3ebf1d9f774f8949c855263ad9624b902 WatchSource:0}: Error finding container 48f468619fe1b45a1bcc2ebd28752bb3ebf1d9f774f8949c855263ad9624b902: Status 404 returned error can't find the container with id 48f468619fe1b45a1bcc2ebd28752bb3ebf1d9f774f8949c855263ad9624b902 Sep 10 15:43:56.945995 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:56.945961 2583 generic.go:356] "Generic (PLEG): container finished" podID="8b0d03d5-0328-428f-94cf-8f57dba68534" containerID="b8f2322ace3948ea87d7928e311969a57cc845313deabd1766424e9e4d7b349b" exitCode=0 Sep 10 15:43:56.946463 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:56.946055 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8hcsc" event={"ID":"8b0d03d5-0328-428f-94cf-8f57dba68534","Type":"ContainerDied","Data":"b8f2322ace3948ea87d7928e311969a57cc845313deabd1766424e9e4d7b349b"} Sep 10 15:43:56.948893 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:56.948871 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-kzbh5" event={"ID":"29dc562d-f7be-4903-bb3a-23ee7b84d3cf","Type":"ContainerStarted","Data":"6c845b9c72d35e733ddc07cffd5c8f2d352f2e596e368e93b0f41473d4a13bca"} Sep 10 15:43:56.950434 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:56.950412 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"590890d6-a6d7-4868-b14e-f02eb06f758a","Type":"ContainerStarted","Data":"48f468619fe1b45a1bcc2ebd28752bb3ebf1d9f774f8949c855263ad9624b902"} Sep 10 15:43:56.983493 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:56.983438 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-kzbh5" podStartSLOduration=2.049611585 podStartE2EDuration="2.983419714s" totalImagesPullingTime="933.808129ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:43:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:56.982091355 +0000 UTC m=+113.957351309" watchObservedRunningTime="2026-09-10 15:43:56.983419714 +0000 UTC m=+113.958679671" Sep 10 15:43:57.303649 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:57.303578 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:57.403767 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:57.403737 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:57.459525 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:57.459494 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5f68bcbf87-57tzb"] Sep 10 15:43:57.462334 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:57.462319 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5f68bcbf87-57tzb" Sep 10 15:43:57.464627 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:57.464603 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 10 15:43:57.464753 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:57.464604 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 10 15:43:57.464906 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:57.464878 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 10 15:43:57.464906 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:57.464888 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 10 15:43:57.465055 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:57.464971 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 10 15:43:57.465489 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:57.465448 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-5d6j1s5o3capj\"" Sep 10 15:43:57.466200 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:57.465975 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-l74wn\"" Sep 10 15:43:57.476408 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:57.476362 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5f68bcbf87-57tzb"] Sep 10 15:43:57.553167 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:57.553126 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/24f85477-eb7b-4fcc-8c84-d9e89d841b57-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f68bcbf87-57tzb\" (UID: \"24f85477-eb7b-4fcc-8c84-d9e89d841b57\") " pod="openshift-monitoring/thanos-querier-5f68bcbf87-57tzb" Sep 10 15:43:57.553262 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:57.553181 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/24f85477-eb7b-4fcc-8c84-d9e89d841b57-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f68bcbf87-57tzb\" (UID: \"24f85477-eb7b-4fcc-8c84-d9e89d841b57\") " pod="openshift-monitoring/thanos-querier-5f68bcbf87-57tzb" Sep 10 15:43:57.553262 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:57.553214 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/24f85477-eb7b-4fcc-8c84-d9e89d841b57-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f68bcbf87-57tzb\" (UID: \"24f85477-eb7b-4fcc-8c84-d9e89d841b57\") " pod="openshift-monitoring/thanos-querier-5f68bcbf87-57tzb" Sep 10 15:43:57.553347 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:57.553291 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24f85477-eb7b-4fcc-8c84-d9e89d841b57-metrics-client-ca\") pod \"thanos-querier-5f68bcbf87-57tzb\" (UID: \"24f85477-eb7b-4fcc-8c84-d9e89d841b57\") " pod="openshift-monitoring/thanos-querier-5f68bcbf87-57tzb" Sep 10 15:43:57.553392 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:57.553363 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/24f85477-eb7b-4fcc-8c84-d9e89d841b57-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f68bcbf87-57tzb\" (UID: \"24f85477-eb7b-4fcc-8c84-d9e89d841b57\") " pod="openshift-monitoring/thanos-querier-5f68bcbf87-57tzb" Sep 10 15:43:57.553431 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:57.553408 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/24f85477-eb7b-4fcc-8c84-d9e89d841b57-secret-grpc-tls\") pod \"thanos-querier-5f68bcbf87-57tzb\" (UID: \"24f85477-eb7b-4fcc-8c84-d9e89d841b57\") " pod="openshift-monitoring/thanos-querier-5f68bcbf87-57tzb" Sep 10 15:43:57.553468 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:57.553433 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/24f85477-eb7b-4fcc-8c84-d9e89d841b57-secret-thanos-querier-tls\") pod \"thanos-querier-5f68bcbf87-57tzb\" (UID: \"24f85477-eb7b-4fcc-8c84-d9e89d841b57\") " pod="openshift-monitoring/thanos-querier-5f68bcbf87-57tzb" Sep 10 15:43:57.553468 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:57.553454 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xrmp\" (UniqueName: \"kubernetes.io/projected/24f85477-eb7b-4fcc-8c84-d9e89d841b57-kube-api-access-4xrmp\") pod \"thanos-querier-5f68bcbf87-57tzb\" (UID: \"24f85477-eb7b-4fcc-8c84-d9e89d841b57\") " pod="openshift-monitoring/thanos-querier-5f68bcbf87-57tzb" Sep 10 15:43:57.654284 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:57.654256 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24f85477-eb7b-4fcc-8c84-d9e89d841b57-metrics-client-ca\") pod \"thanos-querier-5f68bcbf87-57tzb\" (UID: \"24f85477-eb7b-4fcc-8c84-d9e89d841b57\") " pod="openshift-monitoring/thanos-querier-5f68bcbf87-57tzb" Sep 10 15:43:57.654419 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:57.654301 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/24f85477-eb7b-4fcc-8c84-d9e89d841b57-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f68bcbf87-57tzb\" (UID: \"24f85477-eb7b-4fcc-8c84-d9e89d841b57\") " pod="openshift-monitoring/thanos-querier-5f68bcbf87-57tzb" Sep 10 15:43:57.654419 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:57.654321 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/24f85477-eb7b-4fcc-8c84-d9e89d841b57-secret-grpc-tls\") pod \"thanos-querier-5f68bcbf87-57tzb\" (UID: \"24f85477-eb7b-4fcc-8c84-d9e89d841b57\") " pod="openshift-monitoring/thanos-querier-5f68bcbf87-57tzb" Sep 10 15:43:57.654544 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:57.654450 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/24f85477-eb7b-4fcc-8c84-d9e89d841b57-secret-thanos-querier-tls\") pod \"thanos-querier-5f68bcbf87-57tzb\" (UID: \"24f85477-eb7b-4fcc-8c84-d9e89d841b57\") " pod="openshift-monitoring/thanos-querier-5f68bcbf87-57tzb" Sep 10 15:43:57.654544 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:57.654499 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xrmp\" (UniqueName: \"kubernetes.io/projected/24f85477-eb7b-4fcc-8c84-d9e89d841b57-kube-api-access-4xrmp\") pod \"thanos-querier-5f68bcbf87-57tzb\" (UID: \"24f85477-eb7b-4fcc-8c84-d9e89d841b57\") " pod="openshift-monitoring/thanos-querier-5f68bcbf87-57tzb" Sep 10 15:43:57.654653 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:57.654558 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/24f85477-eb7b-4fcc-8c84-d9e89d841b57-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f68bcbf87-57tzb\" (UID: \"24f85477-eb7b-4fcc-8c84-d9e89d841b57\") " pod="openshift-monitoring/thanos-querier-5f68bcbf87-57tzb" Sep 10 15:43:57.654653 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:57.654590 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/24f85477-eb7b-4fcc-8c84-d9e89d841b57-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f68bcbf87-57tzb\" (UID: \"24f85477-eb7b-4fcc-8c84-d9e89d841b57\") " pod="openshift-monitoring/thanos-querier-5f68bcbf87-57tzb" Sep 10 15:43:57.654961 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:57.654939 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/24f85477-eb7b-4fcc-8c84-d9e89d841b57-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f68bcbf87-57tzb\" (UID: \"24f85477-eb7b-4fcc-8c84-d9e89d841b57\") " pod="openshift-monitoring/thanos-querier-5f68bcbf87-57tzb" Sep 10 15:43:57.655140 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:57.655107 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24f85477-eb7b-4fcc-8c84-d9e89d841b57-metrics-client-ca\") pod \"thanos-querier-5f68bcbf87-57tzb\" (UID: \"24f85477-eb7b-4fcc-8c84-d9e89d841b57\") " pod="openshift-monitoring/thanos-querier-5f68bcbf87-57tzb" Sep 10 15:43:57.657250 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:57.657226 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/24f85477-eb7b-4fcc-8c84-d9e89d841b57-secret-grpc-tls\") pod \"thanos-querier-5f68bcbf87-57tzb\" (UID: \"24f85477-eb7b-4fcc-8c84-d9e89d841b57\") " pod="openshift-monitoring/thanos-querier-5f68bcbf87-57tzb" Sep 10 15:43:57.657567 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:57.657540 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/24f85477-eb7b-4fcc-8c84-d9e89d841b57-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5f68bcbf87-57tzb\" (UID: \"24f85477-eb7b-4fcc-8c84-d9e89d841b57\") " pod="openshift-monitoring/thanos-querier-5f68bcbf87-57tzb" Sep 10 15:43:57.657659 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:57.657612 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/24f85477-eb7b-4fcc-8c84-d9e89d841b57-secret-thanos-querier-tls\") pod \"thanos-querier-5f68bcbf87-57tzb\" (UID: \"24f85477-eb7b-4fcc-8c84-d9e89d841b57\") " pod="openshift-monitoring/thanos-querier-5f68bcbf87-57tzb" Sep 10 15:43:57.657711 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:57.657692 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/24f85477-eb7b-4fcc-8c84-d9e89d841b57-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5f68bcbf87-57tzb\" (UID: \"24f85477-eb7b-4fcc-8c84-d9e89d841b57\") " pod="openshift-monitoring/thanos-querier-5f68bcbf87-57tzb" Sep 10 15:43:57.657821 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:57.657796 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/24f85477-eb7b-4fcc-8c84-d9e89d841b57-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5f68bcbf87-57tzb\" (UID: \"24f85477-eb7b-4fcc-8c84-d9e89d841b57\") " pod="openshift-monitoring/thanos-querier-5f68bcbf87-57tzb" Sep 10 15:43:57.658022 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:57.658005 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/24f85477-eb7b-4fcc-8c84-d9e89d841b57-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5f68bcbf87-57tzb\" (UID: \"24f85477-eb7b-4fcc-8c84-d9e89d841b57\") " pod="openshift-monitoring/thanos-querier-5f68bcbf87-57tzb" Sep 10 15:43:57.662339 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:57.662316 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xrmp\" (UniqueName: \"kubernetes.io/projected/24f85477-eb7b-4fcc-8c84-d9e89d841b57-kube-api-access-4xrmp\") pod \"thanos-querier-5f68bcbf87-57tzb\" (UID: \"24f85477-eb7b-4fcc-8c84-d9e89d841b57\") " pod="openshift-monitoring/thanos-querier-5f68bcbf87-57tzb" Sep 10 15:43:57.680030 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:57.680007 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:57.777674 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:57.777652 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5f68bcbf87-57tzb" Sep 10 15:43:57.912600 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:57.912534 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:57.956371 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:57.956344 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8hcsc" event={"ID":"8b0d03d5-0328-428f-94cf-8f57dba68534","Type":"ContainerStarted","Data":"248e27cae1873784e24c5e0069b1499510c878e2c69f3318609476c4775f7392"} Sep 10 15:43:57.956733 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:57.956423 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8hcsc" event={"ID":"8b0d03d5-0328-428f-94cf-8f57dba68534","Type":"ContainerStarted","Data":"eb52711268cb8c0b2486ab18acb55e7013b8d038bbec94059b5bfc303e84c09d"} Sep 10 15:43:57.958025 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:57.958003 2583 generic.go:356] "Generic (PLEG): container finished" podID="590890d6-a6d7-4868-b14e-f02eb06f758a" containerID="6a635ef8af029de6f7cd49ceedb2005cf4b572b8ed52c58e90f014e5f1da69d0" exitCode=0 Sep 10 15:43:57.958127 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:57.958053 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"590890d6-a6d7-4868-b14e-f02eb06f758a","Type":"ContainerDied","Data":"6a635ef8af029de6f7cd49ceedb2005cf4b572b8ed52c58e90f014e5f1da69d0"} Sep 10 15:43:57.977498 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:57.977453 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8hcsc" podStartSLOduration=2.768640079 podStartE2EDuration="3.977439457s" totalImagesPullingTime="1.138933169s" totalInitContainerRuntime="69.866209ms" isStatefulPod=true podCreationTimestamp="2026-09-10 15:43:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:57.976440273 +0000 UTC m=+114.951700226" watchObservedRunningTime="2026-09-10 15:43:57.977439457 +0000 UTC m=+114.952699410" Sep 10 15:43:58.060345 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:58.060318 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5f68bcbf87-57tzb"] Sep 10 15:43:58.063036 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:43:58.063016 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24f85477_eb7b_4fcc_8c84_d9e89d841b57.slice/crio-daecb5971ab67e10faa5c160218a6790cda44335ae066809e4a4b0d06c8da17f WatchSource:0}: Error finding container daecb5971ab67e10faa5c160218a6790cda44335ae066809e4a4b0d06c8da17f: Status 404 returned error can't find the container with id daecb5971ab67e10faa5c160218a6790cda44335ae066809e4a4b0d06c8da17f Sep 10 15:43:58.630589 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:58.630556 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-f9bdf75f9-r2swd"] Sep 10 15:43:58.632935 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:58.632910 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-f9bdf75f9-r2swd" Sep 10 15:43:58.635298 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:58.635122 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Sep 10 15:43:58.636011 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:58.635986 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 10 15:43:58.636117 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:58.636056 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-2kc99b41qm90g\"" Sep 10 15:43:58.636117 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:58.635986 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Sep 10 15:43:58.636409 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:58.636373 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Sep 10 15:43:58.636498 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:58.636432 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-hpt6h\"" Sep 10 15:43:58.641227 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:58.641176 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-f9bdf75f9-r2swd"] Sep 10 15:43:58.764801 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:58.764758 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/27b3c4fd-2cc4-4298-8ebc-0d93e73097f8-audit-log\") pod \"metrics-server-f9bdf75f9-r2swd\" (UID: \"27b3c4fd-2cc4-4298-8ebc-0d93e73097f8\") " pod="openshift-monitoring/metrics-server-f9bdf75f9-r2swd" Sep 10 15:43:58.764971 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:58.764808 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6nlt\" (UniqueName: \"kubernetes.io/projected/27b3c4fd-2cc4-4298-8ebc-0d93e73097f8-kube-api-access-s6nlt\") pod \"metrics-server-f9bdf75f9-r2swd\" (UID: \"27b3c4fd-2cc4-4298-8ebc-0d93e73097f8\") " pod="openshift-monitoring/metrics-server-f9bdf75f9-r2swd" Sep 10 15:43:58.764971 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:58.764926 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/27b3c4fd-2cc4-4298-8ebc-0d93e73097f8-secret-metrics-server-tls\") pod \"metrics-server-f9bdf75f9-r2swd\" (UID: \"27b3c4fd-2cc4-4298-8ebc-0d93e73097f8\") " pod="openshift-monitoring/metrics-server-f9bdf75f9-r2swd" Sep 10 15:43:58.764971 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:58.764959 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/27b3c4fd-2cc4-4298-8ebc-0d93e73097f8-metrics-server-audit-profiles\") pod \"metrics-server-f9bdf75f9-r2swd\" (UID: \"27b3c4fd-2cc4-4298-8ebc-0d93e73097f8\") " pod="openshift-monitoring/metrics-server-f9bdf75f9-r2swd" Sep 10 15:43:58.765128 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:58.764994 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27b3c4fd-2cc4-4298-8ebc-0d93e73097f8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-f9bdf75f9-r2swd\" (UID: \"27b3c4fd-2cc4-4298-8ebc-0d93e73097f8\") " pod="openshift-monitoring/metrics-server-f9bdf75f9-r2swd" Sep 10 15:43:58.765128 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:58.765046 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/27b3c4fd-2cc4-4298-8ebc-0d93e73097f8-client-ca-bundle\") pod \"metrics-server-f9bdf75f9-r2swd\" (UID: \"27b3c4fd-2cc4-4298-8ebc-0d93e73097f8\") " pod="openshift-monitoring/metrics-server-f9bdf75f9-r2swd" Sep 10 15:43:58.765128 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:58.765088 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/27b3c4fd-2cc4-4298-8ebc-0d93e73097f8-secret-metrics-server-client-certs\") pod \"metrics-server-f9bdf75f9-r2swd\" (UID: \"27b3c4fd-2cc4-4298-8ebc-0d93e73097f8\") " pod="openshift-monitoring/metrics-server-f9bdf75f9-r2swd" Sep 10 15:43:58.866294 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:58.866263 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/27b3c4fd-2cc4-4298-8ebc-0d93e73097f8-client-ca-bundle\") pod \"metrics-server-f9bdf75f9-r2swd\" (UID: \"27b3c4fd-2cc4-4298-8ebc-0d93e73097f8\") " pod="openshift-monitoring/metrics-server-f9bdf75f9-r2swd" Sep 10 15:43:58.866456 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:58.866301 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/27b3c4fd-2cc4-4298-8ebc-0d93e73097f8-secret-metrics-server-client-certs\") pod \"metrics-server-f9bdf75f9-r2swd\" (UID: \"27b3c4fd-2cc4-4298-8ebc-0d93e73097f8\") " pod="openshift-monitoring/metrics-server-f9bdf75f9-r2swd" Sep 10 15:43:58.866456 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:58.866362 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/27b3c4fd-2cc4-4298-8ebc-0d93e73097f8-audit-log\") pod \"metrics-server-f9bdf75f9-r2swd\" (UID: \"27b3c4fd-2cc4-4298-8ebc-0d93e73097f8\") " pod="openshift-monitoring/metrics-server-f9bdf75f9-r2swd" Sep 10 15:43:58.866456 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:58.866414 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6nlt\" (UniqueName: \"kubernetes.io/projected/27b3c4fd-2cc4-4298-8ebc-0d93e73097f8-kube-api-access-s6nlt\") pod \"metrics-server-f9bdf75f9-r2swd\" (UID: \"27b3c4fd-2cc4-4298-8ebc-0d93e73097f8\") " pod="openshift-monitoring/metrics-server-f9bdf75f9-r2swd" Sep 10 15:43:58.866606 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:58.866461 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/27b3c4fd-2cc4-4298-8ebc-0d93e73097f8-secret-metrics-server-tls\") pod \"metrics-server-f9bdf75f9-r2swd\" (UID: \"27b3c4fd-2cc4-4298-8ebc-0d93e73097f8\") " pod="openshift-monitoring/metrics-server-f9bdf75f9-r2swd" Sep 10 15:43:58.866606 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:58.866573 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/27b3c4fd-2cc4-4298-8ebc-0d93e73097f8-metrics-server-audit-profiles\") pod \"metrics-server-f9bdf75f9-r2swd\" (UID: \"27b3c4fd-2cc4-4298-8ebc-0d93e73097f8\") " pod="openshift-monitoring/metrics-server-f9bdf75f9-r2swd" Sep 10 15:43:58.866700 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:58.866620 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27b3c4fd-2cc4-4298-8ebc-0d93e73097f8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-f9bdf75f9-r2swd\" (UID: \"27b3c4fd-2cc4-4298-8ebc-0d93e73097f8\") " pod="openshift-monitoring/metrics-server-f9bdf75f9-r2swd" Sep 10 15:43:58.867101 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:58.867065 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/27b3c4fd-2cc4-4298-8ebc-0d93e73097f8-audit-log\") pod \"metrics-server-f9bdf75f9-r2swd\" (UID: \"27b3c4fd-2cc4-4298-8ebc-0d93e73097f8\") " pod="openshift-monitoring/metrics-server-f9bdf75f9-r2swd" Sep 10 15:43:58.867320 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:58.867301 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27b3c4fd-2cc4-4298-8ebc-0d93e73097f8-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-f9bdf75f9-r2swd\" (UID: \"27b3c4fd-2cc4-4298-8ebc-0d93e73097f8\") " pod="openshift-monitoring/metrics-server-f9bdf75f9-r2swd" Sep 10 15:43:58.867676 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:58.867649 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/27b3c4fd-2cc4-4298-8ebc-0d93e73097f8-metrics-server-audit-profiles\") pod \"metrics-server-f9bdf75f9-r2swd\" (UID: \"27b3c4fd-2cc4-4298-8ebc-0d93e73097f8\") " pod="openshift-monitoring/metrics-server-f9bdf75f9-r2swd" Sep 10 15:43:58.868792 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:58.868764 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/27b3c4fd-2cc4-4298-8ebc-0d93e73097f8-client-ca-bundle\") pod \"metrics-server-f9bdf75f9-r2swd\" (UID: \"27b3c4fd-2cc4-4298-8ebc-0d93e73097f8\") " pod="openshift-monitoring/metrics-server-f9bdf75f9-r2swd" Sep 10 15:43:58.868874 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:58.868805 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/27b3c4fd-2cc4-4298-8ebc-0d93e73097f8-secret-metrics-server-client-certs\") pod \"metrics-server-f9bdf75f9-r2swd\" (UID: \"27b3c4fd-2cc4-4298-8ebc-0d93e73097f8\") " pod="openshift-monitoring/metrics-server-f9bdf75f9-r2swd" Sep 10 15:43:58.868922 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:58.868912 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/27b3c4fd-2cc4-4298-8ebc-0d93e73097f8-secret-metrics-server-tls\") pod \"metrics-server-f9bdf75f9-r2swd\" (UID: \"27b3c4fd-2cc4-4298-8ebc-0d93e73097f8\") " pod="openshift-monitoring/metrics-server-f9bdf75f9-r2swd" Sep 10 15:43:58.874454 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:58.874430 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6nlt\" (UniqueName: \"kubernetes.io/projected/27b3c4fd-2cc4-4298-8ebc-0d93e73097f8-kube-api-access-s6nlt\") pod \"metrics-server-f9bdf75f9-r2swd\" (UID: \"27b3c4fd-2cc4-4298-8ebc-0d93e73097f8\") " pod="openshift-monitoring/metrics-server-f9bdf75f9-r2swd" Sep 10 15:43:58.946106 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:58.946023 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-f9bdf75f9-r2swd" Sep 10 15:43:58.965283 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:58.965184 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f68bcbf87-57tzb" event={"ID":"24f85477-eb7b-4fcc-8c84-d9e89d841b57","Type":"ContainerStarted","Data":"daecb5971ab67e10faa5c160218a6790cda44335ae066809e4a4b0d06c8da17f"} Sep 10 15:43:59.107859 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:59.107781 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:59.116519 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:59.116487 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-t4sw4"] Sep 10 15:43:59.119167 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:59.119145 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t4sw4" Sep 10 15:43:59.121441 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:59.121418 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-dvlvv\"" Sep 10 15:43:59.121835 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:59.121811 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 10 15:43:59.129969 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:59.129948 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-t4sw4"] Sep 10 15:43:59.185549 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:59.184620 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:59.226743 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:59.226639 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-f9bdf75f9-r2swd"] Sep 10 15:43:59.271050 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:59.271015 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z72w\" (UniqueName: \"kubernetes.io/projected/d807c2fa-db5f-4003-8e8d-0938f21950d5-kube-api-access-9z72w\") pod \"monitoring-plugin-85bbbbbf4-t4sw4\" (UID: \"d807c2fa-db5f-4003-8e8d-0938f21950d5\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t4sw4" Sep 10 15:43:59.271203 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:59.271180 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d807c2fa-db5f-4003-8e8d-0938f21950d5-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-t4sw4\" (UID: \"d807c2fa-db5f-4003-8e8d-0938f21950d5\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t4sw4" Sep 10 15:43:59.372095 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:59.372056 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d807c2fa-db5f-4003-8e8d-0938f21950d5-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-t4sw4\" (UID: \"d807c2fa-db5f-4003-8e8d-0938f21950d5\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t4sw4" Sep 10 15:43:59.372263 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:59.372133 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z72w\" (UniqueName: \"kubernetes.io/projected/d807c2fa-db5f-4003-8e8d-0938f21950d5-kube-api-access-9z72w\") pod \"monitoring-plugin-85bbbbbf4-t4sw4\" (UID: \"d807c2fa-db5f-4003-8e8d-0938f21950d5\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t4sw4" Sep 10 15:43:59.372263 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:59.372242 2583 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Sep 10 15:43:59.372400 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:43:59.372330 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d807c2fa-db5f-4003-8e8d-0938f21950d5-monitoring-plugin-cert podName:d807c2fa-db5f-4003-8e8d-0938f21950d5 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:59.87230717 +0000 UTC m=+116.847567104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/d807c2fa-db5f-4003-8e8d-0938f21950d5-monitoring-plugin-cert") pod "monitoring-plugin-85bbbbbf4-t4sw4" (UID: "d807c2fa-db5f-4003-8e8d-0938f21950d5") : secret "monitoring-plugin-cert" not found Sep 10 15:43:59.380722 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:59.380696 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z72w\" (UniqueName: \"kubernetes.io/projected/d807c2fa-db5f-4003-8e8d-0938f21950d5-kube-api-access-9z72w\") pod \"monitoring-plugin-85bbbbbf4-t4sw4\" (UID: \"d807c2fa-db5f-4003-8e8d-0938f21950d5\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t4sw4" Sep 10 15:43:59.479749 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:43:59.479686 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27b3c4fd_2cc4_4298_8ebc_0d93e73097f8.slice/crio-4932a8fcdd2e28e591b3740aa237cb1d5f3ed9a8552c2ef8f294c6c5d183db1a WatchSource:0}: Error finding container 4932a8fcdd2e28e591b3740aa237cb1d5f3ed9a8552c2ef8f294c6c5d183db1a: Status 404 returned error can't find the container with id 4932a8fcdd2e28e591b3740aa237cb1d5f3ed9a8552c2ef8f294c6c5d183db1a Sep 10 15:43:59.782117 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:59.782026 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:59.876370 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:59.876340 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d807c2fa-db5f-4003-8e8d-0938f21950d5-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-t4sw4\" (UID: \"d807c2fa-db5f-4003-8e8d-0938f21950d5\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t4sw4" Sep 10 15:43:59.878955 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:59.878926 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d807c2fa-db5f-4003-8e8d-0938f21950d5-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-t4sw4\" (UID: \"d807c2fa-db5f-4003-8e8d-0938f21950d5\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t4sw4" Sep 10 15:43:59.950933 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:59.950894 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:43:59.971172 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:43:59.971140 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-f9bdf75f9-r2swd" event={"ID":"27b3c4fd-2cc4-4298-8ebc-0d93e73097f8","Type":"ContainerStarted","Data":"4932a8fcdd2e28e591b3740aa237cb1d5f3ed9a8552c2ef8f294c6c5d183db1a"} Sep 10 15:44:00.035095 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:00.035068 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t4sw4" Sep 10 15:44:00.313256 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:00.313150 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-t4sw4"] Sep 10 15:44:00.315834 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:44:00.315807 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd807c2fa_db5f_4003_8e8d_0938f21950d5.slice/crio-bde428ee7ee3bbdf79a28b22af2a6fbf5a26c5bff30542185c556329cb7fcd4c WatchSource:0}: Error finding container bde428ee7ee3bbdf79a28b22af2a6fbf5a26c5bff30542185c556329cb7fcd4c: Status 404 returned error can't find the container with id bde428ee7ee3bbdf79a28b22af2a6fbf5a26c5bff30542185c556329cb7fcd4c Sep 10 15:44:00.571981 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:00.571899 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:44:00.987710 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:00.987316 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"590890d6-a6d7-4868-b14e-f02eb06f758a","Type":"ContainerStarted","Data":"9f5208c613cd4e022a22c861a6750a212888fd0c9a1187ff441961f3395ab1c9"} Sep 10 15:44:00.987710 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:00.987428 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"590890d6-a6d7-4868-b14e-f02eb06f758a","Type":"ContainerStarted","Data":"9cfbe988baf73a285b355c2d644f6bd28c0171abb05ba43a621f084bdd351415"} Sep 10 15:44:00.987710 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:00.987488 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"590890d6-a6d7-4868-b14e-f02eb06f758a","Type":"ContainerStarted","Data":"ee7ff93c66d4a8da632c4ea47b2c95ee830e3baa0492cac93a773cf138fb02b1"} Sep 10 15:44:00.987710 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:00.987529 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"590890d6-a6d7-4868-b14e-f02eb06f758a","Type":"ContainerStarted","Data":"9556d29156f3b3d8c1e1863b9e7acc9e5ff4cdc2ae148ddfe6b724a3952b1f7d"} Sep 10 15:44:00.987710 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:00.987567 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"590890d6-a6d7-4868-b14e-f02eb06f758a","Type":"ContainerStarted","Data":"1758dae745b43db7dd6db9133eca848f62092a8be93ca8287049bcd22e3e6474"} Sep 10 15:44:00.990549 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:00.990521 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f68bcbf87-57tzb" event={"ID":"24f85477-eb7b-4fcc-8c84-d9e89d841b57","Type":"ContainerStarted","Data":"94562e1fbd350e740ce629436f0b05054c6510f64cb712dc8d1c48fe6a173aab"} Sep 10 15:44:00.990657 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:00.990592 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f68bcbf87-57tzb" event={"ID":"24f85477-eb7b-4fcc-8c84-d9e89d841b57","Type":"ContainerStarted","Data":"5344fb2ef3549d65071ba2409b4953a17663cc473f783c60efdfe7fcaa40ea66"} Sep 10 15:44:00.990657 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:00.990623 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f68bcbf87-57tzb" event={"ID":"24f85477-eb7b-4fcc-8c84-d9e89d841b57","Type":"ContainerStarted","Data":"832acc58e5c3f6525b3f80f98d1773433a8c036933f74aa82473991c1e4bed34"} Sep 10 15:44:00.992048 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:00.992021 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t4sw4" event={"ID":"d807c2fa-db5f-4003-8e8d-0938f21950d5","Type":"ContainerStarted","Data":"bde428ee7ee3bbdf79a28b22af2a6fbf5a26c5bff30542185c556329cb7fcd4c"} Sep 10 15:44:01.096815 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:01.096761 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:44:01.185211 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:01.185124 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:44:01.718688 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:01.718655 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:44:01.807520 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:01.807485 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:44:01.998126 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:01.998095 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-f9bdf75f9-r2swd" event={"ID":"27b3c4fd-2cc4-4298-8ebc-0d93e73097f8","Type":"ContainerStarted","Data":"3dabda61e2adcf15ffb2a83b2edac536c0b0eaf3b520320019798f7dd3337eb8"} Sep 10 15:44:02.002735 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:02.002703 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"590890d6-a6d7-4868-b14e-f02eb06f758a","Type":"ContainerStarted","Data":"affbf47a9f571bede8b273fe080d474c3e401c34a5b4f2d6ee031a60f96bc416"} Sep 10 15:44:02.006835 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:02.006367 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f68bcbf87-57tzb" event={"ID":"24f85477-eb7b-4fcc-8c84-d9e89d841b57","Type":"ContainerStarted","Data":"c3b12a1a930772cbf98cb2c0161a8e37d9c88f9ca4487f8ca2d10c5dd54e98e4"} Sep 10 15:44:02.006835 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:02.006449 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f68bcbf87-57tzb" event={"ID":"24f85477-eb7b-4fcc-8c84-d9e89d841b57","Type":"ContainerStarted","Data":"20fd20fedcfa6079d9cd747efc85b678418d44b3e9252c290da608fb6f4ede82"} Sep 10 15:44:02.006835 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:02.006491 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5f68bcbf87-57tzb" event={"ID":"24f85477-eb7b-4fcc-8c84-d9e89d841b57","Type":"ContainerStarted","Data":"c8bd62bda3f04eab992de7be394de1d2847786860c9495f633795a32b93275a4"} Sep 10 15:44:02.007093 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:02.006930 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5f68bcbf87-57tzb" Sep 10 15:44:02.008459 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:02.008436 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t4sw4" event={"ID":"d807c2fa-db5f-4003-8e8d-0938f21950d5","Type":"ContainerStarted","Data":"cc522d3c2bd9cb2b83037911daeef8f54cb13b4747472879e282f6281c162969"} Sep 10 15:44:02.008677 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:02.008629 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t4sw4" Sep 10 15:44:02.013550 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:02.013529 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t4sw4" Sep 10 15:44:02.024647 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:02.024576 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-f9bdf75f9-r2swd" podStartSLOduration=1.734830588 podStartE2EDuration="4.024556805s" totalImagesPullingTime="2.289726217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:43:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:44:02.022121676 +0000 UTC m=+118.997381628" watchObservedRunningTime="2026-09-10 15:44:02.024556805 +0000 UTC m=+118.999816759" Sep 10 15:44:02.039203 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:02.039155 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-t4sw4" podStartSLOduration=1.583664895 podStartE2EDuration="3.039143909s" totalImagesPullingTime="1.455479014s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:43:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:44:02.037800218 +0000 UTC m=+119.013060172" watchObservedRunningTime="2026-09-10 15:44:02.039143909 +0000 UTC m=+119.014403865" Sep 10 15:44:02.062215 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:02.062170 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.643547465 podStartE2EDuration="7.062155587s" totalImagesPullingTime="4.405346532s" totalInitContainerRuntime="13.26159ms" isStatefulPod=false podCreationTimestamp="2026-09-10 15:43:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:44:02.060578881 +0000 UTC m=+119.035838865" watchObservedRunningTime="2026-09-10 15:44:02.062155587 +0000 UTC m=+119.037415576" Sep 10 15:44:02.082079 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:02.082016 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5f68bcbf87-57tzb" podStartSLOduration=1.572879113 podStartE2EDuration="5.081994773s" totalImagesPullingTime="3.50911566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:43:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:44:02.080358167 +0000 UTC m=+119.055618124" watchObservedRunningTime="2026-09-10 15:44:02.081994773 +0000 UTC m=+119.057254727" Sep 10 15:44:02.488188 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:02.488161 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-84ff49d5c7-bh94t" Sep 10 15:44:02.677093 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:02.677065 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:44:02.776994 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:02.776901 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:44:04.299625 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:04.299589 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:44:04.369346 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:04.369302 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:44:05.220212 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:05.220155 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:44:05.317692 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:05.317657 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:44:05.419489 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:05.419458 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:44:06.595336 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:06.595307 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:44:07.506587 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:07.506548 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-84ff49d5c7-bh94t" podUID="b69645e0-943e-4301-982a-9d1532d25e56" containerName="registry" containerID="cri-o://68af216535d6e37ae68e175e0586a236f5699f538f764b7c7178148b5d43504e" gracePeriod=30 Sep 10 15:44:07.740840 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:07.740812 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84ff49d5c7-bh94t" Sep 10 15:44:07.844633 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:07.844570 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b69645e0-943e-4301-982a-9d1532d25e56-image-registry-private-configuration\") pod \"b69645e0-943e-4301-982a-9d1532d25e56\" (UID: \"b69645e0-943e-4301-982a-9d1532d25e56\") " Sep 10 15:44:07.844752 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:07.844634 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qw9q\" (UniqueName: \"kubernetes.io/projected/b69645e0-943e-4301-982a-9d1532d25e56-kube-api-access-5qw9q\") pod \"b69645e0-943e-4301-982a-9d1532d25e56\" (UID: \"b69645e0-943e-4301-982a-9d1532d25e56\") " Sep 10 15:44:07.844752 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:07.844655 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b69645e0-943e-4301-982a-9d1532d25e56-installation-pull-secrets\") pod \"b69645e0-943e-4301-982a-9d1532d25e56\" (UID: \"b69645e0-943e-4301-982a-9d1532d25e56\") " Sep 10 15:44:07.844752 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:07.844677 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b69645e0-943e-4301-982a-9d1532d25e56-registry-tls\") pod \"b69645e0-943e-4301-982a-9d1532d25e56\" (UID: \"b69645e0-943e-4301-982a-9d1532d25e56\") " Sep 10 15:44:07.844752 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:07.844718 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b69645e0-943e-4301-982a-9d1532d25e56-trusted-ca\") pod \"b69645e0-943e-4301-982a-9d1532d25e56\" (UID: \"b69645e0-943e-4301-982a-9d1532d25e56\") " Sep 10 15:44:07.844752 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:07.844742 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b69645e0-943e-4301-982a-9d1532d25e56-registry-certificates\") pod \"b69645e0-943e-4301-982a-9d1532d25e56\" (UID: \"b69645e0-943e-4301-982a-9d1532d25e56\") " Sep 10 15:44:07.844996 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:07.844762 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b69645e0-943e-4301-982a-9d1532d25e56-ca-trust-extracted\") pod \"b69645e0-943e-4301-982a-9d1532d25e56\" (UID: \"b69645e0-943e-4301-982a-9d1532d25e56\") " Sep 10 15:44:07.844996 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:07.844840 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b69645e0-943e-4301-982a-9d1532d25e56-bound-sa-token\") pod \"b69645e0-943e-4301-982a-9d1532d25e56\" (UID: \"b69645e0-943e-4301-982a-9d1532d25e56\") " Sep 10 15:44:07.845243 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:07.845210 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b69645e0-943e-4301-982a-9d1532d25e56-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "b69645e0-943e-4301-982a-9d1532d25e56" (UID: "b69645e0-943e-4301-982a-9d1532d25e56"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:44:07.845582 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:07.845553 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b69645e0-943e-4301-982a-9d1532d25e56-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "b69645e0-943e-4301-982a-9d1532d25e56" (UID: "b69645e0-943e-4301-982a-9d1532d25e56"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:44:07.847043 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:07.847016 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b69645e0-943e-4301-982a-9d1532d25e56-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "b69645e0-943e-4301-982a-9d1532d25e56" (UID: "b69645e0-943e-4301-982a-9d1532d25e56"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:44:07.847292 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:07.847269 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b69645e0-943e-4301-982a-9d1532d25e56-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "b69645e0-943e-4301-982a-9d1532d25e56" (UID: "b69645e0-943e-4301-982a-9d1532d25e56"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:44:07.847530 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:07.847481 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b69645e0-943e-4301-982a-9d1532d25e56-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "b69645e0-943e-4301-982a-9d1532d25e56" (UID: "b69645e0-943e-4301-982a-9d1532d25e56"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:44:07.847599 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:07.847531 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b69645e0-943e-4301-982a-9d1532d25e56-kube-api-access-5qw9q" (OuterVolumeSpecName: "kube-api-access-5qw9q") pod "b69645e0-943e-4301-982a-9d1532d25e56" (UID: "b69645e0-943e-4301-982a-9d1532d25e56"). InnerVolumeSpecName "kube-api-access-5qw9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:44:07.847599 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:07.847568 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b69645e0-943e-4301-982a-9d1532d25e56-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b69645e0-943e-4301-982a-9d1532d25e56" (UID: "b69645e0-943e-4301-982a-9d1532d25e56"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:44:07.853298 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:07.853273 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b69645e0-943e-4301-982a-9d1532d25e56-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "b69645e0-943e-4301-982a-9d1532d25e56" (UID: "b69645e0-943e-4301-982a-9d1532d25e56"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:44:07.946319 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:07.946296 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qw9q\" (UniqueName: \"kubernetes.io/projected/b69645e0-943e-4301-982a-9d1532d25e56-kube-api-access-5qw9q\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:44:07.946319 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:07.946317 2583 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b69645e0-943e-4301-982a-9d1532d25e56-installation-pull-secrets\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:44:07.946456 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:07.946327 2583 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b69645e0-943e-4301-982a-9d1532d25e56-registry-tls\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:44:07.946456 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:07.946335 2583 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b69645e0-943e-4301-982a-9d1532d25e56-trusted-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:44:07.946456 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:07.946342 2583 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b69645e0-943e-4301-982a-9d1532d25e56-registry-certificates\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:44:07.946456 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:07.946350 2583 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b69645e0-943e-4301-982a-9d1532d25e56-ca-trust-extracted\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:44:07.946456 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:07.946358 2583 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b69645e0-943e-4301-982a-9d1532d25e56-bound-sa-token\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:44:07.946456 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:07.946367 2583 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b69645e0-943e-4301-982a-9d1532d25e56-image-registry-private-configuration\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:44:08.018764 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:08.018745 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5f68bcbf87-57tzb" Sep 10 15:44:08.032426 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:08.032405 2583 generic.go:356] "Generic (PLEG): container finished" podID="b69645e0-943e-4301-982a-9d1532d25e56" containerID="68af216535d6e37ae68e175e0586a236f5699f538f764b7c7178148b5d43504e" exitCode=0 Sep 10 15:44:08.032510 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:08.032436 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-84ff49d5c7-bh94t" Sep 10 15:44:08.032510 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:08.032456 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84ff49d5c7-bh94t" event={"ID":"b69645e0-943e-4301-982a-9d1532d25e56","Type":"ContainerDied","Data":"68af216535d6e37ae68e175e0586a236f5699f538f764b7c7178148b5d43504e"} Sep 10 15:44:08.032510 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:08.032484 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-84ff49d5c7-bh94t" event={"ID":"b69645e0-943e-4301-982a-9d1532d25e56","Type":"ContainerDied","Data":"65fde700fcec4175428c1a4b92164731f7f151996d2ade89b34ab1581dbf53ef"} Sep 10 15:44:08.032510 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:08.032502 2583 scope.go:117] "RemoveContainer" containerID="68af216535d6e37ae68e175e0586a236f5699f538f764b7c7178148b5d43504e" Sep 10 15:44:08.041221 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:08.041201 2583 scope.go:117] "RemoveContainer" containerID="68af216535d6e37ae68e175e0586a236f5699f538f764b7c7178148b5d43504e" Sep 10 15:44:08.041549 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:44:08.041518 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68af216535d6e37ae68e175e0586a236f5699f538f764b7c7178148b5d43504e\": container with ID starting with 68af216535d6e37ae68e175e0586a236f5699f538f764b7c7178148b5d43504e not found: ID does not exist" containerID="68af216535d6e37ae68e175e0586a236f5699f538f764b7c7178148b5d43504e" Sep 10 15:44:08.041655 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:08.041550 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68af216535d6e37ae68e175e0586a236f5699f538f764b7c7178148b5d43504e"} err="failed to get container status \"68af216535d6e37ae68e175e0586a236f5699f538f764b7c7178148b5d43504e\": rpc error: code = NotFound desc = could not find container \"68af216535d6e37ae68e175e0586a236f5699f538f764b7c7178148b5d43504e\": container with ID starting with 68af216535d6e37ae68e175e0586a236f5699f538f764b7c7178148b5d43504e not found: ID does not exist" Sep 10 15:44:08.059894 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:08.059873 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-84ff49d5c7-bh94t"] Sep 10 15:44:08.062689 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:08.062666 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-84ff49d5c7-bh94t"] Sep 10 15:44:09.514395 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:09.514359 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b69645e0-943e-4301-982a-9d1532d25e56" path="/var/lib/kubelet/pods/b69645e0-943e-4301-982a-9d1532d25e56/volumes" Sep 10 15:44:10.032914 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:10.032828 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:44:10.099636 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:10.099603 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:44:10.383841 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:10.383815 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:44:10.485664 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:10.485637 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:44:10.550194 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:10.550171 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:44:11.679797 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:11.679764 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:44:11.739112 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:11.739091 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:44:12.180613 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:12.180579 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:44:12.274967 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:12.274940 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:44:13.393216 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:13.393177 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcf35967-2a15-4bd4-8837-a058d81d1aff-metrics-certs\") pod \"network-metrics-daemon-qztq2\" (UID: \"bcf35967-2a15-4bd4-8837-a058d81d1aff\") " pod="openshift-multus/network-metrics-daemon-qztq2" Sep 10 15:44:13.395567 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:13.395538 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcf35967-2a15-4bd4-8837-a058d81d1aff-metrics-certs\") pod \"network-metrics-daemon-qztq2\" (UID: \"bcf35967-2a15-4bd4-8837-a058d81d1aff\") " pod="openshift-multus/network-metrics-daemon-qztq2" Sep 10 15:44:13.628468 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:13.628438 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-b79h8\"" Sep 10 15:44:13.637067 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:13.637040 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qztq2" Sep 10 15:44:13.825235 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:13.825197 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:44:13.916823 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:13.916750 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qztq2"] Sep 10 15:44:13.919942 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:44:13.919906 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcf35967_2a15_4bd4_8837_a058d81d1aff.slice/crio-56f479d5cbb09807531d92ab99bcf5a4d208138a0494bb1db27671c5cb1f161c WatchSource:0}: Error finding container 56f479d5cbb09807531d92ab99bcf5a4d208138a0494bb1db27671c5cb1f161c: Status 404 returned error can't find the container with id 56f479d5cbb09807531d92ab99bcf5a4d208138a0494bb1db27671c5cb1f161c Sep 10 15:44:13.920067 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:13.919958 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:44:14.057466 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:14.057425 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qztq2" event={"ID":"bcf35967-2a15-4bd4-8837-a058d81d1aff","Type":"ContainerStarted","Data":"56f479d5cbb09807531d92ab99bcf5a4d208138a0494bb1db27671c5cb1f161c"} Sep 10 15:44:14.343769 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:14.343689 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:44:14.442373 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:14.442344 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:44:14.539447 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:14.539418 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:44:15.063642 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:15.063570 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qztq2" event={"ID":"bcf35967-2a15-4bd4-8837-a058d81d1aff","Type":"ContainerStarted","Data":"ab3b136b933d9dff0e2c2489aebbf01b67db68abd55518567ea80a747eee6806"} Sep 10 15:44:15.063642 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:15.063629 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qztq2" event={"ID":"bcf35967-2a15-4bd4-8837-a058d81d1aff","Type":"ContainerStarted","Data":"83eda58164ecb1ed6feaa5c2cf415172d384facfe38f523ef191cc1ca626fb93"} Sep 10 15:44:15.089786 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:15.089732 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qztq2" podStartSLOduration=131.2329336 podStartE2EDuration="2m12.089717296s" totalImagesPullingTime="856.783696ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:42:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:44:15.088715232 +0000 UTC m=+132.063975184" watchObservedRunningTime="2026-09-10 15:44:15.089717296 +0000 UTC m=+132.064977249" Sep 10 15:44:15.203007 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:15.202983 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:44:15.539767 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:15.539736 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:44:15.636396 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:15.636331 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:44:17.676525 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:17.676492 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:44:17.738263 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:17.738235 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:44:18.946840 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:18.946810 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-f9bdf75f9-r2swd" Sep 10 15:44:18.947189 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:18.946861 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-f9bdf75f9-r2swd" Sep 10 15:44:19.112012 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:19.111983 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:44:19.212697 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:19.212627 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:44:38.951901 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:38.951866 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-f9bdf75f9-r2swd" Sep 10 15:44:38.955669 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:38.955646 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-f9bdf75f9-r2swd" Sep 10 15:44:47.676887 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:47.676849 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:44:47.746796 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:47.746759 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:44:50.198271 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:50.198238 2583 generic.go:356] "Generic (PLEG): container finished" podID="6d7bbb7b-e101-4a09-9211-b2ee0dadd7c6" containerID="2aee758d7f068d0b5f9e1f978e16a1c98cc1718ab2fa625a071026c0e067b97b" exitCode=0 Sep 10 15:44:50.198656 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:50.198310 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-xrvfj" event={"ID":"6d7bbb7b-e101-4a09-9211-b2ee0dadd7c6","Type":"ContainerDied","Data":"2aee758d7f068d0b5f9e1f978e16a1c98cc1718ab2fa625a071026c0e067b97b"} Sep 10 15:44:50.198656 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:50.198629 2583 scope.go:117] "RemoveContainer" containerID="2aee758d7f068d0b5f9e1f978e16a1c98cc1718ab2fa625a071026c0e067b97b" Sep 10 15:44:51.203864 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:44:51.203827 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-xrvfj" event={"ID":"6d7bbb7b-e101-4a09-9211-b2ee0dadd7c6","Type":"ContainerStarted","Data":"384adce058058f969d5dc14f8f434ee1ae6b6b81f23be06f4491ce41e38cd213"} Sep 10 15:45:09.825309 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:09.825270 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:45:09.895989 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:09.895956 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:45:13.601516 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:13.601476 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:45:14.218171 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:14.218138 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:45:18.696889 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:18.696848 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-dcd5cd59c-xhqp9"] Sep 10 15:45:18.697359 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:18.697339 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b69645e0-943e-4301-982a-9d1532d25e56" containerName="registry" Sep 10 15:45:18.697359 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:18.697358 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b69645e0-943e-4301-982a-9d1532d25e56" containerName="registry" Sep 10 15:45:18.697506 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:18.697470 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b69645e0-943e-4301-982a-9d1532d25e56" containerName="registry" Sep 10 15:45:18.700846 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:18.700824 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-dcd5cd59c-xhqp9" Sep 10 15:45:18.702653 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:18.702630 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-9n5qp\"" Sep 10 15:45:18.703188 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:18.703167 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 10 15:45:18.703339 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:18.703318 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 10 15:45:18.703449 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:18.703340 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 10 15:45:18.703449 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:18.703361 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 10 15:45:18.703449 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:18.703367 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 10 15:45:18.714849 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:18.714827 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 10 15:45:18.714939 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:18.714924 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-dcd5cd59c-xhqp9"] Sep 10 15:45:18.772707 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:18.772675 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a1907b32-48aa-4b25-a41e-45c8201a69dd-secret-telemeter-client\") pod \"telemeter-client-dcd5cd59c-xhqp9\" (UID: \"a1907b32-48aa-4b25-a41e-45c8201a69dd\") " pod="openshift-monitoring/telemeter-client-dcd5cd59c-xhqp9" Sep 10 15:45:18.772837 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:18.772714 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a1907b32-48aa-4b25-a41e-45c8201a69dd-telemeter-client-tls\") pod \"telemeter-client-dcd5cd59c-xhqp9\" (UID: \"a1907b32-48aa-4b25-a41e-45c8201a69dd\") " pod="openshift-monitoring/telemeter-client-dcd5cd59c-xhqp9" Sep 10 15:45:18.772837 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:18.772738 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a1907b32-48aa-4b25-a41e-45c8201a69dd-federate-client-tls\") pod \"telemeter-client-dcd5cd59c-xhqp9\" (UID: \"a1907b32-48aa-4b25-a41e-45c8201a69dd\") " pod="openshift-monitoring/telemeter-client-dcd5cd59c-xhqp9" Sep 10 15:45:18.772837 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:18.772816 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1907b32-48aa-4b25-a41e-45c8201a69dd-serving-certs-ca-bundle\") pod \"telemeter-client-dcd5cd59c-xhqp9\" (UID: \"a1907b32-48aa-4b25-a41e-45c8201a69dd\") " pod="openshift-monitoring/telemeter-client-dcd5cd59c-xhqp9" Sep 10 15:45:18.772947 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:18.772890 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1907b32-48aa-4b25-a41e-45c8201a69dd-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-dcd5cd59c-xhqp9\" (UID: \"a1907b32-48aa-4b25-a41e-45c8201a69dd\") " pod="openshift-monitoring/telemeter-client-dcd5cd59c-xhqp9" Sep 10 15:45:18.772947 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:18.772927 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwx9k\" (UniqueName: \"kubernetes.io/projected/a1907b32-48aa-4b25-a41e-45c8201a69dd-kube-api-access-bwx9k\") pod \"telemeter-client-dcd5cd59c-xhqp9\" (UID: \"a1907b32-48aa-4b25-a41e-45c8201a69dd\") " pod="openshift-monitoring/telemeter-client-dcd5cd59c-xhqp9" Sep 10 15:45:18.773012 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:18.772952 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1907b32-48aa-4b25-a41e-45c8201a69dd-metrics-client-ca\") pod \"telemeter-client-dcd5cd59c-xhqp9\" (UID: \"a1907b32-48aa-4b25-a41e-45c8201a69dd\") " pod="openshift-monitoring/telemeter-client-dcd5cd59c-xhqp9" Sep 10 15:45:18.773012 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:18.772981 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1907b32-48aa-4b25-a41e-45c8201a69dd-telemeter-trusted-ca-bundle\") pod \"telemeter-client-dcd5cd59c-xhqp9\" (UID: \"a1907b32-48aa-4b25-a41e-45c8201a69dd\") " pod="openshift-monitoring/telemeter-client-dcd5cd59c-xhqp9" Sep 10 15:45:18.873454 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:18.873428 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1907b32-48aa-4b25-a41e-45c8201a69dd-serving-certs-ca-bundle\") pod \"telemeter-client-dcd5cd59c-xhqp9\" (UID: \"a1907b32-48aa-4b25-a41e-45c8201a69dd\") " pod="openshift-monitoring/telemeter-client-dcd5cd59c-xhqp9" Sep 10 15:45:18.873558 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:18.873462 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1907b32-48aa-4b25-a41e-45c8201a69dd-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-dcd5cd59c-xhqp9\" (UID: \"a1907b32-48aa-4b25-a41e-45c8201a69dd\") " pod="openshift-monitoring/telemeter-client-dcd5cd59c-xhqp9" Sep 10 15:45:18.873558 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:18.873484 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwx9k\" (UniqueName: \"kubernetes.io/projected/a1907b32-48aa-4b25-a41e-45c8201a69dd-kube-api-access-bwx9k\") pod \"telemeter-client-dcd5cd59c-xhqp9\" (UID: \"a1907b32-48aa-4b25-a41e-45c8201a69dd\") " pod="openshift-monitoring/telemeter-client-dcd5cd59c-xhqp9" Sep 10 15:45:18.873558 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:18.873507 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1907b32-48aa-4b25-a41e-45c8201a69dd-metrics-client-ca\") pod \"telemeter-client-dcd5cd59c-xhqp9\" (UID: \"a1907b32-48aa-4b25-a41e-45c8201a69dd\") " pod="openshift-monitoring/telemeter-client-dcd5cd59c-xhqp9" Sep 10 15:45:18.873558 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:18.873533 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1907b32-48aa-4b25-a41e-45c8201a69dd-telemeter-trusted-ca-bundle\") pod \"telemeter-client-dcd5cd59c-xhqp9\" (UID: \"a1907b32-48aa-4b25-a41e-45c8201a69dd\") " pod="openshift-monitoring/telemeter-client-dcd5cd59c-xhqp9" Sep 10 15:45:18.873727 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:18.873596 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a1907b32-48aa-4b25-a41e-45c8201a69dd-secret-telemeter-client\") pod \"telemeter-client-dcd5cd59c-xhqp9\" (UID: \"a1907b32-48aa-4b25-a41e-45c8201a69dd\") " pod="openshift-monitoring/telemeter-client-dcd5cd59c-xhqp9" Sep 10 15:45:18.873727 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:18.873619 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a1907b32-48aa-4b25-a41e-45c8201a69dd-telemeter-client-tls\") pod \"telemeter-client-dcd5cd59c-xhqp9\" (UID: \"a1907b32-48aa-4b25-a41e-45c8201a69dd\") " pod="openshift-monitoring/telemeter-client-dcd5cd59c-xhqp9" Sep 10 15:45:18.873727 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:18.873633 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a1907b32-48aa-4b25-a41e-45c8201a69dd-federate-client-tls\") pod \"telemeter-client-dcd5cd59c-xhqp9\" (UID: \"a1907b32-48aa-4b25-a41e-45c8201a69dd\") " pod="openshift-monitoring/telemeter-client-dcd5cd59c-xhqp9" Sep 10 15:45:18.874188 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:18.874162 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1907b32-48aa-4b25-a41e-45c8201a69dd-serving-certs-ca-bundle\") pod \"telemeter-client-dcd5cd59c-xhqp9\" (UID: \"a1907b32-48aa-4b25-a41e-45c8201a69dd\") " pod="openshift-monitoring/telemeter-client-dcd5cd59c-xhqp9" Sep 10 15:45:18.874341 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:18.874314 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1907b32-48aa-4b25-a41e-45c8201a69dd-metrics-client-ca\") pod \"telemeter-client-dcd5cd59c-xhqp9\" (UID: \"a1907b32-48aa-4b25-a41e-45c8201a69dd\") " pod="openshift-monitoring/telemeter-client-dcd5cd59c-xhqp9" Sep 10 15:45:18.874507 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:18.874431 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1907b32-48aa-4b25-a41e-45c8201a69dd-telemeter-trusted-ca-bundle\") pod \"telemeter-client-dcd5cd59c-xhqp9\" (UID: \"a1907b32-48aa-4b25-a41e-45c8201a69dd\") " pod="openshift-monitoring/telemeter-client-dcd5cd59c-xhqp9" Sep 10 15:45:18.876211 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:18.876192 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a1907b32-48aa-4b25-a41e-45c8201a69dd-federate-client-tls\") pod \"telemeter-client-dcd5cd59c-xhqp9\" (UID: \"a1907b32-48aa-4b25-a41e-45c8201a69dd\") " pod="openshift-monitoring/telemeter-client-dcd5cd59c-xhqp9" Sep 10 15:45:18.876285 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:18.876212 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a1907b32-48aa-4b25-a41e-45c8201a69dd-secret-telemeter-client\") pod \"telemeter-client-dcd5cd59c-xhqp9\" (UID: \"a1907b32-48aa-4b25-a41e-45c8201a69dd\") " pod="openshift-monitoring/telemeter-client-dcd5cd59c-xhqp9" Sep 10 15:45:18.876536 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:18.876516 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1907b32-48aa-4b25-a41e-45c8201a69dd-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-dcd5cd59c-xhqp9\" (UID: \"a1907b32-48aa-4b25-a41e-45c8201a69dd\") " pod="openshift-monitoring/telemeter-client-dcd5cd59c-xhqp9" Sep 10 15:45:18.876583 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:18.876569 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a1907b32-48aa-4b25-a41e-45c8201a69dd-telemeter-client-tls\") pod \"telemeter-client-dcd5cd59c-xhqp9\" (UID: \"a1907b32-48aa-4b25-a41e-45c8201a69dd\") " pod="openshift-monitoring/telemeter-client-dcd5cd59c-xhqp9" Sep 10 15:45:18.882031 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:18.882009 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwx9k\" (UniqueName: \"kubernetes.io/projected/a1907b32-48aa-4b25-a41e-45c8201a69dd-kube-api-access-bwx9k\") pod \"telemeter-client-dcd5cd59c-xhqp9\" (UID: \"a1907b32-48aa-4b25-a41e-45c8201a69dd\") " pod="openshift-monitoring/telemeter-client-dcd5cd59c-xhqp9" Sep 10 15:45:19.014928 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:19.014843 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-dcd5cd59c-xhqp9" Sep 10 15:45:19.291556 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:19.291531 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-dcd5cd59c-xhqp9"] Sep 10 15:45:19.294138 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:45:19.294111 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1907b32_48aa_4b25_a41e_45c8201a69dd.slice/crio-25fec2f2d0ca94b66b24644d33611bcf4937be0fbf0e44f6c9317af109f17aa5 WatchSource:0}: Error finding container 25fec2f2d0ca94b66b24644d33611bcf4937be0fbf0e44f6c9317af109f17aa5: Status 404 returned error can't find the container with id 25fec2f2d0ca94b66b24644d33611bcf4937be0fbf0e44f6c9317af109f17aa5 Sep 10 15:45:19.306532 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:19.306507 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-dcd5cd59c-xhqp9" event={"ID":"a1907b32-48aa-4b25-a41e-45c8201a69dd","Type":"ContainerStarted","Data":"25fec2f2d0ca94b66b24644d33611bcf4937be0fbf0e44f6c9317af109f17aa5"} Sep 10 15:45:22.319997 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:22.319963 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-dcd5cd59c-xhqp9" event={"ID":"a1907b32-48aa-4b25-a41e-45c8201a69dd","Type":"ContainerStarted","Data":"bfeef78d2b54e32066cb678b5ad389d042d55877db35c80672a7a57dbc5c74c1"} Sep 10 15:45:22.320421 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:22.320027 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-dcd5cd59c-xhqp9" event={"ID":"a1907b32-48aa-4b25-a41e-45c8201a69dd","Type":"ContainerStarted","Data":"12192f8b8bf20de366dfbdbb987bff146748aa9918dd79f38f7c42e4b683652b"} Sep 10 15:45:22.320421 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:22.320056 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-dcd5cd59c-xhqp9" event={"ID":"a1907b32-48aa-4b25-a41e-45c8201a69dd","Type":"ContainerStarted","Data":"fe8215ea2013d9196549f0ea4afe1bd3a1224fa8a66bd20cbb982672f5d2fae0"} Sep 10 15:45:22.342369 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:22.342323 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-dcd5cd59c-xhqp9" podStartSLOduration=1.525253374 podStartE2EDuration="4.34230809s" totalImagesPullingTime="2.817054716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:45:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:45:22.341375893 +0000 UTC m=+199.316635845" watchObservedRunningTime="2026-09-10 15:45:22.34230809 +0000 UTC m=+199.317568056" Sep 10 15:45:22.483792 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:22.483736 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:45:24.032619 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:24.032583 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:45:28.882599 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:28.882550 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:45:28.943920 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:28.943890 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:45:30.621583 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:30.621553 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:45:30.735613 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:30.735583 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:45:32.236367 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:32.236333 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:45:32.296057 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:32.296028 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:45:40.755556 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:40.755523 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:45:40.816780 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:40.816752 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:45:41.770872 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:41.770832 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:45:41.881978 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:41.881942 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:45:41.943530 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:41.943501 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:45:47.702515 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:47.702481 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:45:47.806180 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:45:47.806152 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:46:14.497189 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:14.497155 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:46:14.591934 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:14.591899 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:46:15.327475 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:15.327448 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dtm2h"] Sep 10 15:46:15.330753 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:15.330738 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dtm2h" Sep 10 15:46:15.333015 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:15.332987 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 10 15:46:15.333204 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:15.333031 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 10 15:46:15.333622 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:15.333606 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-7k9nf\"" Sep 10 15:46:15.341250 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:15.341227 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dtm2h"] Sep 10 15:46:15.391783 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:15.391758 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf225e37-88c0-4f9c-95f2-d3138e251a5d-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dtm2h\" (UID: \"cf225e37-88c0-4f9c-95f2-d3138e251a5d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dtm2h" Sep 10 15:46:15.391904 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:15.391792 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf225e37-88c0-4f9c-95f2-d3138e251a5d-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dtm2h\" (UID: \"cf225e37-88c0-4f9c-95f2-d3138e251a5d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dtm2h" Sep 10 15:46:15.391904 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:15.391874 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-824gn\" (UniqueName: \"kubernetes.io/projected/cf225e37-88c0-4f9c-95f2-d3138e251a5d-kube-api-access-824gn\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dtm2h\" (UID: \"cf225e37-88c0-4f9c-95f2-d3138e251a5d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dtm2h" Sep 10 15:46:15.492532 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:15.492504 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-824gn\" (UniqueName: \"kubernetes.io/projected/cf225e37-88c0-4f9c-95f2-d3138e251a5d-kube-api-access-824gn\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dtm2h\" (UID: \"cf225e37-88c0-4f9c-95f2-d3138e251a5d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dtm2h" Sep 10 15:46:15.492626 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:15.492567 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf225e37-88c0-4f9c-95f2-d3138e251a5d-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dtm2h\" (UID: \"cf225e37-88c0-4f9c-95f2-d3138e251a5d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dtm2h" Sep 10 15:46:15.492626 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:15.492592 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf225e37-88c0-4f9c-95f2-d3138e251a5d-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dtm2h\" (UID: \"cf225e37-88c0-4f9c-95f2-d3138e251a5d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dtm2h" Sep 10 15:46:15.492942 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:15.492924 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf225e37-88c0-4f9c-95f2-d3138e251a5d-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dtm2h\" (UID: \"cf225e37-88c0-4f9c-95f2-d3138e251a5d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dtm2h" Sep 10 15:46:15.493011 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:15.492957 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf225e37-88c0-4f9c-95f2-d3138e251a5d-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dtm2h\" (UID: \"cf225e37-88c0-4f9c-95f2-d3138e251a5d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dtm2h" Sep 10 15:46:15.501871 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:15.501851 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-824gn\" (UniqueName: \"kubernetes.io/projected/cf225e37-88c0-4f9c-95f2-d3138e251a5d-kube-api-access-824gn\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dtm2h\" (UID: \"cf225e37-88c0-4f9c-95f2-d3138e251a5d\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dtm2h" Sep 10 15:46:15.640159 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:15.640137 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dtm2h" Sep 10 15:46:15.909056 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:15.908983 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dtm2h"] Sep 10 15:46:15.911932 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:46:15.911897 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf225e37_88c0_4f9c_95f2_d3138e251a5d.slice/crio-0981c99cca5ecc9cac482712f3ab73636914593e3038b5c7955aab3d73c3664c WatchSource:0}: Error finding container 0981c99cca5ecc9cac482712f3ab73636914593e3038b5c7955aab3d73c3664c: Status 404 returned error can't find the container with id 0981c99cca5ecc9cac482712f3ab73636914593e3038b5c7955aab3d73c3664c Sep 10 15:46:16.527040 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:16.527003 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dtm2h" event={"ID":"cf225e37-88c0-4f9c-95f2-d3138e251a5d","Type":"ContainerStarted","Data":"0981c99cca5ecc9cac482712f3ab73636914593e3038b5c7955aab3d73c3664c"} Sep 10 15:46:23.559532 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:23.559496 2583 generic.go:356] "Generic (PLEG): container finished" podID="cf225e37-88c0-4f9c-95f2-d3138e251a5d" containerID="4f1ce0038b86b29e72eecb3ccbf98062f99ec804b3077a347d215627db9ef599" exitCode=0 Sep 10 15:46:23.559991 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:23.559577 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dtm2h" event={"ID":"cf225e37-88c0-4f9c-95f2-d3138e251a5d","Type":"ContainerDied","Data":"4f1ce0038b86b29e72eecb3ccbf98062f99ec804b3077a347d215627db9ef599"} Sep 10 15:46:26.573901 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:26.573876 2583 generic.go:356] "Generic (PLEG): container finished" podID="cf225e37-88c0-4f9c-95f2-d3138e251a5d" containerID="1219ea46973b04a3312551c3e92d6058177e6380667454792037bc212653e8c7" exitCode=0 Sep 10 15:46:26.574182 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:26.573911 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dtm2h" event={"ID":"cf225e37-88c0-4f9c-95f2-d3138e251a5d","Type":"ContainerDied","Data":"1219ea46973b04a3312551c3e92d6058177e6380667454792037bc212653e8c7"} Sep 10 15:46:33.606949 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:33.606918 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dtm2h" event={"ID":"cf225e37-88c0-4f9c-95f2-d3138e251a5d","Type":"ContainerStarted","Data":"5ea08aff9853cdc695142bf7c1cca945d19a5823420a630de82c998e116f2cef"} Sep 10 15:46:33.625297 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:33.625243 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dtm2h" podStartSLOduration=2.170774105 podStartE2EDuration="18.625228523s" totalImagesPullingTime="16.454454418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:46:15 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:46:33.622633244 +0000 UTC m=+270.597893188" watchObservedRunningTime="2026-09-10 15:46:33.625228523 +0000 UTC m=+270.600488476" Sep 10 15:46:34.612551 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:34.612516 2583 generic.go:356] "Generic (PLEG): container finished" podID="cf225e37-88c0-4f9c-95f2-d3138e251a5d" containerID="5ea08aff9853cdc695142bf7c1cca945d19a5823420a630de82c998e116f2cef" exitCode=0 Sep 10 15:46:34.612948 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:34.612596 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dtm2h" event={"ID":"cf225e37-88c0-4f9c-95f2-d3138e251a5d","Type":"ContainerDied","Data":"5ea08aff9853cdc695142bf7c1cca945d19a5823420a630de82c998e116f2cef"} Sep 10 15:46:35.739142 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:35.739119 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dtm2h" Sep 10 15:46:35.861866 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:35.861835 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf225e37-88c0-4f9c-95f2-d3138e251a5d-util\") pod \"cf225e37-88c0-4f9c-95f2-d3138e251a5d\" (UID: \"cf225e37-88c0-4f9c-95f2-d3138e251a5d\") " Sep 10 15:46:35.862002 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:35.861891 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf225e37-88c0-4f9c-95f2-d3138e251a5d-bundle\") pod \"cf225e37-88c0-4f9c-95f2-d3138e251a5d\" (UID: \"cf225e37-88c0-4f9c-95f2-d3138e251a5d\") " Sep 10 15:46:35.862002 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:35.861927 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-824gn\" (UniqueName: \"kubernetes.io/projected/cf225e37-88c0-4f9c-95f2-d3138e251a5d-kube-api-access-824gn\") pod \"cf225e37-88c0-4f9c-95f2-d3138e251a5d\" (UID: \"cf225e37-88c0-4f9c-95f2-d3138e251a5d\") " Sep 10 15:46:35.862605 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:35.862580 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf225e37-88c0-4f9c-95f2-d3138e251a5d-bundle" (OuterVolumeSpecName: "bundle") pod "cf225e37-88c0-4f9c-95f2-d3138e251a5d" (UID: "cf225e37-88c0-4f9c-95f2-d3138e251a5d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:46:35.864062 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:35.864036 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf225e37-88c0-4f9c-95f2-d3138e251a5d-kube-api-access-824gn" (OuterVolumeSpecName: "kube-api-access-824gn") pod "cf225e37-88c0-4f9c-95f2-d3138e251a5d" (UID: "cf225e37-88c0-4f9c-95f2-d3138e251a5d"). InnerVolumeSpecName "kube-api-access-824gn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:46:35.866971 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:35.866925 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf225e37-88c0-4f9c-95f2-d3138e251a5d-util" (OuterVolumeSpecName: "util") pod "cf225e37-88c0-4f9c-95f2-d3138e251a5d" (UID: "cf225e37-88c0-4f9c-95f2-d3138e251a5d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:46:35.962826 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:35.962798 2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf225e37-88c0-4f9c-95f2-d3138e251a5d-bundle\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:46:35.962826 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:35.962820 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-824gn\" (UniqueName: \"kubernetes.io/projected/cf225e37-88c0-4f9c-95f2-d3138e251a5d-kube-api-access-824gn\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:46:35.962826 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:35.962829 2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf225e37-88c0-4f9c-95f2-d3138e251a5d-util\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:46:36.622254 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:36.622220 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dtm2h" Sep 10 15:46:36.622415 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:36.622247 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540dtm2h" event={"ID":"cf225e37-88c0-4f9c-95f2-d3138e251a5d","Type":"ContainerDied","Data":"0981c99cca5ecc9cac482712f3ab73636914593e3038b5c7955aab3d73c3664c"} Sep 10 15:46:36.622415 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:36.622300 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0981c99cca5ecc9cac482712f3ab73636914593e3038b5c7955aab3d73c3664c" Sep 10 15:46:36.696920 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:36.696892 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:46:41.453127 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:41.453095 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:46:42.229349 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:42.229316 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:46:43.514862 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:43.514832 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6hpmd"] Sep 10 15:46:43.515228 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:43.515110 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf225e37-88c0-4f9c-95f2-d3138e251a5d" containerName="util" Sep 10 15:46:43.515228 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:43.515121 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf225e37-88c0-4f9c-95f2-d3138e251a5d" containerName="util" Sep 10 15:46:43.515228 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:43.515131 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf225e37-88c0-4f9c-95f2-d3138e251a5d" containerName="pull" Sep 10 15:46:43.515228 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:43.515136 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf225e37-88c0-4f9c-95f2-d3138e251a5d" containerName="pull" Sep 10 15:46:43.515228 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:43.515152 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf225e37-88c0-4f9c-95f2-d3138e251a5d" containerName="extract" Sep 10 15:46:43.515228 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:43.515158 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf225e37-88c0-4f9c-95f2-d3138e251a5d" containerName="extract" Sep 10 15:46:43.515228 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:43.515225 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf225e37-88c0-4f9c-95f2-d3138e251a5d" containerName="extract" Sep 10 15:46:43.518839 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:43.518824 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6hpmd" Sep 10 15:46:43.522865 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:43.522846 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-v2p72\"" Sep 10 15:46:43.523241 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:43.523227 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 10 15:46:43.523519 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:43.523508 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 10 15:46:43.541982 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:43.541963 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6hpmd"] Sep 10 15:46:43.617759 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:43.617733 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d27db943-3cfb-40c4-a4e5-4c0ba6d06f2d-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-6hpmd\" (UID: \"d27db943-3cfb-40c4-a4e5-4c0ba6d06f2d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6hpmd" Sep 10 15:46:43.617861 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:43.617779 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rcjh\" (UniqueName: \"kubernetes.io/projected/d27db943-3cfb-40c4-a4e5-4c0ba6d06f2d-kube-api-access-8rcjh\") pod \"cert-manager-operator-controller-manager-55c444bf86-6hpmd\" (UID: \"d27db943-3cfb-40c4-a4e5-4c0ba6d06f2d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6hpmd" Sep 10 15:46:43.718329 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:43.718304 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d27db943-3cfb-40c4-a4e5-4c0ba6d06f2d-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-6hpmd\" (UID: \"d27db943-3cfb-40c4-a4e5-4c0ba6d06f2d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6hpmd" Sep 10 15:46:43.718440 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:43.718349 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rcjh\" (UniqueName: \"kubernetes.io/projected/d27db943-3cfb-40c4-a4e5-4c0ba6d06f2d-kube-api-access-8rcjh\") pod \"cert-manager-operator-controller-manager-55c444bf86-6hpmd\" (UID: \"d27db943-3cfb-40c4-a4e5-4c0ba6d06f2d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6hpmd" Sep 10 15:46:43.718734 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:43.718716 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d27db943-3cfb-40c4-a4e5-4c0ba6d06f2d-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-6hpmd\" (UID: \"d27db943-3cfb-40c4-a4e5-4c0ba6d06f2d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6hpmd" Sep 10 15:46:43.727530 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:43.727505 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rcjh\" (UniqueName: \"kubernetes.io/projected/d27db943-3cfb-40c4-a4e5-4c0ba6d06f2d-kube-api-access-8rcjh\") pod \"cert-manager-operator-controller-manager-55c444bf86-6hpmd\" (UID: \"d27db943-3cfb-40c4-a4e5-4c0ba6d06f2d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6hpmd" Sep 10 15:46:43.827640 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:43.827582 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6hpmd" Sep 10 15:46:44.106789 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:44.106764 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6hpmd"] Sep 10 15:46:44.109583 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:46:44.109553 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd27db943_3cfb_40c4_a4e5_4c0ba6d06f2d.slice/crio-305665577adb3fd44d54729b6fd1b3828990631abcaccdac79423a6dad331f5f WatchSource:0}: Error finding container 305665577adb3fd44d54729b6fd1b3828990631abcaccdac79423a6dad331f5f: Status 404 returned error can't find the container with id 305665577adb3fd44d54729b6fd1b3828990631abcaccdac79423a6dad331f5f Sep 10 15:46:44.653604 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:44.653565 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6hpmd" event={"ID":"d27db943-3cfb-40c4-a4e5-4c0ba6d06f2d","Type":"ContainerStarted","Data":"305665577adb3fd44d54729b6fd1b3828990631abcaccdac79423a6dad331f5f"} Sep 10 15:46:45.061181 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:45.061101 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:46:45.161065 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:45.161038 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:46:46.665160 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:46.665123 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6hpmd" event={"ID":"d27db943-3cfb-40c4-a4e5-4c0ba6d06f2d","Type":"ContainerStarted","Data":"cbb8fa3c6d88a3ca734833e1ea79b1291a7a84973b4b0057a045d6810142ab02"} Sep 10 15:46:46.686669 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:46.686611 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-6hpmd" podStartSLOduration=1.693388518 podStartE2EDuration="3.686591545s" totalImagesPullingTime="1.993203027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:46:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:46:46.68469584 +0000 UTC m=+283.659955817" watchObservedRunningTime="2026-09-10 15:46:46.686591545 +0000 UTC m=+283.661851502" Sep 10 15:46:46.785216 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:46.785185 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:46:46.883291 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:46.883259 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:46:46.906925 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:46.906890 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-tr7zk"] Sep 10 15:46:46.909080 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:46.909061 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-tr7zk" Sep 10 15:46:46.910957 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:46.910935 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 10 15:46:46.911051 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:46.911036 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 10 15:46:46.921872 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:46.921825 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-tr7zk"] Sep 10 15:46:47.046502 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:47.046469 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brbhb\" (UniqueName: \"kubernetes.io/projected/8f941db6-d480-466b-86b8-e1a584b2b3d9-kube-api-access-brbhb\") pod \"cert-manager-webhook-6486ffd997-tr7zk\" (UID: \"8f941db6-d480-466b-86b8-e1a584b2b3d9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-tr7zk" Sep 10 15:46:47.046658 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:47.046526 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f941db6-d480-466b-86b8-e1a584b2b3d9-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-tr7zk\" (UID: \"8f941db6-d480-466b-86b8-e1a584b2b3d9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-tr7zk" Sep 10 15:46:47.060398 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:47.060353 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:46:47.147725 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:47.147698 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brbhb\" (UniqueName: \"kubernetes.io/projected/8f941db6-d480-466b-86b8-e1a584b2b3d9-kube-api-access-brbhb\") pod \"cert-manager-webhook-6486ffd997-tr7zk\" (UID: \"8f941db6-d480-466b-86b8-e1a584b2b3d9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-tr7zk" Sep 10 15:46:47.147879 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:47.147771 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f941db6-d480-466b-86b8-e1a584b2b3d9-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-tr7zk\" (UID: \"8f941db6-d480-466b-86b8-e1a584b2b3d9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-tr7zk" Sep 10 15:46:47.166733 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:47.166710 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8f941db6-d480-466b-86b8-e1a584b2b3d9-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-tr7zk\" (UID: \"8f941db6-d480-466b-86b8-e1a584b2b3d9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-tr7zk" Sep 10 15:46:47.175086 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:47.175032 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brbhb\" (UniqueName: \"kubernetes.io/projected/8f941db6-d480-466b-86b8-e1a584b2b3d9-kube-api-access-brbhb\") pod \"cert-manager-webhook-6486ffd997-tr7zk\" (UID: \"8f941db6-d480-466b-86b8-e1a584b2b3d9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-tr7zk" Sep 10 15:46:47.195460 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:47.195437 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:46:47.232034 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:47.232008 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-tr7zk" Sep 10 15:46:47.520072 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:47.519952 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-tr7zk"] Sep 10 15:46:47.522616 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:46:47.522592 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f941db6_d480_466b_86b8_e1a584b2b3d9.slice/crio-a484fb83a2a506b25741492d584790ba5ba5126b423ea01c8fb3ad8c56c494b5 WatchSource:0}: Error finding container a484fb83a2a506b25741492d584790ba5ba5126b423ea01c8fb3ad8c56c494b5: Status 404 returned error can't find the container with id a484fb83a2a506b25741492d584790ba5ba5126b423ea01c8fb3ad8c56c494b5 Sep 10 15:46:47.670418 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:47.670362 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-tr7zk" event={"ID":"8f941db6-d480-466b-86b8-e1a584b2b3d9","Type":"ContainerStarted","Data":"a484fb83a2a506b25741492d584790ba5ba5126b423ea01c8fb3ad8c56c494b5"} Sep 10 15:46:47.835171 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:47.835100 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f89l7x"] Sep 10 15:46:47.838012 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:47.837996 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f89l7x" Sep 10 15:46:47.840046 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:47.840029 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 10 15:46:47.840672 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:47.840657 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 10 15:46:47.840739 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:47.840660 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-7k9nf\"" Sep 10 15:46:47.848101 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:47.848081 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f89l7x"] Sep 10 15:46:47.955048 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:47.955020 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a15504b-5438-4498-a6dd-162743981b05-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f89l7x\" (UID: \"0a15504b-5438-4498-a6dd-162743981b05\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f89l7x" Sep 10 15:46:47.955171 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:47.955063 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a15504b-5438-4498-a6dd-162743981b05-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f89l7x\" (UID: \"0a15504b-5438-4498-a6dd-162743981b05\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f89l7x" Sep 10 15:46:47.955212 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:47.955169 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx8xx\" (UniqueName: \"kubernetes.io/projected/0a15504b-5438-4498-a6dd-162743981b05-kube-api-access-hx8xx\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f89l7x\" (UID: \"0a15504b-5438-4498-a6dd-162743981b05\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f89l7x" Sep 10 15:46:48.056151 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:48.056115 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx8xx\" (UniqueName: \"kubernetes.io/projected/0a15504b-5438-4498-a6dd-162743981b05-kube-api-access-hx8xx\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f89l7x\" (UID: \"0a15504b-5438-4498-a6dd-162743981b05\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f89l7x" Sep 10 15:46:48.056290 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:48.056161 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a15504b-5438-4498-a6dd-162743981b05-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f89l7x\" (UID: \"0a15504b-5438-4498-a6dd-162743981b05\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f89l7x" Sep 10 15:46:48.056290 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:48.056204 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a15504b-5438-4498-a6dd-162743981b05-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f89l7x\" (UID: \"0a15504b-5438-4498-a6dd-162743981b05\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f89l7x" Sep 10 15:46:48.056556 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:48.056537 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a15504b-5438-4498-a6dd-162743981b05-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f89l7x\" (UID: \"0a15504b-5438-4498-a6dd-162743981b05\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f89l7x" Sep 10 15:46:48.056628 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:48.056570 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a15504b-5438-4498-a6dd-162743981b05-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f89l7x\" (UID: \"0a15504b-5438-4498-a6dd-162743981b05\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f89l7x" Sep 10 15:46:48.068977 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:48.068946 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx8xx\" (UniqueName: \"kubernetes.io/projected/0a15504b-5438-4498-a6dd-162743981b05-kube-api-access-hx8xx\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f89l7x\" (UID: \"0a15504b-5438-4498-a6dd-162743981b05\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f89l7x" Sep 10 15:46:48.148269 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:48.148245 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f89l7x" Sep 10 15:46:48.420193 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:48.420128 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f89l7x"] Sep 10 15:46:48.422488 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:46:48.422448 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a15504b_5438_4498_a6dd_162743981b05.slice/crio-94cf8bc0b3dfb6adaa808f38b275677fd64dfa8eeb940c268d21b479835e716a WatchSource:0}: Error finding container 94cf8bc0b3dfb6adaa808f38b275677fd64dfa8eeb940c268d21b479835e716a: Status 404 returned error can't find the container with id 94cf8bc0b3dfb6adaa808f38b275677fd64dfa8eeb940c268d21b479835e716a Sep 10 15:46:48.679049 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:48.678965 2583 generic.go:356] "Generic (PLEG): container finished" podID="0a15504b-5438-4498-a6dd-162743981b05" containerID="99fed59fc4f601b99bd2ee22630ac66e14dae3f3fa713df5dc2b017461edae2b" exitCode=0 Sep 10 15:46:48.679792 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:48.679147 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f89l7x" event={"ID":"0a15504b-5438-4498-a6dd-162743981b05","Type":"ContainerDied","Data":"99fed59fc4f601b99bd2ee22630ac66e14dae3f3fa713df5dc2b017461edae2b"} Sep 10 15:46:48.679792 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:48.679693 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f89l7x" event={"ID":"0a15504b-5438-4498-a6dd-162743981b05","Type":"ContainerStarted","Data":"94cf8bc0b3dfb6adaa808f38b275677fd64dfa8eeb940c268d21b479835e716a"} Sep 10 15:46:49.018694 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:49.018614 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:46:49.223206 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:49.223171 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:46:49.780197 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:49.780163 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-k8974"] Sep 10 15:46:49.782543 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:49.782521 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-k8974" Sep 10 15:46:49.784494 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:49.784471 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-gjsbn\"" Sep 10 15:46:49.794097 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:49.794074 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-k8974"] Sep 10 15:46:49.972561 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:49.972526 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/25cda49d-80ef-4f7e-82fe-e117d7d3f9c7-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-k8974\" (UID: \"25cda49d-80ef-4f7e-82fe-e117d7d3f9c7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-k8974" Sep 10 15:46:49.972730 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:49.972576 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdxxk\" (UniqueName: \"kubernetes.io/projected/25cda49d-80ef-4f7e-82fe-e117d7d3f9c7-kube-api-access-hdxxk\") pod \"cert-manager-cainjector-559cdd8c87-k8974\" (UID: \"25cda49d-80ef-4f7e-82fe-e117d7d3f9c7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-k8974" Sep 10 15:46:50.073889 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:50.073805 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/25cda49d-80ef-4f7e-82fe-e117d7d3f9c7-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-k8974\" (UID: \"25cda49d-80ef-4f7e-82fe-e117d7d3f9c7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-k8974" Sep 10 15:46:50.073889 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:50.073863 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdxxk\" (UniqueName: \"kubernetes.io/projected/25cda49d-80ef-4f7e-82fe-e117d7d3f9c7-kube-api-access-hdxxk\") pod \"cert-manager-cainjector-559cdd8c87-k8974\" (UID: \"25cda49d-80ef-4f7e-82fe-e117d7d3f9c7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-k8974" Sep 10 15:46:50.087555 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:50.087475 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/25cda49d-80ef-4f7e-82fe-e117d7d3f9c7-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-k8974\" (UID: \"25cda49d-80ef-4f7e-82fe-e117d7d3f9c7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-k8974" Sep 10 15:46:50.087715 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:50.087634 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdxxk\" (UniqueName: \"kubernetes.io/projected/25cda49d-80ef-4f7e-82fe-e117d7d3f9c7-kube-api-access-hdxxk\") pod \"cert-manager-cainjector-559cdd8c87-k8974\" (UID: \"25cda49d-80ef-4f7e-82fe-e117d7d3f9c7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-k8974" Sep 10 15:46:50.092762 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:50.092733 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-k8974" Sep 10 15:46:51.361759 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:51.361733 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-k8974"] Sep 10 15:46:51.364136 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:46:51.364114 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod25cda49d_80ef_4f7e_82fe_e117d7d3f9c7.slice/crio-cc72af1afb556095d5ac064ccff54ee1ffc200f7af02ef81814f50268d81803a WatchSource:0}: Error finding container cc72af1afb556095d5ac064ccff54ee1ffc200f7af02ef81814f50268d81803a: Status 404 returned error can't find the container with id cc72af1afb556095d5ac064ccff54ee1ffc200f7af02ef81814f50268d81803a Sep 10 15:46:51.695034 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:51.694998 2583 generic.go:356] "Generic (PLEG): container finished" podID="0a15504b-5438-4498-a6dd-162743981b05" containerID="2667bb6a45b4f6821d76a9633192ee6fc08bb0f4421802b0f96a6322dc560df3" exitCode=0 Sep 10 15:46:51.695212 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:51.695080 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f89l7x" event={"ID":"0a15504b-5438-4498-a6dd-162743981b05","Type":"ContainerDied","Data":"2667bb6a45b4f6821d76a9633192ee6fc08bb0f4421802b0f96a6322dc560df3"} Sep 10 15:46:51.696827 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:51.696804 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-k8974" event={"ID":"25cda49d-80ef-4f7e-82fe-e117d7d3f9c7","Type":"ContainerStarted","Data":"ebd757ec34cdebb06e9771508bfcbf2da7153689f6611af265e1aad7e811b5ce"} Sep 10 15:46:51.696930 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:51.696843 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-k8974" event={"ID":"25cda49d-80ef-4f7e-82fe-e117d7d3f9c7","Type":"ContainerStarted","Data":"cc72af1afb556095d5ac064ccff54ee1ffc200f7af02ef81814f50268d81803a"} Sep 10 15:46:51.698546 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:51.698524 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-tr7zk" event={"ID":"8f941db6-d480-466b-86b8-e1a584b2b3d9","Type":"ContainerStarted","Data":"1d21ff403cbf240623320f3ac120b4302aba045fca3950a7e13d64c451a8a081"} Sep 10 15:46:51.699016 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:51.698995 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-tr7zk" Sep 10 15:46:51.730215 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:51.730173 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-k8974" podStartSLOduration=2.7301607 podStartE2EDuration="2.7301607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:46:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:46:51.728524197 +0000 UTC m=+288.703784150" watchObservedRunningTime="2026-09-10 15:46:51.7301607 +0000 UTC m=+288.705420655" Sep 10 15:46:51.744130 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:51.744077 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-tr7zk" podStartSLOduration=2.165559553 podStartE2EDuration="5.744058951s" totalImagesPullingTime="3.578499398s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:46:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:46:51.743287906 +0000 UTC m=+288.718547861" watchObservedRunningTime="2026-09-10 15:46:51.744058951 +0000 UTC m=+288.719318906" Sep 10 15:46:51.854957 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:51.854929 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:46:51.927988 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:51.927965 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:46:52.704543 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:52.704508 2583 generic.go:356] "Generic (PLEG): container finished" podID="0a15504b-5438-4498-a6dd-162743981b05" containerID="62e1434212f7e606a470055bf1c6e7ebf8d6f045cc5876a5129f4ff6c7762286" exitCode=0 Sep 10 15:46:52.704879 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:52.704584 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f89l7x" event={"ID":"0a15504b-5438-4498-a6dd-162743981b05","Type":"ContainerDied","Data":"62e1434212f7e606a470055bf1c6e7ebf8d6f045cc5876a5129f4ff6c7762286"} Sep 10 15:46:53.835945 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:53.835921 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f89l7x" Sep 10 15:46:53.903461 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:53.903433 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a15504b-5438-4498-a6dd-162743981b05-util\") pod \"0a15504b-5438-4498-a6dd-162743981b05\" (UID: \"0a15504b-5438-4498-a6dd-162743981b05\") " Sep 10 15:46:53.903566 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:53.903474 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a15504b-5438-4498-a6dd-162743981b05-bundle\") pod \"0a15504b-5438-4498-a6dd-162743981b05\" (UID: \"0a15504b-5438-4498-a6dd-162743981b05\") " Sep 10 15:46:53.903566 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:53.903515 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hx8xx\" (UniqueName: \"kubernetes.io/projected/0a15504b-5438-4498-a6dd-162743981b05-kube-api-access-hx8xx\") pod \"0a15504b-5438-4498-a6dd-162743981b05\" (UID: \"0a15504b-5438-4498-a6dd-162743981b05\") " Sep 10 15:46:53.903890 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:53.903862 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a15504b-5438-4498-a6dd-162743981b05-bundle" (OuterVolumeSpecName: "bundle") pod "0a15504b-5438-4498-a6dd-162743981b05" (UID: "0a15504b-5438-4498-a6dd-162743981b05"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:46:53.905590 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:53.905568 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a15504b-5438-4498-a6dd-162743981b05-kube-api-access-hx8xx" (OuterVolumeSpecName: "kube-api-access-hx8xx") pod "0a15504b-5438-4498-a6dd-162743981b05" (UID: "0a15504b-5438-4498-a6dd-162743981b05"). InnerVolumeSpecName "kube-api-access-hx8xx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:46:53.910165 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:53.910120 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a15504b-5438-4498-a6dd-162743981b05-util" (OuterVolumeSpecName: "util") pod "0a15504b-5438-4498-a6dd-162743981b05" (UID: "0a15504b-5438-4498-a6dd-162743981b05"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:46:54.004367 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:54.004305 2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0a15504b-5438-4498-a6dd-162743981b05-util\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:46:54.004367 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:54.004331 2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0a15504b-5438-4498-a6dd-162743981b05-bundle\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:46:54.004367 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:54.004345 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx8xx\" (UniqueName: \"kubernetes.io/projected/0a15504b-5438-4498-a6dd-162743981b05-kube-api-access-hx8xx\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:46:54.715126 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:54.715087 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f89l7x" event={"ID":"0a15504b-5438-4498-a6dd-162743981b05","Type":"ContainerDied","Data":"94cf8bc0b3dfb6adaa808f38b275677fd64dfa8eeb940c268d21b479835e716a"} Sep 10 15:46:54.715292 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:54.715131 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f89l7x" Sep 10 15:46:54.715292 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:54.715138 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94cf8bc0b3dfb6adaa808f38b275677fd64dfa8eeb940c268d21b479835e716a" Sep 10 15:46:57.385203 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:57.385171 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:46:57.707145 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:57.707077 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-tr7zk" Sep 10 15:46:58.283338 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:58.283304 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:46:58.381716 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:46:58.381685 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:00.274701 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:00.274667 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tktqk"] Sep 10 15:47:00.275058 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:00.274986 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a15504b-5438-4498-a6dd-162743981b05" containerName="util" Sep 10 15:47:00.275058 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:00.274998 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a15504b-5438-4498-a6dd-162743981b05" containerName="util" Sep 10 15:47:00.275058 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:00.275008 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a15504b-5438-4498-a6dd-162743981b05" containerName="pull" Sep 10 15:47:00.275058 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:00.275013 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a15504b-5438-4498-a6dd-162743981b05" containerName="pull" Sep 10 15:47:00.275058 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:00.275018 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a15504b-5438-4498-a6dd-162743981b05" containerName="extract" Sep 10 15:47:00.275058 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:00.275025 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a15504b-5438-4498-a6dd-162743981b05" containerName="extract" Sep 10 15:47:00.275247 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:00.275076 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a15504b-5438-4498-a6dd-162743981b05" containerName="extract" Sep 10 15:47:00.278500 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:00.278483 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tktqk" Sep 10 15:47:00.280320 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:00.280302 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-jk74f\"" Sep 10 15:47:00.281016 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:00.280999 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 10 15:47:00.281090 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:00.281025 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 10 15:47:00.287820 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:00.287800 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tktqk"] Sep 10 15:47:00.349614 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:00.349589 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/881c5d57-02e1-4ef1-89be-2c4b0a6a0b48-tmp\") pod \"openshift-lws-operator-fd8ccff4c-tktqk\" (UID: \"881c5d57-02e1-4ef1-89be-2c4b0a6a0b48\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tktqk" Sep 10 15:47:00.349704 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:00.349685 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmm4c\" (UniqueName: \"kubernetes.io/projected/881c5d57-02e1-4ef1-89be-2c4b0a6a0b48-kube-api-access-zmm4c\") pod \"openshift-lws-operator-fd8ccff4c-tktqk\" (UID: \"881c5d57-02e1-4ef1-89be-2c4b0a6a0b48\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tktqk" Sep 10 15:47:00.450870 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:00.450837 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmm4c\" (UniqueName: \"kubernetes.io/projected/881c5d57-02e1-4ef1-89be-2c4b0a6a0b48-kube-api-access-zmm4c\") pod \"openshift-lws-operator-fd8ccff4c-tktqk\" (UID: \"881c5d57-02e1-4ef1-89be-2c4b0a6a0b48\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tktqk" Sep 10 15:47:00.450983 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:00.450900 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/881c5d57-02e1-4ef1-89be-2c4b0a6a0b48-tmp\") pod \"openshift-lws-operator-fd8ccff4c-tktqk\" (UID: \"881c5d57-02e1-4ef1-89be-2c4b0a6a0b48\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tktqk" Sep 10 15:47:00.451184 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:00.451165 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/881c5d57-02e1-4ef1-89be-2c4b0a6a0b48-tmp\") pod \"openshift-lws-operator-fd8ccff4c-tktqk\" (UID: \"881c5d57-02e1-4ef1-89be-2c4b0a6a0b48\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tktqk" Sep 10 15:47:00.459025 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:00.459005 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmm4c\" (UniqueName: \"kubernetes.io/projected/881c5d57-02e1-4ef1-89be-2c4b0a6a0b48-kube-api-access-zmm4c\") pod \"openshift-lws-operator-fd8ccff4c-tktqk\" (UID: \"881c5d57-02e1-4ef1-89be-2c4b0a6a0b48\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tktqk" Sep 10 15:47:00.587031 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:00.586978 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tktqk" Sep 10 15:47:00.856291 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:00.856160 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tktqk"] Sep 10 15:47:00.858856 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:47:00.858827 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod881c5d57_02e1_4ef1_89be_2c4b0a6a0b48.slice/crio-0c5b237f1aa2f63a97ea6d1d44502d3e1c3124edfe9b079653fb1769bd33bbc5 WatchSource:0}: Error finding container 0c5b237f1aa2f63a97ea6d1d44502d3e1c3124edfe9b079653fb1769bd33bbc5: Status 404 returned error can't find the container with id 0c5b237f1aa2f63a97ea6d1d44502d3e1c3124edfe9b079653fb1769bd33bbc5 Sep 10 15:47:01.747517 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:01.747483 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tktqk" event={"ID":"881c5d57-02e1-4ef1-89be-2c4b0a6a0b48","Type":"ContainerStarted","Data":"0c5b237f1aa2f63a97ea6d1d44502d3e1c3124edfe9b079653fb1769bd33bbc5"} Sep 10 15:47:03.454561 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:03.454535 2583 kubelet.go:1648] "Image garbage collection succeeded" Sep 10 15:47:03.757590 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:03.757556 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tktqk" event={"ID":"881c5d57-02e1-4ef1-89be-2c4b0a6a0b48","Type":"ContainerStarted","Data":"6f00f31ad6b6f8600f93c5b16ec0203abe13c67f71a5c796b924bbb364fc95b6"} Sep 10 15:47:03.897634 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:03.775996 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-tktqk" podStartSLOduration=1.403289894 podStartE2EDuration="3.775981877s" totalImagesPullingTime="2.372691983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:47:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:47:03.774678286 +0000 UTC m=+300.749938241" watchObservedRunningTime="2026-09-10 15:47:03.775981877 +0000 UTC m=+300.751241830" Sep 10 15:47:04.467250 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:04.467206 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:04.567067 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:04.567025 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:06.178088 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:06.178057 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:08.075709 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:08.075680 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmchlj"] Sep 10 15:47:08.078112 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:08.078093 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmchlj" Sep 10 15:47:08.080309 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:08.080288 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 10 15:47:08.080494 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:08.080474 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 10 15:47:08.081226 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:08.081181 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-7k9nf\"" Sep 10 15:47:08.088041 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:08.088022 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmchlj"] Sep 10 15:47:08.113571 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:08.113547 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/846d46ff-921c-4f7d-a683-20aaa4b6329a-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmchlj\" (UID: \"846d46ff-921c-4f7d-a683-20aaa4b6329a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmchlj" Sep 10 15:47:08.113670 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:08.113580 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2zrr\" (UniqueName: \"kubernetes.io/projected/846d46ff-921c-4f7d-a683-20aaa4b6329a-kube-api-access-p2zrr\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmchlj\" (UID: \"846d46ff-921c-4f7d-a683-20aaa4b6329a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmchlj" Sep 10 15:47:08.113670 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:08.113629 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/846d46ff-921c-4f7d-a683-20aaa4b6329a-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmchlj\" (UID: \"846d46ff-921c-4f7d-a683-20aaa4b6329a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmchlj" Sep 10 15:47:08.214286 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:08.214265 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/846d46ff-921c-4f7d-a683-20aaa4b6329a-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmchlj\" (UID: \"846d46ff-921c-4f7d-a683-20aaa4b6329a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmchlj" Sep 10 15:47:08.214425 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:08.214295 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2zrr\" (UniqueName: \"kubernetes.io/projected/846d46ff-921c-4f7d-a683-20aaa4b6329a-kube-api-access-p2zrr\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmchlj\" (UID: \"846d46ff-921c-4f7d-a683-20aaa4b6329a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmchlj" Sep 10 15:47:08.214425 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:08.214323 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/846d46ff-921c-4f7d-a683-20aaa4b6329a-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmchlj\" (UID: \"846d46ff-921c-4f7d-a683-20aaa4b6329a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmchlj" Sep 10 15:47:08.214707 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:08.214687 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/846d46ff-921c-4f7d-a683-20aaa4b6329a-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmchlj\" (UID: \"846d46ff-921c-4f7d-a683-20aaa4b6329a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmchlj" Sep 10 15:47:08.214776 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:08.214715 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/846d46ff-921c-4f7d-a683-20aaa4b6329a-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmchlj\" (UID: \"846d46ff-921c-4f7d-a683-20aaa4b6329a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmchlj" Sep 10 15:47:08.222304 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:08.222285 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2zrr\" (UniqueName: \"kubernetes.io/projected/846d46ff-921c-4f7d-a683-20aaa4b6329a-kube-api-access-p2zrr\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmchlj\" (UID: \"846d46ff-921c-4f7d-a683-20aaa4b6329a\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmchlj" Sep 10 15:47:08.388336 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:08.388315 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmchlj" Sep 10 15:47:08.657703 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:08.657680 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmchlj"] Sep 10 15:47:08.660024 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:47:08.659998 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod846d46ff_921c_4f7d_a683_20aaa4b6329a.slice/crio-d81b13f41b4d385b631534618d2c3d2afb4438fe7c9def7572ac90f06908a8aa WatchSource:0}: Error finding container d81b13f41b4d385b631534618d2c3d2afb4438fe7c9def7572ac90f06908a8aa: Status 404 returned error can't find the container with id d81b13f41b4d385b631534618d2c3d2afb4438fe7c9def7572ac90f06908a8aa Sep 10 15:47:08.780168 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:08.780139 2583 generic.go:356] "Generic (PLEG): container finished" podID="846d46ff-921c-4f7d-a683-20aaa4b6329a" containerID="33ad888860d6082a00e2120ab9fcc85ea6fd61794ea269cd099df5b092e20c01" exitCode=0 Sep 10 15:47:08.780279 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:08.780221 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmchlj" event={"ID":"846d46ff-921c-4f7d-a683-20aaa4b6329a","Type":"ContainerDied","Data":"33ad888860d6082a00e2120ab9fcc85ea6fd61794ea269cd099df5b092e20c01"} Sep 10 15:47:08.780279 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:08.780264 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmchlj" event={"ID":"846d46ff-921c-4f7d-a683-20aaa4b6329a","Type":"ContainerStarted","Data":"d81b13f41b4d385b631534618d2c3d2afb4438fe7c9def7572ac90f06908a8aa"} Sep 10 15:47:08.780695 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:08.780678 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 15:47:09.787374 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:09.787340 2583 generic.go:356] "Generic (PLEG): container finished" podID="846d46ff-921c-4f7d-a683-20aaa4b6329a" containerID="4f827a7d52f6f69ed706b9bf4fa6945faea10a0764340d610da35f5ae1dd8eff" exitCode=0 Sep 10 15:47:09.787763 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:09.787420 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmchlj" event={"ID":"846d46ff-921c-4f7d-a683-20aaa4b6329a","Type":"ContainerDied","Data":"4f827a7d52f6f69ed706b9bf4fa6945faea10a0764340d610da35f5ae1dd8eff"} Sep 10 15:47:09.836566 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:09.835724 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:09.896530 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:09.896496 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:10.793936 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:10.793904 2583 generic.go:356] "Generic (PLEG): container finished" podID="846d46ff-921c-4f7d-a683-20aaa4b6329a" containerID="22ecc373cae672ca9d359906a004761a9f98fefb7089aa748b47c332a63bd269" exitCode=0 Sep 10 15:47:10.794288 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:10.793985 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmchlj" event={"ID":"846d46ff-921c-4f7d-a683-20aaa4b6329a","Type":"ContainerDied","Data":"22ecc373cae672ca9d359906a004761a9f98fefb7089aa748b47c332a63bd269"} Sep 10 15:47:11.927408 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:11.927367 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmchlj" Sep 10 15:47:12.047221 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:12.047186 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p2zrr\" (UniqueName: \"kubernetes.io/projected/846d46ff-921c-4f7d-a683-20aaa4b6329a-kube-api-access-p2zrr\") pod \"846d46ff-921c-4f7d-a683-20aaa4b6329a\" (UID: \"846d46ff-921c-4f7d-a683-20aaa4b6329a\") " Sep 10 15:47:12.047420 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:12.047233 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/846d46ff-921c-4f7d-a683-20aaa4b6329a-util\") pod \"846d46ff-921c-4f7d-a683-20aaa4b6329a\" (UID: \"846d46ff-921c-4f7d-a683-20aaa4b6329a\") " Sep 10 15:47:12.047420 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:12.047271 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/846d46ff-921c-4f7d-a683-20aaa4b6329a-bundle\") pod \"846d46ff-921c-4f7d-a683-20aaa4b6329a\" (UID: \"846d46ff-921c-4f7d-a683-20aaa4b6329a\") " Sep 10 15:47:12.048083 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:12.048054 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/846d46ff-921c-4f7d-a683-20aaa4b6329a-bundle" (OuterVolumeSpecName: "bundle") pod "846d46ff-921c-4f7d-a683-20aaa4b6329a" (UID: "846d46ff-921c-4f7d-a683-20aaa4b6329a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:47:12.049340 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:12.049294 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/846d46ff-921c-4f7d-a683-20aaa4b6329a-kube-api-access-p2zrr" (OuterVolumeSpecName: "kube-api-access-p2zrr") pod "846d46ff-921c-4f7d-a683-20aaa4b6329a" (UID: "846d46ff-921c-4f7d-a683-20aaa4b6329a"). InnerVolumeSpecName "kube-api-access-p2zrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:47:12.053095 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:12.053061 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/846d46ff-921c-4f7d-a683-20aaa4b6329a-util" (OuterVolumeSpecName: "util") pod "846d46ff-921c-4f7d-a683-20aaa4b6329a" (UID: "846d46ff-921c-4f7d-a683-20aaa4b6329a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:47:12.148860 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:12.148827 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2zrr\" (UniqueName: \"kubernetes.io/projected/846d46ff-921c-4f7d-a683-20aaa4b6329a-kube-api-access-p2zrr\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:47:12.148860 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:12.148855 2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/846d46ff-921c-4f7d-a683-20aaa4b6329a-util\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:47:12.148860 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:12.148865 2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/846d46ff-921c-4f7d-a683-20aaa4b6329a-bundle\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:47:12.804221 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:12.804189 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmchlj" Sep 10 15:47:12.804221 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:12.804211 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afmchlj" event={"ID":"846d46ff-921c-4f7d-a683-20aaa4b6329a","Type":"ContainerDied","Data":"d81b13f41b4d385b631534618d2c3d2afb4438fe7c9def7572ac90f06908a8aa"} Sep 10 15:47:12.804467 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:12.804268 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d81b13f41b4d385b631534618d2c3d2afb4438fe7c9def7572ac90f06908a8aa" Sep 10 15:47:17.473918 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:17.473888 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gmcbv"] Sep 10 15:47:17.474303 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:17.474230 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="846d46ff-921c-4f7d-a683-20aaa4b6329a" containerName="util" Sep 10 15:47:17.474303 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:17.474243 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="846d46ff-921c-4f7d-a683-20aaa4b6329a" containerName="util" Sep 10 15:47:17.474303 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:17.474255 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="846d46ff-921c-4f7d-a683-20aaa4b6329a" containerName="extract" Sep 10 15:47:17.474303 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:17.474261 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="846d46ff-921c-4f7d-a683-20aaa4b6329a" containerName="extract" Sep 10 15:47:17.474303 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:17.474268 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="846d46ff-921c-4f7d-a683-20aaa4b6329a" containerName="pull" Sep 10 15:47:17.474303 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:17.474273 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="846d46ff-921c-4f7d-a683-20aaa4b6329a" containerName="pull" Sep 10 15:47:17.474506 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:17.474343 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="846d46ff-921c-4f7d-a683-20aaa4b6329a" containerName="extract" Sep 10 15:47:17.477036 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:17.477020 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gmcbv" Sep 10 15:47:17.479168 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:17.479136 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 10 15:47:17.479168 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:17.479144 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 10 15:47:17.479817 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:17.479799 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-7k9nf\"" Sep 10 15:47:17.492193 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:17.492155 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gmcbv"] Sep 10 15:47:17.594029 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:17.593999 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9b77ff21-e1a0-4150-857f-5670ffbcef0d-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gmcbv\" (UID: \"9b77ff21-e1a0-4150-857f-5670ffbcef0d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gmcbv" Sep 10 15:47:17.594185 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:17.594042 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4wnh\" (UniqueName: \"kubernetes.io/projected/9b77ff21-e1a0-4150-857f-5670ffbcef0d-kube-api-access-c4wnh\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gmcbv\" (UID: \"9b77ff21-e1a0-4150-857f-5670ffbcef0d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gmcbv" Sep 10 15:47:17.594185 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:17.594068 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9b77ff21-e1a0-4150-857f-5670ffbcef0d-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gmcbv\" (UID: \"9b77ff21-e1a0-4150-857f-5670ffbcef0d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gmcbv" Sep 10 15:47:17.694930 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:17.694900 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9b77ff21-e1a0-4150-857f-5670ffbcef0d-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gmcbv\" (UID: \"9b77ff21-e1a0-4150-857f-5670ffbcef0d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gmcbv" Sep 10 15:47:17.695071 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:17.694940 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4wnh\" (UniqueName: \"kubernetes.io/projected/9b77ff21-e1a0-4150-857f-5670ffbcef0d-kube-api-access-c4wnh\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gmcbv\" (UID: \"9b77ff21-e1a0-4150-857f-5670ffbcef0d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gmcbv" Sep 10 15:47:17.695071 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:17.694959 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9b77ff21-e1a0-4150-857f-5670ffbcef0d-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gmcbv\" (UID: \"9b77ff21-e1a0-4150-857f-5670ffbcef0d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gmcbv" Sep 10 15:47:17.695296 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:17.695273 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9b77ff21-e1a0-4150-857f-5670ffbcef0d-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gmcbv\" (UID: \"9b77ff21-e1a0-4150-857f-5670ffbcef0d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gmcbv" Sep 10 15:47:17.695332 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:17.695298 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9b77ff21-e1a0-4150-857f-5670ffbcef0d-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gmcbv\" (UID: \"9b77ff21-e1a0-4150-857f-5670ffbcef0d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gmcbv" Sep 10 15:47:17.704828 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:17.704803 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4wnh\" (UniqueName: \"kubernetes.io/projected/9b77ff21-e1a0-4150-857f-5670ffbcef0d-kube-api-access-c4wnh\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gmcbv\" (UID: \"9b77ff21-e1a0-4150-857f-5670ffbcef0d\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gmcbv" Sep 10 15:47:17.786812 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:17.786756 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gmcbv" Sep 10 15:47:18.057892 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:18.057867 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gmcbv"] Sep 10 15:47:18.060237 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:47:18.060212 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b77ff21_e1a0_4150_857f_5670ffbcef0d.slice/crio-87203eb30a87a0d7aaa4e3617086b1e8b56ba7b8b3376d2b912173f3a11af41b WatchSource:0}: Error finding container 87203eb30a87a0d7aaa4e3617086b1e8b56ba7b8b3376d2b912173f3a11af41b: Status 404 returned error can't find the container with id 87203eb30a87a0d7aaa4e3617086b1e8b56ba7b8b3376d2b912173f3a11af41b Sep 10 15:47:18.771633 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:18.771602 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:18.834303 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:18.834274 2583 generic.go:356] "Generic (PLEG): container finished" podID="9b77ff21-e1a0-4150-857f-5670ffbcef0d" containerID="1a3d556168d75e1515f7142fe75071857356114252f953db8e16f7fa0afbbbe7" exitCode=0 Sep 10 15:47:18.834436 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:18.834344 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gmcbv" event={"ID":"9b77ff21-e1a0-4150-857f-5670ffbcef0d","Type":"ContainerDied","Data":"1a3d556168d75e1515f7142fe75071857356114252f953db8e16f7fa0afbbbe7"} Sep 10 15:47:18.834436 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:18.834402 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gmcbv" event={"ID":"9b77ff21-e1a0-4150-857f-5670ffbcef0d","Type":"ContainerStarted","Data":"87203eb30a87a0d7aaa4e3617086b1e8b56ba7b8b3376d2b912173f3a11af41b"} Sep 10 15:47:18.860776 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:18.860750 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:19.398756 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:19.398732 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:19.709702 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:19.709672 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:19.755480 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:19.755453 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-bt7tx"] Sep 10 15:47:19.757771 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:19.757756 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-bt7tx" Sep 10 15:47:19.760213 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:19.760191 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 10 15:47:19.760493 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:19.760429 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 10 15:47:19.760493 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:19.760446 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 10 15:47:19.760675 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:19.760447 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 10 15:47:19.760729 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:19.760698 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-s4hf4\"" Sep 10 15:47:19.772610 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:19.772588 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-bt7tx"] Sep 10 15:47:19.805169 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:19.805137 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:19.812925 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:19.812905 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2e49b1f-82b5-49ee-b670-8948f7ea4997-tmp\") pod \"opendatahub-operator-controller-manager-8fbcf4bcf-bt7tx\" (UID: \"b2e49b1f-82b5-49ee-b670-8948f7ea4997\") " pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-bt7tx" Sep 10 15:47:19.812995 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:19.812932 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b2e49b1f-82b5-49ee-b670-8948f7ea4997-webhook-cert\") pod \"opendatahub-operator-controller-manager-8fbcf4bcf-bt7tx\" (UID: \"b2e49b1f-82b5-49ee-b670-8948f7ea4997\") " pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-bt7tx" Sep 10 15:47:19.812995 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:19.812954 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b2e49b1f-82b5-49ee-b670-8948f7ea4997-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8fbcf4bcf-bt7tx\" (UID: \"b2e49b1f-82b5-49ee-b670-8948f7ea4997\") " pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-bt7tx" Sep 10 15:47:19.812995 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:19.812990 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b2e49b1f-82b5-49ee-b670-8948f7ea4997-manifests\") pod \"opendatahub-operator-controller-manager-8fbcf4bcf-bt7tx\" (UID: \"b2e49b1f-82b5-49ee-b670-8948f7ea4997\") " pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-bt7tx" Sep 10 15:47:19.813091 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:19.813047 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swpsp\" (UniqueName: \"kubernetes.io/projected/b2e49b1f-82b5-49ee-b670-8948f7ea4997-kube-api-access-swpsp\") pod \"opendatahub-operator-controller-manager-8fbcf4bcf-bt7tx\" (UID: \"b2e49b1f-82b5-49ee-b670-8948f7ea4997\") " pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-bt7tx" Sep 10 15:47:19.841058 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:19.841031 2583 generic.go:356] "Generic (PLEG): container finished" podID="9b77ff21-e1a0-4150-857f-5670ffbcef0d" containerID="c8bb627de4e487e9e4f0e4ac17d84545f13801ff097b9373a22fe35869a72ede" exitCode=0 Sep 10 15:47:19.841157 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:19.841091 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gmcbv" event={"ID":"9b77ff21-e1a0-4150-857f-5670ffbcef0d","Type":"ContainerDied","Data":"c8bb627de4e487e9e4f0e4ac17d84545f13801ff097b9373a22fe35869a72ede"} Sep 10 15:47:19.912275 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:19.912247 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:19.914355 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:19.914325 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2e49b1f-82b5-49ee-b670-8948f7ea4997-tmp\") pod \"opendatahub-operator-controller-manager-8fbcf4bcf-bt7tx\" (UID: \"b2e49b1f-82b5-49ee-b670-8948f7ea4997\") " pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-bt7tx" Sep 10 15:47:19.914479 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:19.914360 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b2e49b1f-82b5-49ee-b670-8948f7ea4997-webhook-cert\") pod \"opendatahub-operator-controller-manager-8fbcf4bcf-bt7tx\" (UID: \"b2e49b1f-82b5-49ee-b670-8948f7ea4997\") " pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-bt7tx" Sep 10 15:47:19.914479 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:19.914402 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b2e49b1f-82b5-49ee-b670-8948f7ea4997-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8fbcf4bcf-bt7tx\" (UID: \"b2e49b1f-82b5-49ee-b670-8948f7ea4997\") " pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-bt7tx" Sep 10 15:47:19.914479 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:19.914436 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b2e49b1f-82b5-49ee-b670-8948f7ea4997-manifests\") pod \"opendatahub-operator-controller-manager-8fbcf4bcf-bt7tx\" (UID: \"b2e49b1f-82b5-49ee-b670-8948f7ea4997\") " pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-bt7tx" Sep 10 15:47:19.914643 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:19.914482 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swpsp\" (UniqueName: \"kubernetes.io/projected/b2e49b1f-82b5-49ee-b670-8948f7ea4997-kube-api-access-swpsp\") pod \"opendatahub-operator-controller-manager-8fbcf4bcf-bt7tx\" (UID: \"b2e49b1f-82b5-49ee-b670-8948f7ea4997\") " pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-bt7tx" Sep 10 15:47:19.915309 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:19.915285 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b2e49b1f-82b5-49ee-b670-8948f7ea4997-manifests\") pod \"opendatahub-operator-controller-manager-8fbcf4bcf-bt7tx\" (UID: \"b2e49b1f-82b5-49ee-b670-8948f7ea4997\") " pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-bt7tx" Sep 10 15:47:19.916892 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:19.916873 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b2e49b1f-82b5-49ee-b670-8948f7ea4997-tmp\") pod \"opendatahub-operator-controller-manager-8fbcf4bcf-bt7tx\" (UID: \"b2e49b1f-82b5-49ee-b670-8948f7ea4997\") " pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-bt7tx" Sep 10 15:47:19.917116 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:19.917089 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b2e49b1f-82b5-49ee-b670-8948f7ea4997-webhook-cert\") pod \"opendatahub-operator-controller-manager-8fbcf4bcf-bt7tx\" (UID: \"b2e49b1f-82b5-49ee-b670-8948f7ea4997\") " pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-bt7tx" Sep 10 15:47:19.917494 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:19.917474 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b2e49b1f-82b5-49ee-b670-8948f7ea4997-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8fbcf4bcf-bt7tx\" (UID: \"b2e49b1f-82b5-49ee-b670-8948f7ea4997\") " pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-bt7tx" Sep 10 15:47:19.926281 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:19.926259 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swpsp\" (UniqueName: \"kubernetes.io/projected/b2e49b1f-82b5-49ee-b670-8948f7ea4997-kube-api-access-swpsp\") pod \"opendatahub-operator-controller-manager-8fbcf4bcf-bt7tx\" (UID: \"b2e49b1f-82b5-49ee-b670-8948f7ea4997\") " pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-bt7tx" Sep 10 15:47:20.015300 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:20.015271 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:20.067878 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:20.067854 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-bt7tx" Sep 10 15:47:20.119773 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:20.119743 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:20.213309 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:20.213279 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:20.309801 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:20.309730 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:20.341124 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:20.341103 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-bt7tx"] Sep 10 15:47:20.342445 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:47:20.342425 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2e49b1f_82b5_49ee_b670_8948f7ea4997.slice/crio-b88d71286ad78db2e95fe4b2bfedcf8ee475d076c4cadf05b90eb71f22d0415d WatchSource:0}: Error finding container b88d71286ad78db2e95fe4b2bfedcf8ee475d076c4cadf05b90eb71f22d0415d: Status 404 returned error can't find the container with id b88d71286ad78db2e95fe4b2bfedcf8ee475d076c4cadf05b90eb71f22d0415d Sep 10 15:47:20.847773 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:20.847740 2583 generic.go:356] "Generic (PLEG): container finished" podID="9b77ff21-e1a0-4150-857f-5670ffbcef0d" containerID="7fa2b05176d8e233efa2c477410410b0ba0784bf724ee3c2e4e55fe51e73fc0e" exitCode=0 Sep 10 15:47:20.848175 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:20.847821 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gmcbv" event={"ID":"9b77ff21-e1a0-4150-857f-5670ffbcef0d","Type":"ContainerDied","Data":"7fa2b05176d8e233efa2c477410410b0ba0784bf724ee3c2e4e55fe51e73fc0e"} Sep 10 15:47:20.849135 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:20.849110 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-bt7tx" event={"ID":"b2e49b1f-82b5-49ee-b670-8948f7ea4997","Type":"ContainerStarted","Data":"b88d71286ad78db2e95fe4b2bfedcf8ee475d076c4cadf05b90eb71f22d0415d"} Sep 10 15:47:22.565626 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:22.565601 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gmcbv" Sep 10 15:47:22.638161 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:22.638131 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9b77ff21-e1a0-4150-857f-5670ffbcef0d-util\") pod \"9b77ff21-e1a0-4150-857f-5670ffbcef0d\" (UID: \"9b77ff21-e1a0-4150-857f-5670ffbcef0d\") " Sep 10 15:47:22.638338 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:22.638179 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9b77ff21-e1a0-4150-857f-5670ffbcef0d-bundle\") pod \"9b77ff21-e1a0-4150-857f-5670ffbcef0d\" (UID: \"9b77ff21-e1a0-4150-857f-5670ffbcef0d\") " Sep 10 15:47:22.638338 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:22.638210 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4wnh\" (UniqueName: \"kubernetes.io/projected/9b77ff21-e1a0-4150-857f-5670ffbcef0d-kube-api-access-c4wnh\") pod \"9b77ff21-e1a0-4150-857f-5670ffbcef0d\" (UID: \"9b77ff21-e1a0-4150-857f-5670ffbcef0d\") " Sep 10 15:47:22.639036 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:22.639009 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b77ff21-e1a0-4150-857f-5670ffbcef0d-bundle" (OuterVolumeSpecName: "bundle") pod "9b77ff21-e1a0-4150-857f-5670ffbcef0d" (UID: "9b77ff21-e1a0-4150-857f-5670ffbcef0d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:47:22.640763 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:22.640733 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b77ff21-e1a0-4150-857f-5670ffbcef0d-kube-api-access-c4wnh" (OuterVolumeSpecName: "kube-api-access-c4wnh") pod "9b77ff21-e1a0-4150-857f-5670ffbcef0d" (UID: "9b77ff21-e1a0-4150-857f-5670ffbcef0d"). InnerVolumeSpecName "kube-api-access-c4wnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:47:22.644799 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:22.644750 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b77ff21-e1a0-4150-857f-5670ffbcef0d-util" (OuterVolumeSpecName: "util") pod "9b77ff21-e1a0-4150-857f-5670ffbcef0d" (UID: "9b77ff21-e1a0-4150-857f-5670ffbcef0d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:47:22.739188 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:22.739157 2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9b77ff21-e1a0-4150-857f-5670ffbcef0d-util\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:47:22.739188 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:22.739190 2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9b77ff21-e1a0-4150-857f-5670ffbcef0d-bundle\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:47:22.739369 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:22.739204 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4wnh\" (UniqueName: \"kubernetes.io/projected/9b77ff21-e1a0-4150-857f-5670ffbcef0d-kube-api-access-c4wnh\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:47:22.862848 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:22.862780 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gmcbv" Sep 10 15:47:22.862848 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:22.862800 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b0gmcbv" event={"ID":"9b77ff21-e1a0-4150-857f-5670ffbcef0d","Type":"ContainerDied","Data":"87203eb30a87a0d7aaa4e3617086b1e8b56ba7b8b3376d2b912173f3a11af41b"} Sep 10 15:47:22.863023 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:22.862852 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87203eb30a87a0d7aaa4e3617086b1e8b56ba7b8b3376d2b912173f3a11af41b" Sep 10 15:47:23.869526 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:23.869451 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-bt7tx" event={"ID":"b2e49b1f-82b5-49ee-b670-8948f7ea4997","Type":"ContainerStarted","Data":"74b3f813549dede8a12e8d936fa4521a7b50ae5246d949eb78d0ab285d207038"} Sep 10 15:47:24.875606 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:24.875524 2583 generic.go:356] "Generic (PLEG): container finished" podID="b2e49b1f-82b5-49ee-b670-8948f7ea4997" containerID="74b3f813549dede8a12e8d936fa4521a7b50ae5246d949eb78d0ab285d207038" exitCode=0 Sep 10 15:47:24.875606 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:24.875574 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-bt7tx" event={"ID":"b2e49b1f-82b5-49ee-b670-8948f7ea4997","Type":"ContainerDied","Data":"74b3f813549dede8a12e8d936fa4521a7b50ae5246d949eb78d0ab285d207038"} Sep 10 15:47:25.882798 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:25.882751 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-bt7tx" event={"ID":"b2e49b1f-82b5-49ee-b670-8948f7ea4997","Type":"ContainerStarted","Data":"8cbfc61ab18ec586eb7ca4cfcaacfe42880e48348de231994805551d38ef4697"} Sep 10 15:47:25.883217 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:25.882830 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-bt7tx" Sep 10 15:47:25.904353 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:25.904300 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-bt7tx" podStartSLOduration=2.36970406 podStartE2EDuration="6.904281229s" totalImagesPullingTime="3.494375616s" totalInitContainerRuntime="1.040201553s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:47:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:47:25.902583878 +0000 UTC m=+322.877843832" watchObservedRunningTime="2026-09-10 15:47:25.904281229 +0000 UTC m=+322.879541183" Sep 10 15:47:30.217056 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:30.217020 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:30.312486 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:30.312458 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:34.304041 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:34.304004 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:34.411290 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:34.411257 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:36.890848 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:36.890814 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-bt7tx" Sep 10 15:47:38.369366 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:38.369333 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-f8dc47f5d-rwzg9"] Sep 10 15:47:38.369733 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:38.369671 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b77ff21-e1a0-4150-857f-5670ffbcef0d" containerName="util" Sep 10 15:47:38.369733 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:38.369683 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b77ff21-e1a0-4150-857f-5670ffbcef0d" containerName="util" Sep 10 15:47:38.369733 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:38.369706 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b77ff21-e1a0-4150-857f-5670ffbcef0d" containerName="pull" Sep 10 15:47:38.369733 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:38.369711 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b77ff21-e1a0-4150-857f-5670ffbcef0d" containerName="pull" Sep 10 15:47:38.369733 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:38.369719 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b77ff21-e1a0-4150-857f-5670ffbcef0d" containerName="extract" Sep 10 15:47:38.369733 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:38.369724 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b77ff21-e1a0-4150-857f-5670ffbcef0d" containerName="extract" Sep 10 15:47:38.369950 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:38.369781 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b77ff21-e1a0-4150-857f-5670ffbcef0d" containerName="extract" Sep 10 15:47:38.377234 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:38.377210 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-f8dc47f5d-rwzg9" Sep 10 15:47:38.379614 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:38.379596 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 10 15:47:38.380816 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:38.380591 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 10 15:47:38.380816 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:38.380627 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 10 15:47:38.380816 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:38.380627 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-4fq4m\"" Sep 10 15:47:38.383763 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:38.383741 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-f8dc47f5d-rwzg9"] Sep 10 15:47:38.473375 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:38.473347 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9fce4356-4a66-4635-b017-ec95c65229ae-manager-config\") pod \"lws-controller-manager-f8dc47f5d-rwzg9\" (UID: \"9fce4356-4a66-4635-b017-ec95c65229ae\") " pod="openshift-lws-operator/lws-controller-manager-f8dc47f5d-rwzg9" Sep 10 15:47:38.473515 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:38.473408 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fce4356-4a66-4635-b017-ec95c65229ae-cert\") pod \"lws-controller-manager-f8dc47f5d-rwzg9\" (UID: \"9fce4356-4a66-4635-b017-ec95c65229ae\") " pod="openshift-lws-operator/lws-controller-manager-f8dc47f5d-rwzg9" Sep 10 15:47:38.473515 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:38.473434 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlp44\" (UniqueName: \"kubernetes.io/projected/9fce4356-4a66-4635-b017-ec95c65229ae-kube-api-access-jlp44\") pod \"lws-controller-manager-f8dc47f5d-rwzg9\" (UID: \"9fce4356-4a66-4635-b017-ec95c65229ae\") " pod="openshift-lws-operator/lws-controller-manager-f8dc47f5d-rwzg9" Sep 10 15:47:38.473626 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:38.473563 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9fce4356-4a66-4635-b017-ec95c65229ae-metrics-cert\") pod \"lws-controller-manager-f8dc47f5d-rwzg9\" (UID: \"9fce4356-4a66-4635-b017-ec95c65229ae\") " pod="openshift-lws-operator/lws-controller-manager-f8dc47f5d-rwzg9" Sep 10 15:47:38.574676 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:38.574651 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fce4356-4a66-4635-b017-ec95c65229ae-cert\") pod \"lws-controller-manager-f8dc47f5d-rwzg9\" (UID: \"9fce4356-4a66-4635-b017-ec95c65229ae\") " pod="openshift-lws-operator/lws-controller-manager-f8dc47f5d-rwzg9" Sep 10 15:47:38.574676 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:38.574681 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlp44\" (UniqueName: \"kubernetes.io/projected/9fce4356-4a66-4635-b017-ec95c65229ae-kube-api-access-jlp44\") pod \"lws-controller-manager-f8dc47f5d-rwzg9\" (UID: \"9fce4356-4a66-4635-b017-ec95c65229ae\") " pod="openshift-lws-operator/lws-controller-manager-f8dc47f5d-rwzg9" Sep 10 15:47:38.574900 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:38.574781 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9fce4356-4a66-4635-b017-ec95c65229ae-metrics-cert\") pod \"lws-controller-manager-f8dc47f5d-rwzg9\" (UID: \"9fce4356-4a66-4635-b017-ec95c65229ae\") " pod="openshift-lws-operator/lws-controller-manager-f8dc47f5d-rwzg9" Sep 10 15:47:38.574900 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:38.574811 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9fce4356-4a66-4635-b017-ec95c65229ae-manager-config\") pod \"lws-controller-manager-f8dc47f5d-rwzg9\" (UID: \"9fce4356-4a66-4635-b017-ec95c65229ae\") " pod="openshift-lws-operator/lws-controller-manager-f8dc47f5d-rwzg9" Sep 10 15:47:38.575517 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:38.575494 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9fce4356-4a66-4635-b017-ec95c65229ae-manager-config\") pod \"lws-controller-manager-f8dc47f5d-rwzg9\" (UID: \"9fce4356-4a66-4635-b017-ec95c65229ae\") " pod="openshift-lws-operator/lws-controller-manager-f8dc47f5d-rwzg9" Sep 10 15:47:38.577221 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:38.577191 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fce4356-4a66-4635-b017-ec95c65229ae-cert\") pod \"lws-controller-manager-f8dc47f5d-rwzg9\" (UID: \"9fce4356-4a66-4635-b017-ec95c65229ae\") " pod="openshift-lws-operator/lws-controller-manager-f8dc47f5d-rwzg9" Sep 10 15:47:38.577404 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:38.577287 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9fce4356-4a66-4635-b017-ec95c65229ae-metrics-cert\") pod \"lws-controller-manager-f8dc47f5d-rwzg9\" (UID: \"9fce4356-4a66-4635-b017-ec95c65229ae\") " pod="openshift-lws-operator/lws-controller-manager-f8dc47f5d-rwzg9" Sep 10 15:47:38.592693 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:38.592666 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlp44\" (UniqueName: \"kubernetes.io/projected/9fce4356-4a66-4635-b017-ec95c65229ae-kube-api-access-jlp44\") pod \"lws-controller-manager-f8dc47f5d-rwzg9\" (UID: \"9fce4356-4a66-4635-b017-ec95c65229ae\") " pod="openshift-lws-operator/lws-controller-manager-f8dc47f5d-rwzg9" Sep 10 15:47:38.687971 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:38.687938 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-f8dc47f5d-rwzg9" Sep 10 15:47:38.697922 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:38.697893 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:38.805965 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:38.805934 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:38.993246 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:38.993213 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-f8dc47f5d-rwzg9"] Sep 10 15:47:38.995002 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:47:38.994967 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fce4356_4a66_4635_b017_ec95c65229ae.slice/crio-11e99002e30a222480534c20af15b48e389007f999db148e5becf7fca9605e19 WatchSource:0}: Error finding container 11e99002e30a222480534c20af15b48e389007f999db148e5becf7fca9605e19: Status 404 returned error can't find the container with id 11e99002e30a222480534c20af15b48e389007f999db148e5becf7fca9605e19 Sep 10 15:47:39.031654 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:39.031618 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:39.394347 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:39.394311 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:39.951992 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:39.951960 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-f8dc47f5d-rwzg9" event={"ID":"9fce4356-4a66-4635-b017-ec95c65229ae","Type":"ContainerStarted","Data":"11e99002e30a222480534c20af15b48e389007f999db148e5becf7fca9605e19"} Sep 10 15:47:40.354689 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:40.354657 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:40.455134 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:40.455086 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:41.415698 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:41.415667 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:42.756989 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:42.756958 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:42.829236 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:42.829214 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:42.967703 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:42.967674 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-f8dc47f5d-rwzg9" event={"ID":"9fce4356-4a66-4635-b017-ec95c65229ae","Type":"ContainerStarted","Data":"dc8ba6cd51e3af165af2c699204893d7b3e92c897cff5c0aea9d63c70d39a9c0"} Sep 10 15:47:42.967874 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:42.967804 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-f8dc47f5d-rwzg9" Sep 10 15:47:42.983904 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:42.983859 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-f8dc47f5d-rwzg9" podStartSLOduration=1.984161082 podStartE2EDuration="4.983846344s" totalImagesPullingTime="2.999685262s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:47:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:47:42.982570892 +0000 UTC m=+339.957830845" watchObservedRunningTime="2026-09-10 15:47:42.983846344 +0000 UTC m=+339.959106297" Sep 10 15:47:45.291056 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:45.291019 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:45.389544 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:45.389512 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:45.547972 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:45.547897 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2"] Sep 10 15:47:45.552404 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:45.552364 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2" Sep 10 15:47:45.554656 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:45.554634 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 10 15:47:45.554763 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:45.554665 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-dk6qc\"" Sep 10 15:47:45.562796 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:45.562776 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2"] Sep 10 15:47:45.575974 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:45.575949 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:45.718740 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:45.718713 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:45.732898 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:45.732869 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aebac9a-c55c-48af-95fa-4c1431657fe7-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2\" (UID: \"4aebac9a-c55c-48af-95fa-4c1431657fe7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2" Sep 10 15:47:45.733025 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:45.732902 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aebac9a-c55c-48af-95fa-4c1431657fe7-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2\" (UID: \"4aebac9a-c55c-48af-95fa-4c1431657fe7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2" Sep 10 15:47:45.733025 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:45.732963 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aebac9a-c55c-48af-95fa-4c1431657fe7-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2\" (UID: \"4aebac9a-c55c-48af-95fa-4c1431657fe7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2" Sep 10 15:47:45.733025 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:45.732995 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aebac9a-c55c-48af-95fa-4c1431657fe7-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2\" (UID: \"4aebac9a-c55c-48af-95fa-4c1431657fe7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2" Sep 10 15:47:45.733025 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:45.733014 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aebac9a-c55c-48af-95fa-4c1431657fe7-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2\" (UID: \"4aebac9a-c55c-48af-95fa-4c1431657fe7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2" Sep 10 15:47:45.733234 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:45.733029 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aebac9a-c55c-48af-95fa-4c1431657fe7-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2\" (UID: \"4aebac9a-c55c-48af-95fa-4c1431657fe7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2" Sep 10 15:47:45.733234 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:45.733122 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aebac9a-c55c-48af-95fa-4c1431657fe7-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2\" (UID: \"4aebac9a-c55c-48af-95fa-4c1431657fe7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2" Sep 10 15:47:45.733234 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:45.733154 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aebac9a-c55c-48af-95fa-4c1431657fe7-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2\" (UID: \"4aebac9a-c55c-48af-95fa-4c1431657fe7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2" Sep 10 15:47:45.733234 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:45.733215 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55jss\" (UniqueName: \"kubernetes.io/projected/4aebac9a-c55c-48af-95fa-4c1431657fe7-kube-api-access-55jss\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2\" (UID: \"4aebac9a-c55c-48af-95fa-4c1431657fe7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2" Sep 10 15:47:45.817419 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:45.817339 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:45.834515 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:45.834490 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aebac9a-c55c-48af-95fa-4c1431657fe7-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2\" (UID: \"4aebac9a-c55c-48af-95fa-4c1431657fe7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2" Sep 10 15:47:45.834609 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:45.834521 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aebac9a-c55c-48af-95fa-4c1431657fe7-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2\" (UID: \"4aebac9a-c55c-48af-95fa-4c1431657fe7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2" Sep 10 15:47:45.834609 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:45.834537 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aebac9a-c55c-48af-95fa-4c1431657fe7-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2\" (UID: \"4aebac9a-c55c-48af-95fa-4c1431657fe7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2" Sep 10 15:47:45.834609 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:45.834553 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aebac9a-c55c-48af-95fa-4c1431657fe7-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2\" (UID: \"4aebac9a-c55c-48af-95fa-4c1431657fe7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2" Sep 10 15:47:45.834609 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:45.834589 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aebac9a-c55c-48af-95fa-4c1431657fe7-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2\" (UID: \"4aebac9a-c55c-48af-95fa-4c1431657fe7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2" Sep 10 15:47:45.834809 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:45.834614 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aebac9a-c55c-48af-95fa-4c1431657fe7-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2\" (UID: \"4aebac9a-c55c-48af-95fa-4c1431657fe7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2" Sep 10 15:47:45.834809 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:45.834656 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55jss\" (UniqueName: \"kubernetes.io/projected/4aebac9a-c55c-48af-95fa-4c1431657fe7-kube-api-access-55jss\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2\" (UID: \"4aebac9a-c55c-48af-95fa-4c1431657fe7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2" Sep 10 15:47:45.834809 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:45.834714 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aebac9a-c55c-48af-95fa-4c1431657fe7-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2\" (UID: \"4aebac9a-c55c-48af-95fa-4c1431657fe7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2" Sep 10 15:47:45.834809 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:45.834737 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aebac9a-c55c-48af-95fa-4c1431657fe7-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2\" (UID: \"4aebac9a-c55c-48af-95fa-4c1431657fe7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2" Sep 10 15:47:45.835162 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:45.835139 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4aebac9a-c55c-48af-95fa-4c1431657fe7-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2\" (UID: \"4aebac9a-c55c-48af-95fa-4c1431657fe7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2" Sep 10 15:47:45.835264 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:45.835183 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aebac9a-c55c-48af-95fa-4c1431657fe7-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2\" (UID: \"4aebac9a-c55c-48af-95fa-4c1431657fe7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2" Sep 10 15:47:45.835313 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:45.835258 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4aebac9a-c55c-48af-95fa-4c1431657fe7-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2\" (UID: \"4aebac9a-c55c-48af-95fa-4c1431657fe7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2" Sep 10 15:47:45.835313 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:45.835276 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4aebac9a-c55c-48af-95fa-4c1431657fe7-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2\" (UID: \"4aebac9a-c55c-48af-95fa-4c1431657fe7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2" Sep 10 15:47:45.835413 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:45.835340 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4aebac9a-c55c-48af-95fa-4c1431657fe7-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2\" (UID: \"4aebac9a-c55c-48af-95fa-4c1431657fe7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2" Sep 10 15:47:45.837321 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:45.837296 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4aebac9a-c55c-48af-95fa-4c1431657fe7-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2\" (UID: \"4aebac9a-c55c-48af-95fa-4c1431657fe7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2" Sep 10 15:47:45.837644 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:45.837629 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4aebac9a-c55c-48af-95fa-4c1431657fe7-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2\" (UID: \"4aebac9a-c55c-48af-95fa-4c1431657fe7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2" Sep 10 15:47:45.842266 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:45.842243 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4aebac9a-c55c-48af-95fa-4c1431657fe7-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2\" (UID: \"4aebac9a-c55c-48af-95fa-4c1431657fe7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2" Sep 10 15:47:45.842532 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:45.842512 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-55jss\" (UniqueName: \"kubernetes.io/projected/4aebac9a-c55c-48af-95fa-4c1431657fe7-kube-api-access-55jss\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2\" (UID: \"4aebac9a-c55c-48af-95fa-4c1431657fe7\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2" Sep 10 15:47:45.864577 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:45.864554 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2" Sep 10 15:47:46.138605 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:46.138581 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2"] Sep 10 15:47:46.140145 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:47:46.140119 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aebac9a_c55c_48af_95fa_4c1431657fe7.slice/crio-3334fd22d3d864cc15298651320987ee688a7cfa49093bf8c07c47d031a17f66 WatchSource:0}: Error finding container 3334fd22d3d864cc15298651320987ee688a7cfa49093bf8c07c47d031a17f66: Status 404 returned error can't find the container with id 3334fd22d3d864cc15298651320987ee688a7cfa49093bf8c07c47d031a17f66 Sep 10 15:47:46.991503 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:46.991455 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2" event={"ID":"4aebac9a-c55c-48af-95fa-4c1431657fe7","Type":"ContainerStarted","Data":"3334fd22d3d864cc15298651320987ee688a7cfa49093bf8c07c47d031a17f66"} Sep 10 15:47:48.729932 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:48.729893 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 15:47:48.730297 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:48.729978 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 15:47:48.730297 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:48.730030 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 15:47:49.009521 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:49.009433 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2" event={"ID":"4aebac9a-c55c-48af-95fa-4c1431657fe7","Type":"ContainerStarted","Data":"37968f7d48bffde75b68d3749400ad6230eb3546512e93fe92f57ebfdd982cbe"} Sep 10 15:47:49.031102 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:49.031049 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2" podStartSLOduration=1.4435141470000001 podStartE2EDuration="4.03103175s" totalImagesPullingTime="2.587517603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:47:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:47:49.030015405 +0000 UTC m=+346.005275376" watchObservedRunningTime="2026-09-10 15:47:49.03103175 +0000 UTC m=+346.006291683" Sep 10 15:47:49.853026 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:49.852989 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-tlfmm"] Sep 10 15:47:49.857573 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:49.857556 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-tlfmm" Sep 10 15:47:49.859741 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:49.859720 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-2dhml\"" Sep 10 15:47:49.859819 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:49.859732 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 10 15:47:49.864866 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:49.864846 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2" Sep 10 15:47:49.870139 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:49.870116 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2" Sep 10 15:47:49.874726 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:49.874697 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-tlfmm"] Sep 10 15:47:49.937576 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:49.937517 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:49.970280 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:49.970253 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkvzd\" (UniqueName: \"kubernetes.io/projected/1c7f7317-43f6-43d7-b12d-769c7d0bb5e8-kube-api-access-rkvzd\") pod \"odh-model-controller-7998c67dc4-tlfmm\" (UID: \"1c7f7317-43f6-43d7-b12d-769c7d0bb5e8\") " pod="opendatahub/odh-model-controller-7998c67dc4-tlfmm" Sep 10 15:47:49.970451 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:49.970428 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c7f7317-43f6-43d7-b12d-769c7d0bb5e8-cert\") pod \"odh-model-controller-7998c67dc4-tlfmm\" (UID: \"1c7f7317-43f6-43d7-b12d-769c7d0bb5e8\") " pod="opendatahub/odh-model-controller-7998c67dc4-tlfmm" Sep 10 15:47:50.015820 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:50.015778 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2" Sep 10 15:47:50.017051 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:50.017026 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2" Sep 10 15:47:50.071613 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:50.071578 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c7f7317-43f6-43d7-b12d-769c7d0bb5e8-cert\") pod \"odh-model-controller-7998c67dc4-tlfmm\" (UID: \"1c7f7317-43f6-43d7-b12d-769c7d0bb5e8\") " pod="opendatahub/odh-model-controller-7998c67dc4-tlfmm" Sep 10 15:47:50.071794 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:50.071641 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkvzd\" (UniqueName: \"kubernetes.io/projected/1c7f7317-43f6-43d7-b12d-769c7d0bb5e8-kube-api-access-rkvzd\") pod \"odh-model-controller-7998c67dc4-tlfmm\" (UID: \"1c7f7317-43f6-43d7-b12d-769c7d0bb5e8\") " pod="opendatahub/odh-model-controller-7998c67dc4-tlfmm" Sep 10 15:47:50.071794 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:47:50.071750 2583 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 10 15:47:50.072123 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:47:50.072094 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c7f7317-43f6-43d7-b12d-769c7d0bb5e8-cert podName:1c7f7317-43f6-43d7-b12d-769c7d0bb5e8 nodeName:}" failed. No retries permitted until 2026-09-10 15:47:50.572069236 +0000 UTC m=+347.547329173 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/1c7f7317-43f6-43d7-b12d-769c7d0bb5e8-cert") pod "odh-model-controller-7998c67dc4-tlfmm" (UID: "1c7f7317-43f6-43d7-b12d-769c7d0bb5e8") : secret "odh-model-controller-webhook-cert" not found Sep 10 15:47:50.088219 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:50.088189 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkvzd\" (UniqueName: \"kubernetes.io/projected/1c7f7317-43f6-43d7-b12d-769c7d0bb5e8-kube-api-access-rkvzd\") pod \"odh-model-controller-7998c67dc4-tlfmm\" (UID: \"1c7f7317-43f6-43d7-b12d-769c7d0bb5e8\") " pod="opendatahub/odh-model-controller-7998c67dc4-tlfmm" Sep 10 15:47:50.146170 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:50.146140 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:50.310984 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:50.310942 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:50.551051 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:50.550976 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:50.577240 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:50.577212 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c7f7317-43f6-43d7-b12d-769c7d0bb5e8-cert\") pod \"odh-model-controller-7998c67dc4-tlfmm\" (UID: \"1c7f7317-43f6-43d7-b12d-769c7d0bb5e8\") " pod="opendatahub/odh-model-controller-7998c67dc4-tlfmm" Sep 10 15:47:50.579761 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:50.579735 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c7f7317-43f6-43d7-b12d-769c7d0bb5e8-cert\") pod \"odh-model-controller-7998c67dc4-tlfmm\" (UID: \"1c7f7317-43f6-43d7-b12d-769c7d0bb5e8\") " pod="opendatahub/odh-model-controller-7998c67dc4-tlfmm" Sep 10 15:47:50.654104 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:50.654074 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:50.768612 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:50.768580 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-tlfmm" Sep 10 15:47:51.041815 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:51.041788 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-tlfmm"] Sep 10 15:47:51.043444 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:47:51.043412 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c7f7317_43f6_43d7_b12d_769c7d0bb5e8.slice/crio-3ac44657b799a4aa423797b8922ae5dff86f616d959f0c1c1cad8e256b767260 WatchSource:0}: Error finding container 3ac44657b799a4aa423797b8922ae5dff86f616d959f0c1c1cad8e256b767260: Status 404 returned error can't find the container with id 3ac44657b799a4aa423797b8922ae5dff86f616d959f0c1c1cad8e256b767260 Sep 10 15:47:52.031227 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:52.031188 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-tlfmm" event={"ID":"1c7f7317-43f6-43d7-b12d-769c7d0bb5e8","Type":"ContainerStarted","Data":"3ac44657b799a4aa423797b8922ae5dff86f616d959f0c1c1cad8e256b767260"} Sep 10 15:47:53.975656 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:53.975628 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-f8dc47f5d-rwzg9" Sep 10 15:47:54.046932 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:54.046882 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-tlfmm" event={"ID":"1c7f7317-43f6-43d7-b12d-769c7d0bb5e8","Type":"ContainerStarted","Data":"03fc78bfa8ef052e22718068144533cfc3d825bce77ab14d6f693e4cb489299e"} Sep 10 15:47:54.047095 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:54.047079 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-tlfmm" Sep 10 15:47:54.069985 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:54.069928 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-tlfmm" podStartSLOduration=2.175595705 podStartE2EDuration="5.069909291s" totalImagesPullingTime="2.894313586s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:47:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:47:54.069497201 +0000 UTC m=+351.044757157" watchObservedRunningTime="2026-09-10 15:47:54.069909291 +0000 UTC m=+351.045169245" Sep 10 15:47:56.759230 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:56.759153 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-zdjj4"] Sep 10 15:47:56.762463 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:56.762444 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zdjj4" Sep 10 15:47:56.764557 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:56.764537 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 10 15:47:56.764645 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:56.764588 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 10 15:47:56.764701 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:56.764689 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 10 15:47:56.764814 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:56.764799 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-zfrwz\"" Sep 10 15:47:56.774075 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:56.774051 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-zdjj4"] Sep 10 15:47:56.832958 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:56.832927 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a33616ed-885f-4af2-9bea-eb1f9e9dcf3c-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-zdjj4\" (UID: \"a33616ed-885f-4af2-9bea-eb1f9e9dcf3c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zdjj4" Sep 10 15:47:56.833088 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:56.832977 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a33616ed-885f-4af2-9bea-eb1f9e9dcf3c-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-zdjj4\" (UID: \"a33616ed-885f-4af2-9bea-eb1f9e9dcf3c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zdjj4" Sep 10 15:47:56.833088 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:56.832996 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a33616ed-885f-4af2-9bea-eb1f9e9dcf3c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-zdjj4\" (UID: \"a33616ed-885f-4af2-9bea-eb1f9e9dcf3c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zdjj4" Sep 10 15:47:56.833088 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:56.833061 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-949js\" (UniqueName: \"kubernetes.io/projected/a33616ed-885f-4af2-9bea-eb1f9e9dcf3c-kube-api-access-949js\") pod \"istiod-openshift-gateway-5677ccb49b-zdjj4\" (UID: \"a33616ed-885f-4af2-9bea-eb1f9e9dcf3c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zdjj4" Sep 10 15:47:56.833203 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:56.833120 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a33616ed-885f-4af2-9bea-eb1f9e9dcf3c-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-zdjj4\" (UID: \"a33616ed-885f-4af2-9bea-eb1f9e9dcf3c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zdjj4" Sep 10 15:47:56.833203 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:56.833153 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a33616ed-885f-4af2-9bea-eb1f9e9dcf3c-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-zdjj4\" (UID: \"a33616ed-885f-4af2-9bea-eb1f9e9dcf3c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zdjj4" Sep 10 15:47:56.833203 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:56.833180 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a33616ed-885f-4af2-9bea-eb1f9e9dcf3c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-zdjj4\" (UID: \"a33616ed-885f-4af2-9bea-eb1f9e9dcf3c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zdjj4" Sep 10 15:47:56.933856 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:56.933822 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a33616ed-885f-4af2-9bea-eb1f9e9dcf3c-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-zdjj4\" (UID: \"a33616ed-885f-4af2-9bea-eb1f9e9dcf3c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zdjj4" Sep 10 15:47:56.933856 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:56.933858 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a33616ed-885f-4af2-9bea-eb1f9e9dcf3c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-zdjj4\" (UID: \"a33616ed-885f-4af2-9bea-eb1f9e9dcf3c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zdjj4" Sep 10 15:47:56.934062 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:56.933884 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-949js\" (UniqueName: \"kubernetes.io/projected/a33616ed-885f-4af2-9bea-eb1f9e9dcf3c-kube-api-access-949js\") pod \"istiod-openshift-gateway-5677ccb49b-zdjj4\" (UID: \"a33616ed-885f-4af2-9bea-eb1f9e9dcf3c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zdjj4" Sep 10 15:47:56.934062 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:56.933909 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a33616ed-885f-4af2-9bea-eb1f9e9dcf3c-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-zdjj4\" (UID: \"a33616ed-885f-4af2-9bea-eb1f9e9dcf3c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zdjj4" Sep 10 15:47:56.934062 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:56.933938 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a33616ed-885f-4af2-9bea-eb1f9e9dcf3c-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-zdjj4\" (UID: \"a33616ed-885f-4af2-9bea-eb1f9e9dcf3c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zdjj4" Sep 10 15:47:56.934062 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:56.933963 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a33616ed-885f-4af2-9bea-eb1f9e9dcf3c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-zdjj4\" (UID: \"a33616ed-885f-4af2-9bea-eb1f9e9dcf3c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zdjj4" Sep 10 15:47:56.934062 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:56.934048 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a33616ed-885f-4af2-9bea-eb1f9e9dcf3c-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-zdjj4\" (UID: \"a33616ed-885f-4af2-9bea-eb1f9e9dcf3c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zdjj4" Sep 10 15:47:56.934682 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:56.934655 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/a33616ed-885f-4af2-9bea-eb1f9e9dcf3c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-zdjj4\" (UID: \"a33616ed-885f-4af2-9bea-eb1f9e9dcf3c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zdjj4" Sep 10 15:47:56.936403 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:56.936352 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/a33616ed-885f-4af2-9bea-eb1f9e9dcf3c-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-zdjj4\" (UID: \"a33616ed-885f-4af2-9bea-eb1f9e9dcf3c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zdjj4" Sep 10 15:47:56.936514 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:56.936476 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/a33616ed-885f-4af2-9bea-eb1f9e9dcf3c-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-zdjj4\" (UID: \"a33616ed-885f-4af2-9bea-eb1f9e9dcf3c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zdjj4" Sep 10 15:47:56.936725 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:56.936693 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/a33616ed-885f-4af2-9bea-eb1f9e9dcf3c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-zdjj4\" (UID: \"a33616ed-885f-4af2-9bea-eb1f9e9dcf3c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zdjj4" Sep 10 15:47:56.936785 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:56.936742 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a33616ed-885f-4af2-9bea-eb1f9e9dcf3c-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-zdjj4\" (UID: \"a33616ed-885f-4af2-9bea-eb1f9e9dcf3c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zdjj4" Sep 10 15:47:56.944796 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:56.944766 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a33616ed-885f-4af2-9bea-eb1f9e9dcf3c-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-zdjj4\" (UID: \"a33616ed-885f-4af2-9bea-eb1f9e9dcf3c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zdjj4" Sep 10 15:47:56.945511 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:56.945485 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-949js\" (UniqueName: \"kubernetes.io/projected/a33616ed-885f-4af2-9bea-eb1f9e9dcf3c-kube-api-access-949js\") pod \"istiod-openshift-gateway-5677ccb49b-zdjj4\" (UID: \"a33616ed-885f-4af2-9bea-eb1f9e9dcf3c\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zdjj4" Sep 10 15:47:57.073021 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:57.072940 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zdjj4" Sep 10 15:47:57.365793 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:57.365631 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-zdjj4"] Sep 10 15:47:57.368582 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:47:57.368557 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda33616ed_885f_4af2_9bea_eb1f9e9dcf3c.slice/crio-c06c7426cccfe8058b09e88d98343f68b7f248fa9a733a6a27ac354452e9b2d2 WatchSource:0}: Error finding container c06c7426cccfe8058b09e88d98343f68b7f248fa9a733a6a27ac354452e9b2d2: Status 404 returned error can't find the container with id c06c7426cccfe8058b09e88d98343f68b7f248fa9a733a6a27ac354452e9b2d2 Sep 10 15:47:58.071725 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:58.071691 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zdjj4" event={"ID":"a33616ed-885f-4af2-9bea-eb1f9e9dcf3c","Type":"ContainerStarted","Data":"c06c7426cccfe8058b09e88d98343f68b7f248fa9a733a6a27ac354452e9b2d2"} Sep 10 15:47:59.163763 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:59.163670 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:59.354653 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:59.354613 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:59.536630 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:59.536547 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:59.636170 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:59.636135 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:47:59.652116 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:59.652074 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 15:47:59.652337 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:47:59.652155 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 15:48:00.087727 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:00.087685 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zdjj4" event={"ID":"a33616ed-885f-4af2-9bea-eb1f9e9dcf3c","Type":"ContainerStarted","Data":"13ed5a5e17813404d574e456a15a4b0f1c965ecfa819a8daaa49834323e80e27"} Sep 10 15:48:00.087902 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:00.087855 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zdjj4" Sep 10 15:48:00.089687 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:00.089636 2583 patch_prober.go:28] interesting pod/istiod-openshift-gateway-5677ccb49b-zdjj4 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 10 15:48:00.089791 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:00.089751 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zdjj4" podUID="a33616ed-885f-4af2-9bea-eb1f9e9dcf3c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 15:48:00.112543 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:00.112499 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zdjj4" podStartSLOduration=1.831802816 podStartE2EDuration="4.11248666s" totalImagesPullingTime="2.280683844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:47:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:48:00.111979936 +0000 UTC m=+357.087239902" watchObservedRunningTime="2026-09-10 15:48:00.11248666 +0000 UTC m=+357.087746623" Sep 10 15:48:00.471607 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:00.471576 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:48:00.570529 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:00.570492 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:48:00.634623 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:00.634594 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:48:01.093139 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:01.093115 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-zdjj4" Sep 10 15:48:05.054346 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:05.054309 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-tlfmm" Sep 10 15:48:07.182165 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:07.182135 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:48:07.284753 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:07.284727 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:48:34.849366 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:34.849331 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:48:34.915811 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:34.915784 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:48:40.870020 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:40.869985 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:48:40.931036 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:40.931006 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:48:50.820038 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:50.819994 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:48:50.934291 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:50.934252 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:48:54.649710 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:54.649669 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c697p4n"] Sep 10 15:48:54.659731 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:54.659699 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c697p4n" Sep 10 15:48:54.661322 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:54.661297 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c697p4n"] Sep 10 15:48:54.662216 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:54.662199 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 10 15:48:54.662687 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:54.662669 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 10 15:48:54.663158 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:54.663135 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-7k9nf\"" Sep 10 15:48:54.706536 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:54.706510 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21de073c-385b-41ae-ae3b-3c7c01f66b7c-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c697p4n\" (UID: \"21de073c-385b-41ae-ae3b-3c7c01f66b7c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c697p4n" Sep 10 15:48:54.706630 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:54.706541 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cp65\" (UniqueName: \"kubernetes.io/projected/21de073c-385b-41ae-ae3b-3c7c01f66b7c-kube-api-access-8cp65\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c697p4n\" (UID: \"21de073c-385b-41ae-ae3b-3c7c01f66b7c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c697p4n" Sep 10 15:48:54.706630 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:54.706567 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21de073c-385b-41ae-ae3b-3c7c01f66b7c-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c697p4n\" (UID: \"21de073c-385b-41ae-ae3b-3c7c01f66b7c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c697p4n" Sep 10 15:48:54.776563 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:54.776538 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl5h88"] Sep 10 15:48:54.780087 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:54.780073 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl5h88" Sep 10 15:48:54.795606 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:54.795578 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl5h88"] Sep 10 15:48:54.807584 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:54.807564 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21de073c-385b-41ae-ae3b-3c7c01f66b7c-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c697p4n\" (UID: \"21de073c-385b-41ae-ae3b-3c7c01f66b7c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c697p4n" Sep 10 15:48:54.807676 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:54.807598 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cp65\" (UniqueName: \"kubernetes.io/projected/21de073c-385b-41ae-ae3b-3c7c01f66b7c-kube-api-access-8cp65\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c697p4n\" (UID: \"21de073c-385b-41ae-ae3b-3c7c01f66b7c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c697p4n" Sep 10 15:48:54.807676 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:54.807629 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21de073c-385b-41ae-ae3b-3c7c01f66b7c-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c697p4n\" (UID: \"21de073c-385b-41ae-ae3b-3c7c01f66b7c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c697p4n" Sep 10 15:48:54.807752 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:54.807681 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ead09c28-f449-4729-a4ca-bad6065ac22e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl5h88\" (UID: \"ead09c28-f449-4729-a4ca-bad6065ac22e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl5h88" Sep 10 15:48:54.807752 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:54.807722 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw877\" (UniqueName: \"kubernetes.io/projected/ead09c28-f449-4729-a4ca-bad6065ac22e-kube-api-access-mw877\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl5h88\" (UID: \"ead09c28-f449-4729-a4ca-bad6065ac22e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl5h88" Sep 10 15:48:54.807853 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:54.807784 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ead09c28-f449-4729-a4ca-bad6065ac22e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl5h88\" (UID: \"ead09c28-f449-4729-a4ca-bad6065ac22e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl5h88" Sep 10 15:48:54.807977 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:54.807954 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21de073c-385b-41ae-ae3b-3c7c01f66b7c-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c697p4n\" (UID: \"21de073c-385b-41ae-ae3b-3c7c01f66b7c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c697p4n" Sep 10 15:48:54.808014 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:54.807979 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21de073c-385b-41ae-ae3b-3c7c01f66b7c-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c697p4n\" (UID: \"21de073c-385b-41ae-ae3b-3c7c01f66b7c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c697p4n" Sep 10 15:48:54.827148 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:54.827128 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cp65\" (UniqueName: \"kubernetes.io/projected/21de073c-385b-41ae-ae3b-3c7c01f66b7c-kube-api-access-8cp65\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c697p4n\" (UID: \"21de073c-385b-41ae-ae3b-3c7c01f66b7c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c697p4n" Sep 10 15:48:54.895966 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:54.895940 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afcvn5"] Sep 10 15:48:54.899791 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:54.899747 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afcvn5" Sep 10 15:48:54.908242 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:54.908218 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ead09c28-f449-4729-a4ca-bad6065ac22e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl5h88\" (UID: \"ead09c28-f449-4729-a4ca-bad6065ac22e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl5h88" Sep 10 15:48:54.908515 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:54.908486 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ead09c28-f449-4729-a4ca-bad6065ac22e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl5h88\" (UID: \"ead09c28-f449-4729-a4ca-bad6065ac22e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl5h88" Sep 10 15:48:54.908616 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:54.908537 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw877\" (UniqueName: \"kubernetes.io/projected/ead09c28-f449-4729-a4ca-bad6065ac22e-kube-api-access-mw877\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl5h88\" (UID: \"ead09c28-f449-4729-a4ca-bad6065ac22e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl5h88" Sep 10 15:48:54.908733 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:54.908708 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ead09c28-f449-4729-a4ca-bad6065ac22e-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl5h88\" (UID: \"ead09c28-f449-4729-a4ca-bad6065ac22e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl5h88" Sep 10 15:48:54.908886 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:54.908851 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ead09c28-f449-4729-a4ca-bad6065ac22e-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl5h88\" (UID: \"ead09c28-f449-4729-a4ca-bad6065ac22e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl5h88" Sep 10 15:48:54.910432 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:54.910411 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afcvn5"] Sep 10 15:48:54.920213 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:54.920192 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw877\" (UniqueName: \"kubernetes.io/projected/ead09c28-f449-4729-a4ca-bad6065ac22e-kube-api-access-mw877\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl5h88\" (UID: \"ead09c28-f449-4729-a4ca-bad6065ac22e\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl5h88" Sep 10 15:48:54.966602 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:54.966573 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hmjnn"] Sep 10 15:48:54.969564 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:54.969546 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c697p4n" Sep 10 15:48:54.970246 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:54.970232 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hmjnn" Sep 10 15:48:54.982050 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:54.981875 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hmjnn"] Sep 10 15:48:55.009904 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:55.009871 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3f2b693-88d0-44fe-9af3-217a56ce65b3-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afcvn5\" (UID: \"b3f2b693-88d0-44fe-9af3-217a56ce65b3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afcvn5" Sep 10 15:48:55.010054 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:55.009918 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3f2b693-88d0-44fe-9af3-217a56ce65b3-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afcvn5\" (UID: \"b3f2b693-88d0-44fe-9af3-217a56ce65b3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afcvn5" Sep 10 15:48:55.010054 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:55.009997 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f58867a1-fbad-4c6f-887c-0e5cb9c4db06-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hmjnn\" (UID: \"f58867a1-fbad-4c6f-887c-0e5cb9c4db06\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hmjnn" Sep 10 15:48:55.010144 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:55.010048 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mt5q9\" (UniqueName: \"kubernetes.io/projected/f58867a1-fbad-4c6f-887c-0e5cb9c4db06-kube-api-access-mt5q9\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hmjnn\" (UID: \"f58867a1-fbad-4c6f-887c-0e5cb9c4db06\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hmjnn" Sep 10 15:48:55.010144 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:55.010125 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnhgh\" (UniqueName: \"kubernetes.io/projected/b3f2b693-88d0-44fe-9af3-217a56ce65b3-kube-api-access-nnhgh\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afcvn5\" (UID: \"b3f2b693-88d0-44fe-9af3-217a56ce65b3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afcvn5" Sep 10 15:48:55.010231 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:55.010157 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f58867a1-fbad-4c6f-887c-0e5cb9c4db06-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hmjnn\" (UID: \"f58867a1-fbad-4c6f-887c-0e5cb9c4db06\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hmjnn" Sep 10 15:48:55.088348 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:55.088317 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl5h88" Sep 10 15:48:55.110597 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:55.110570 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3f2b693-88d0-44fe-9af3-217a56ce65b3-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afcvn5\" (UID: \"b3f2b693-88d0-44fe-9af3-217a56ce65b3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afcvn5" Sep 10 15:48:55.110728 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:55.110619 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f58867a1-fbad-4c6f-887c-0e5cb9c4db06-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hmjnn\" (UID: \"f58867a1-fbad-4c6f-887c-0e5cb9c4db06\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hmjnn" Sep 10 15:48:55.110728 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:55.110642 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mt5q9\" (UniqueName: \"kubernetes.io/projected/f58867a1-fbad-4c6f-887c-0e5cb9c4db06-kube-api-access-mt5q9\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hmjnn\" (UID: \"f58867a1-fbad-4c6f-887c-0e5cb9c4db06\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hmjnn" Sep 10 15:48:55.110728 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:55.110702 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnhgh\" (UniqueName: \"kubernetes.io/projected/b3f2b693-88d0-44fe-9af3-217a56ce65b3-kube-api-access-nnhgh\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afcvn5\" (UID: \"b3f2b693-88d0-44fe-9af3-217a56ce65b3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afcvn5" Sep 10 15:48:55.110893 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:55.110736 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f58867a1-fbad-4c6f-887c-0e5cb9c4db06-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hmjnn\" (UID: \"f58867a1-fbad-4c6f-887c-0e5cb9c4db06\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hmjnn" Sep 10 15:48:55.110893 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:55.110766 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3f2b693-88d0-44fe-9af3-217a56ce65b3-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afcvn5\" (UID: \"b3f2b693-88d0-44fe-9af3-217a56ce65b3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afcvn5" Sep 10 15:48:55.111341 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:55.111027 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f58867a1-fbad-4c6f-887c-0e5cb9c4db06-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hmjnn\" (UID: \"f58867a1-fbad-4c6f-887c-0e5cb9c4db06\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hmjnn" Sep 10 15:48:55.111341 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:55.111192 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3f2b693-88d0-44fe-9af3-217a56ce65b3-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afcvn5\" (UID: \"b3f2b693-88d0-44fe-9af3-217a56ce65b3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afcvn5" Sep 10 15:48:55.111341 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:55.111197 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3f2b693-88d0-44fe-9af3-217a56ce65b3-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afcvn5\" (UID: \"b3f2b693-88d0-44fe-9af3-217a56ce65b3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afcvn5" Sep 10 15:48:55.111341 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:55.111277 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f58867a1-fbad-4c6f-887c-0e5cb9c4db06-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hmjnn\" (UID: \"f58867a1-fbad-4c6f-887c-0e5cb9c4db06\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hmjnn" Sep 10 15:48:55.120464 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:55.120437 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnhgh\" (UniqueName: \"kubernetes.io/projected/b3f2b693-88d0-44fe-9af3-217a56ce65b3-kube-api-access-nnhgh\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afcvn5\" (UID: \"b3f2b693-88d0-44fe-9af3-217a56ce65b3\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afcvn5" Sep 10 15:48:55.120573 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:55.120516 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mt5q9\" (UniqueName: \"kubernetes.io/projected/f58867a1-fbad-4c6f-887c-0e5cb9c4db06-kube-api-access-mt5q9\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hmjnn\" (UID: \"f58867a1-fbad-4c6f-887c-0e5cb9c4db06\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hmjnn" Sep 10 15:48:55.209094 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:55.208975 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afcvn5" Sep 10 15:48:55.245844 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:55.245821 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c697p4n"] Sep 10 15:48:55.248656 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:48:55.248507 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21de073c_385b_41ae_ae3b_3c7c01f66b7c.slice/crio-2d3963e98c7da46f83363332c8ccfa3da3ef88753e473899396bc415b90d36ef WatchSource:0}: Error finding container 2d3963e98c7da46f83363332c8ccfa3da3ef88753e473899396bc415b90d36ef: Status 404 returned error can't find the container with id 2d3963e98c7da46f83363332c8ccfa3da3ef88753e473899396bc415b90d36ef Sep 10 15:48:55.303627 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:55.303603 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hmjnn" Sep 10 15:48:55.368604 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:55.368463 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl5h88"] Sep 10 15:48:55.370758 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:55.370502 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c697p4n" event={"ID":"21de073c-385b-41ae-ae3b-3c7c01f66b7c","Type":"ContainerStarted","Data":"a773bfb8b65a6a6de921165b2560bad7c8dae95d10287cfd91864b6104efe9a5"} Sep 10 15:48:55.370758 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:55.370558 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c697p4n" event={"ID":"21de073c-385b-41ae-ae3b-3c7c01f66b7c","Type":"ContainerStarted","Data":"2d3963e98c7da46f83363332c8ccfa3da3ef88753e473899396bc415b90d36ef"} Sep 10 15:48:55.372290 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:48:55.372162 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podead09c28_f449_4729_a4ca_bad6065ac22e.slice/crio-952d478396e2f16a7e2365f4474859ecc88f41e99433b18091480c06273fdd45 WatchSource:0}: Error finding container 952d478396e2f16a7e2365f4474859ecc88f41e99433b18091480c06273fdd45: Status 404 returned error can't find the container with id 952d478396e2f16a7e2365f4474859ecc88f41e99433b18091480c06273fdd45 Sep 10 15:48:55.485373 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:55.485349 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afcvn5"] Sep 10 15:48:55.487660 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:48:55.487630 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3f2b693_88d0_44fe_9af3_217a56ce65b3.slice/crio-7f061230f9ec0caba4f9f7861faefcc7633c10d62d71431c25c8d13a314a5657 WatchSource:0}: Error finding container 7f061230f9ec0caba4f9f7861faefcc7633c10d62d71431c25c8d13a314a5657: Status 404 returned error can't find the container with id 7f061230f9ec0caba4f9f7861faefcc7633c10d62d71431c25c8d13a314a5657 Sep 10 15:48:55.580355 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:55.580329 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hmjnn"] Sep 10 15:48:55.582447 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:48:55.582424 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf58867a1_fbad_4c6f_887c_0e5cb9c4db06.slice/crio-8364b9817919d5335f3d30f0a826c08008fbab4a53a29fc45ccc65651cef1e52 WatchSource:0}: Error finding container 8364b9817919d5335f3d30f0a826c08008fbab4a53a29fc45ccc65651cef1e52: Status 404 returned error can't find the container with id 8364b9817919d5335f3d30f0a826c08008fbab4a53a29fc45ccc65651cef1e52 Sep 10 15:48:56.378067 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:56.378032 2583 generic.go:356] "Generic (PLEG): container finished" podID="ead09c28-f449-4729-a4ca-bad6065ac22e" containerID="4bca7ecf0b707c41938f341f7ecda73bb2148302241d6f9072b9d26e7fa61d1d" exitCode=0 Sep 10 15:48:56.378502 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:56.378107 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl5h88" event={"ID":"ead09c28-f449-4729-a4ca-bad6065ac22e","Type":"ContainerDied","Data":"4bca7ecf0b707c41938f341f7ecda73bb2148302241d6f9072b9d26e7fa61d1d"} Sep 10 15:48:56.378502 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:56.378164 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl5h88" event={"ID":"ead09c28-f449-4729-a4ca-bad6065ac22e","Type":"ContainerStarted","Data":"952d478396e2f16a7e2365f4474859ecc88f41e99433b18091480c06273fdd45"} Sep 10 15:48:56.380003 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:56.379979 2583 generic.go:356] "Generic (PLEG): container finished" podID="f58867a1-fbad-4c6f-887c-0e5cb9c4db06" containerID="8dc90258d06de9a35c32e80daa9a4dec674df77face5fb0c338f51ba5c9fb347" exitCode=0 Sep 10 15:48:56.380088 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:56.380036 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hmjnn" event={"ID":"f58867a1-fbad-4c6f-887c-0e5cb9c4db06","Type":"ContainerDied","Data":"8dc90258d06de9a35c32e80daa9a4dec674df77face5fb0c338f51ba5c9fb347"} Sep 10 15:48:56.380088 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:56.380082 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hmjnn" event={"ID":"f58867a1-fbad-4c6f-887c-0e5cb9c4db06","Type":"ContainerStarted","Data":"8364b9817919d5335f3d30f0a826c08008fbab4a53a29fc45ccc65651cef1e52"} Sep 10 15:48:56.381746 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:56.381727 2583 generic.go:356] "Generic (PLEG): container finished" podID="b3f2b693-88d0-44fe-9af3-217a56ce65b3" containerID="e402babe4cdf05074435cd6e0c6c21f9468f115674a180938356ff977a638303" exitCode=0 Sep 10 15:48:56.381842 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:56.381810 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afcvn5" event={"ID":"b3f2b693-88d0-44fe-9af3-217a56ce65b3","Type":"ContainerDied","Data":"e402babe4cdf05074435cd6e0c6c21f9468f115674a180938356ff977a638303"} Sep 10 15:48:56.381901 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:56.381854 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afcvn5" event={"ID":"b3f2b693-88d0-44fe-9af3-217a56ce65b3","Type":"ContainerStarted","Data":"7f061230f9ec0caba4f9f7861faefcc7633c10d62d71431c25c8d13a314a5657"} Sep 10 15:48:56.383696 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:56.383677 2583 generic.go:356] "Generic (PLEG): container finished" podID="21de073c-385b-41ae-ae3b-3c7c01f66b7c" containerID="a773bfb8b65a6a6de921165b2560bad7c8dae95d10287cfd91864b6104efe9a5" exitCode=0 Sep 10 15:48:56.383796 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:56.383727 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c697p4n" event={"ID":"21de073c-385b-41ae-ae3b-3c7c01f66b7c","Type":"ContainerDied","Data":"a773bfb8b65a6a6de921165b2560bad7c8dae95d10287cfd91864b6104efe9a5"} Sep 10 15:48:56.464355 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:56.464304 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:48:56.543899 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:56.543871 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:48:57.392396 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:57.392358 2583 generic.go:356] "Generic (PLEG): container finished" podID="b3f2b693-88d0-44fe-9af3-217a56ce65b3" containerID="1b61cf1c4f24a30e80faf49c4b63a8503c6bb7e39833f141db098ec857f9aa8a" exitCode=0 Sep 10 15:48:57.392744 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:57.392424 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afcvn5" event={"ID":"b3f2b693-88d0-44fe-9af3-217a56ce65b3","Type":"ContainerDied","Data":"1b61cf1c4f24a30e80faf49c4b63a8503c6bb7e39833f141db098ec857f9aa8a"} Sep 10 15:48:57.394416 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:57.394394 2583 generic.go:356] "Generic (PLEG): container finished" podID="ead09c28-f449-4729-a4ca-bad6065ac22e" containerID="2908577564c80d8e013dace76a00051258f9ce987eb63e0998fa51574053a884" exitCode=0 Sep 10 15:48:57.394507 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:57.394427 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl5h88" event={"ID":"ead09c28-f449-4729-a4ca-bad6065ac22e","Type":"ContainerDied","Data":"2908577564c80d8e013dace76a00051258f9ce987eb63e0998fa51574053a884"} Sep 10 15:48:58.401804 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:58.401772 2583 generic.go:356] "Generic (PLEG): container finished" podID="21de073c-385b-41ae-ae3b-3c7c01f66b7c" containerID="472a723f0fe45142255b866c737a4de43746b7de0ec790522b71e8c7a56ef3c6" exitCode=0 Sep 10 15:48:58.402214 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:58.401864 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c697p4n" event={"ID":"21de073c-385b-41ae-ae3b-3c7c01f66b7c","Type":"ContainerDied","Data":"472a723f0fe45142255b866c737a4de43746b7de0ec790522b71e8c7a56ef3c6"} Sep 10 15:48:58.404177 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:58.404158 2583 generic.go:356] "Generic (PLEG): container finished" podID="ead09c28-f449-4729-a4ca-bad6065ac22e" containerID="d42078f994001e99bbcaac245596c464215e9ed9a0773afd9577eab819d464da" exitCode=0 Sep 10 15:48:58.404279 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:58.404240 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl5h88" event={"ID":"ead09c28-f449-4729-a4ca-bad6065ac22e","Type":"ContainerDied","Data":"d42078f994001e99bbcaac245596c464215e9ed9a0773afd9577eab819d464da"} Sep 10 15:48:58.406106 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:58.406086 2583 generic.go:356] "Generic (PLEG): container finished" podID="f58867a1-fbad-4c6f-887c-0e5cb9c4db06" containerID="dcdbc71aa940de979c62decd2e9655866c7c5d71b69d9676df3066d1a049675d" exitCode=0 Sep 10 15:48:58.406206 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:58.406153 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hmjnn" event={"ID":"f58867a1-fbad-4c6f-887c-0e5cb9c4db06","Type":"ContainerDied","Data":"dcdbc71aa940de979c62decd2e9655866c7c5d71b69d9676df3066d1a049675d"} Sep 10 15:48:58.408680 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:58.408661 2583 generic.go:356] "Generic (PLEG): container finished" podID="b3f2b693-88d0-44fe-9af3-217a56ce65b3" containerID="efa7f4da1d17b29d3b7bf4cb88e8e15aa33457f9bff9bf2b1d90a34ee2aff334" exitCode=0 Sep 10 15:48:58.408788 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:58.408686 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afcvn5" event={"ID":"b3f2b693-88d0-44fe-9af3-217a56ce65b3","Type":"ContainerDied","Data":"efa7f4da1d17b29d3b7bf4cb88e8e15aa33457f9bff9bf2b1d90a34ee2aff334"} Sep 10 15:48:59.416290 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:59.416252 2583 generic.go:356] "Generic (PLEG): container finished" podID="21de073c-385b-41ae-ae3b-3c7c01f66b7c" containerID="6d64d5374b4b47d2cfafbf92bbefcd08dcf6b55ee271144d930b6140524c47c4" exitCode=0 Sep 10 15:48:59.416719 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:59.416345 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c697p4n" event={"ID":"21de073c-385b-41ae-ae3b-3c7c01f66b7c","Type":"ContainerDied","Data":"6d64d5374b4b47d2cfafbf92bbefcd08dcf6b55ee271144d930b6140524c47c4"} Sep 10 15:48:59.418499 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:59.418480 2583 generic.go:356] "Generic (PLEG): container finished" podID="f58867a1-fbad-4c6f-887c-0e5cb9c4db06" containerID="4ccb575123f5293e8381ca9f9bfab4c98f9495127cdde3e1f065a5d72c45e2a1" exitCode=0 Sep 10 15:48:59.418620 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:59.418535 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hmjnn" event={"ID":"f58867a1-fbad-4c6f-887c-0e5cb9c4db06","Type":"ContainerDied","Data":"4ccb575123f5293e8381ca9f9bfab4c98f9495127cdde3e1f065a5d72c45e2a1"} Sep 10 15:48:59.562393 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:59.562352 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afcvn5" Sep 10 15:48:59.598462 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:59.598440 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl5h88" Sep 10 15:48:59.646242 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:59.646212 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3f2b693-88d0-44fe-9af3-217a56ce65b3-util\") pod \"b3f2b693-88d0-44fe-9af3-217a56ce65b3\" (UID: \"b3f2b693-88d0-44fe-9af3-217a56ce65b3\") " Sep 10 15:48:59.646404 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:59.646258 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3f2b693-88d0-44fe-9af3-217a56ce65b3-bundle\") pod \"b3f2b693-88d0-44fe-9af3-217a56ce65b3\" (UID: \"b3f2b693-88d0-44fe-9af3-217a56ce65b3\") " Sep 10 15:48:59.646404 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:59.646300 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ead09c28-f449-4729-a4ca-bad6065ac22e-util\") pod \"ead09c28-f449-4729-a4ca-bad6065ac22e\" (UID: \"ead09c28-f449-4729-a4ca-bad6065ac22e\") " Sep 10 15:48:59.646404 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:59.646328 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nnhgh\" (UniqueName: \"kubernetes.io/projected/b3f2b693-88d0-44fe-9af3-217a56ce65b3-kube-api-access-nnhgh\") pod \"b3f2b693-88d0-44fe-9af3-217a56ce65b3\" (UID: \"b3f2b693-88d0-44fe-9af3-217a56ce65b3\") " Sep 10 15:48:59.646404 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:59.646376 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ead09c28-f449-4729-a4ca-bad6065ac22e-bundle\") pod \"ead09c28-f449-4729-a4ca-bad6065ac22e\" (UID: \"ead09c28-f449-4729-a4ca-bad6065ac22e\") " Sep 10 15:48:59.646613 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:59.646514 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mw877\" (UniqueName: \"kubernetes.io/projected/ead09c28-f449-4729-a4ca-bad6065ac22e-kube-api-access-mw877\") pod \"ead09c28-f449-4729-a4ca-bad6065ac22e\" (UID: \"ead09c28-f449-4729-a4ca-bad6065ac22e\") " Sep 10 15:48:59.647042 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:59.646989 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ead09c28-f449-4729-a4ca-bad6065ac22e-bundle" (OuterVolumeSpecName: "bundle") pod "ead09c28-f449-4729-a4ca-bad6065ac22e" (UID: "ead09c28-f449-4729-a4ca-bad6065ac22e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:48:59.647151 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:59.647073 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3f2b693-88d0-44fe-9af3-217a56ce65b3-bundle" (OuterVolumeSpecName: "bundle") pod "b3f2b693-88d0-44fe-9af3-217a56ce65b3" (UID: "b3f2b693-88d0-44fe-9af3-217a56ce65b3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:48:59.648948 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:59.648919 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ead09c28-f449-4729-a4ca-bad6065ac22e-kube-api-access-mw877" (OuterVolumeSpecName: "kube-api-access-mw877") pod "ead09c28-f449-4729-a4ca-bad6065ac22e" (UID: "ead09c28-f449-4729-a4ca-bad6065ac22e"). InnerVolumeSpecName "kube-api-access-mw877". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:48:59.649622 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:59.649450 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3f2b693-88d0-44fe-9af3-217a56ce65b3-kube-api-access-nnhgh" (OuterVolumeSpecName: "kube-api-access-nnhgh") pod "b3f2b693-88d0-44fe-9af3-217a56ce65b3" (UID: "b3f2b693-88d0-44fe-9af3-217a56ce65b3"). InnerVolumeSpecName "kube-api-access-nnhgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:48:59.654364 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:59.654333 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ead09c28-f449-4729-a4ca-bad6065ac22e-util" (OuterVolumeSpecName: "util") pod "ead09c28-f449-4729-a4ca-bad6065ac22e" (UID: "ead09c28-f449-4729-a4ca-bad6065ac22e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:48:59.655149 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:59.655120 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3f2b693-88d0-44fe-9af3-217a56ce65b3-util" (OuterVolumeSpecName: "util") pod "b3f2b693-88d0-44fe-9af3-217a56ce65b3" (UID: "b3f2b693-88d0-44fe-9af3-217a56ce65b3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:48:59.748112 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:59.748052 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnhgh\" (UniqueName: \"kubernetes.io/projected/b3f2b693-88d0-44fe-9af3-217a56ce65b3-kube-api-access-nnhgh\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:48:59.748112 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:59.748077 2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ead09c28-f449-4729-a4ca-bad6065ac22e-bundle\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:48:59.748112 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:59.748091 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw877\" (UniqueName: \"kubernetes.io/projected/ead09c28-f449-4729-a4ca-bad6065ac22e-kube-api-access-mw877\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:48:59.748112 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:59.748103 2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b3f2b693-88d0-44fe-9af3-217a56ce65b3-util\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:48:59.748297 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:59.748116 2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b3f2b693-88d0-44fe-9af3-217a56ce65b3-bundle\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:48:59.748297 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:48:59.748127 2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ead09c28-f449-4729-a4ca-bad6065ac22e-util\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:49:00.425699 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:00.425673 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl5h88" Sep 10 15:49:00.426167 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:00.425705 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cl5h88" event={"ID":"ead09c28-f449-4729-a4ca-bad6065ac22e","Type":"ContainerDied","Data":"952d478396e2f16a7e2365f4474859ecc88f41e99433b18091480c06273fdd45"} Sep 10 15:49:00.426167 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:00.425753 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="952d478396e2f16a7e2365f4474859ecc88f41e99433b18091480c06273fdd45" Sep 10 15:49:00.427640 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:00.427618 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afcvn5" Sep 10 15:49:00.427769 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:00.427640 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847afcvn5" event={"ID":"b3f2b693-88d0-44fe-9af3-217a56ce65b3","Type":"ContainerDied","Data":"7f061230f9ec0caba4f9f7861faefcc7633c10d62d71431c25c8d13a314a5657"} Sep 10 15:49:00.427769 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:00.427684 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f061230f9ec0caba4f9f7861faefcc7633c10d62d71431c25c8d13a314a5657" Sep 10 15:49:00.583724 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:00.583701 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hmjnn" Sep 10 15:49:00.585694 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:00.585674 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c697p4n" Sep 10 15:49:00.656218 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:00.656190 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8cp65\" (UniqueName: \"kubernetes.io/projected/21de073c-385b-41ae-ae3b-3c7c01f66b7c-kube-api-access-8cp65\") pod \"21de073c-385b-41ae-ae3b-3c7c01f66b7c\" (UID: \"21de073c-385b-41ae-ae3b-3c7c01f66b7c\") " Sep 10 15:49:00.656367 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:00.656243 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mt5q9\" (UniqueName: \"kubernetes.io/projected/f58867a1-fbad-4c6f-887c-0e5cb9c4db06-kube-api-access-mt5q9\") pod \"f58867a1-fbad-4c6f-887c-0e5cb9c4db06\" (UID: \"f58867a1-fbad-4c6f-887c-0e5cb9c4db06\") " Sep 10 15:49:00.656367 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:00.656291 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f58867a1-fbad-4c6f-887c-0e5cb9c4db06-util\") pod \"f58867a1-fbad-4c6f-887c-0e5cb9c4db06\" (UID: \"f58867a1-fbad-4c6f-887c-0e5cb9c4db06\") " Sep 10 15:49:00.656367 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:00.656322 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21de073c-385b-41ae-ae3b-3c7c01f66b7c-util\") pod \"21de073c-385b-41ae-ae3b-3c7c01f66b7c\" (UID: \"21de073c-385b-41ae-ae3b-3c7c01f66b7c\") " Sep 10 15:49:00.656563 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:00.656404 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21de073c-385b-41ae-ae3b-3c7c01f66b7c-bundle\") pod \"21de073c-385b-41ae-ae3b-3c7c01f66b7c\" (UID: \"21de073c-385b-41ae-ae3b-3c7c01f66b7c\") " Sep 10 15:49:00.656563 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:00.656436 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f58867a1-fbad-4c6f-887c-0e5cb9c4db06-bundle\") pod \"f58867a1-fbad-4c6f-887c-0e5cb9c4db06\" (UID: \"f58867a1-fbad-4c6f-887c-0e5cb9c4db06\") " Sep 10 15:49:00.657115 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:00.656856 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21de073c-385b-41ae-ae3b-3c7c01f66b7c-bundle" (OuterVolumeSpecName: "bundle") pod "21de073c-385b-41ae-ae3b-3c7c01f66b7c" (UID: "21de073c-385b-41ae-ae3b-3c7c01f66b7c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:49:00.657115 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:00.657018 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f58867a1-fbad-4c6f-887c-0e5cb9c4db06-bundle" (OuterVolumeSpecName: "bundle") pod "f58867a1-fbad-4c6f-887c-0e5cb9c4db06" (UID: "f58867a1-fbad-4c6f-887c-0e5cb9c4db06"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:49:00.659201 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:00.659159 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f58867a1-fbad-4c6f-887c-0e5cb9c4db06-kube-api-access-mt5q9" (OuterVolumeSpecName: "kube-api-access-mt5q9") pod "f58867a1-fbad-4c6f-887c-0e5cb9c4db06" (UID: "f58867a1-fbad-4c6f-887c-0e5cb9c4db06"). InnerVolumeSpecName "kube-api-access-mt5q9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:49:00.659310 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:00.659252 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21de073c-385b-41ae-ae3b-3c7c01f66b7c-kube-api-access-8cp65" (OuterVolumeSpecName: "kube-api-access-8cp65") pod "21de073c-385b-41ae-ae3b-3c7c01f66b7c" (UID: "21de073c-385b-41ae-ae3b-3c7c01f66b7c"). InnerVolumeSpecName "kube-api-access-8cp65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:49:00.663638 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:00.663617 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21de073c-385b-41ae-ae3b-3c7c01f66b7c-util" (OuterVolumeSpecName: "util") pod "21de073c-385b-41ae-ae3b-3c7c01f66b7c" (UID: "21de073c-385b-41ae-ae3b-3c7c01f66b7c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:49:00.663723 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:00.663667 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f58867a1-fbad-4c6f-887c-0e5cb9c4db06-util" (OuterVolumeSpecName: "util") pod "f58867a1-fbad-4c6f-887c-0e5cb9c4db06" (UID: "f58867a1-fbad-4c6f-887c-0e5cb9c4db06"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:49:00.757573 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:00.757518 2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f58867a1-fbad-4c6f-887c-0e5cb9c4db06-util\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:49:00.757573 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:00.757540 2583 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/21de073c-385b-41ae-ae3b-3c7c01f66b7c-util\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:49:00.757573 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:00.757549 2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/21de073c-385b-41ae-ae3b-3c7c01f66b7c-bundle\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:49:00.757573 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:00.757557 2583 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f58867a1-fbad-4c6f-887c-0e5cb9c4db06-bundle\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:49:00.757573 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:00.757565 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cp65\" (UniqueName: \"kubernetes.io/projected/21de073c-385b-41ae-ae3b-3c7c01f66b7c-kube-api-access-8cp65\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:49:00.757573 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:00.757574 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mt5q9\" (UniqueName: \"kubernetes.io/projected/f58867a1-fbad-4c6f-887c-0e5cb9c4db06-kube-api-access-mt5q9\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:49:01.439930 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:01.439895 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c697p4n" Sep 10 15:49:01.439930 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:01.439914 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c697p4n" event={"ID":"21de073c-385b-41ae-ae3b-3c7c01f66b7c","Type":"ContainerDied","Data":"2d3963e98c7da46f83363332c8ccfa3da3ef88753e473899396bc415b90d36ef"} Sep 10 15:49:01.440450 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:01.439989 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d3963e98c7da46f83363332c8ccfa3da3ef88753e473899396bc415b90d36ef" Sep 10 15:49:01.441917 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:01.441895 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hmjnn" event={"ID":"f58867a1-fbad-4c6f-887c-0e5cb9c4db06","Type":"ContainerDied","Data":"8364b9817919d5335f3d30f0a826c08008fbab4a53a29fc45ccc65651cef1e52"} Sep 10 15:49:01.442025 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:01.441928 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8364b9817919d5335f3d30f0a826c08008fbab4a53a29fc45ccc65651cef1e52" Sep 10 15:49:01.442025 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:01.441902 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7hmjnn" Sep 10 15:49:06.532458 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:06.532424 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:07.304770 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:07.304736 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:08.122954 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:08.122914 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:11.030812 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:11.030752 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:11.120741 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:11.120711 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:11.554067 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:11.554035 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:13.127097 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:13.127061 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:13.757638 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:13.757606 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:13.862172 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:13.862134 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:16.133614 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:16.133555 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:16.357324 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:16.357293 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:16.531525 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:16.531436 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:18.315158 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:18.315121 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:18.815070 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:18.815037 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:19.108290 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:19.108211 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:21.743761 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:21.743728 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:21.850643 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:21.850612 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:22.144975 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:22.144939 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:22.251938 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:22.251898 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:22.590711 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:22.590632 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:22.739955 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:22.739918 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:28.333257 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:28.333164 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:28.501957 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:28.501924 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:30.241355 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:30.241315 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:30.340115 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:30.340089 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:34.776340 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:34.776308 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:34.875135 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:34.875103 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:36.949489 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:36.949444 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:37.824671 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:37.824634 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:37.981364 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:37.981331 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:42.845957 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:42.845924 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:42.946845 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:42.946810 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:43.048086 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:43.048053 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:49.227951 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:49.227922 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:49.325151 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:49.325091 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:52.801518 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:52.801484 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:55.734127 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:55.734095 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:55.836572 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:55.836542 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:58.730839 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:58.730805 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:49:58.831066 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:49:58.831032 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:50:00.140318 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:50:00.140281 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:50:00.558214 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:50:00.558140 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:50:03.980787 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:50:03.980756 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:50:04.353855 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:50:04.353765 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:50:04.452231 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:50:04.452203 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:50:04.553287 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:50:04.553259 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:50:04.619715 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:50:04.619637 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:50:06.181701 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:50:06.181663 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:50:06.281622 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:50:06.281593 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:50:06.798455 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:50:06.798424 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:50:06.891188 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:50:06.891162 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:50:11.274860 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:50:11.274829 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:50:11.581283 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:50:11.581189 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:50:17.546544 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:50:17.546503 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:50:17.658989 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:50:17.658953 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:50:20.545023 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:50:20.544988 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:50:20.645431 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:50:20.645372 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:50:27.099225 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:50:27.099186 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:50:28.995096 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:50:28.995063 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:50:54.474654 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:50:54.474617 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:50:54.747738 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:50:54.747656 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:51:00.859284 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:00.859211 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:51:00.956793 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:00.956758 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:51:01.025688 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:01.025662 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:51:01.119861 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:01.119780 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:51:04.509198 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:04.509161 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:51:04.664177 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:04.664143 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:51:36.603659 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:36.603622 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:51:36.698768 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:36.698730 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:51:37.997602 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:37.997551 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:51:38.094939 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.094894 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:51:38.147955 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.147922 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7f5df8d666-pnkxh"] Sep 10 15:51:38.148302 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.148287 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3f2b693-88d0-44fe-9af3-217a56ce65b3" containerName="util" Sep 10 15:51:38.148302 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.148301 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3f2b693-88d0-44fe-9af3-217a56ce65b3" containerName="util" Sep 10 15:51:38.148507 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.148313 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ead09c28-f449-4729-a4ca-bad6065ac22e" containerName="extract" Sep 10 15:51:38.148507 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.148318 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ead09c28-f449-4729-a4ca-bad6065ac22e" containerName="extract" Sep 10 15:51:38.148507 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.148325 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f58867a1-fbad-4c6f-887c-0e5cb9c4db06" containerName="pull" Sep 10 15:51:38.148507 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.148331 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f58867a1-fbad-4c6f-887c-0e5cb9c4db06" containerName="pull" Sep 10 15:51:38.148507 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.148340 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f58867a1-fbad-4c6f-887c-0e5cb9c4db06" containerName="util" Sep 10 15:51:38.148507 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.148348 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f58867a1-fbad-4c6f-887c-0e5cb9c4db06" containerName="util" Sep 10 15:51:38.148507 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.148370 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3f2b693-88d0-44fe-9af3-217a56ce65b3" containerName="pull" Sep 10 15:51:38.148507 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.148375 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3f2b693-88d0-44fe-9af3-217a56ce65b3" containerName="pull" Sep 10 15:51:38.148507 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.148397 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21de073c-385b-41ae-ae3b-3c7c01f66b7c" containerName="util" Sep 10 15:51:38.148507 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.148402 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="21de073c-385b-41ae-ae3b-3c7c01f66b7c" containerName="util" Sep 10 15:51:38.148507 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.148418 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ead09c28-f449-4729-a4ca-bad6065ac22e" containerName="util" Sep 10 15:51:38.148507 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.148426 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ead09c28-f449-4729-a4ca-bad6065ac22e" containerName="util" Sep 10 15:51:38.148507 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.148434 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21de073c-385b-41ae-ae3b-3c7c01f66b7c" containerName="extract" Sep 10 15:51:38.148507 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.148442 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="21de073c-385b-41ae-ae3b-3c7c01f66b7c" containerName="extract" Sep 10 15:51:38.148507 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.148449 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21de073c-385b-41ae-ae3b-3c7c01f66b7c" containerName="pull" Sep 10 15:51:38.148507 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.148454 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="21de073c-385b-41ae-ae3b-3c7c01f66b7c" containerName="pull" Sep 10 15:51:38.148507 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.148459 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3f2b693-88d0-44fe-9af3-217a56ce65b3" containerName="extract" Sep 10 15:51:38.148507 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.148464 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3f2b693-88d0-44fe-9af3-217a56ce65b3" containerName="extract" Sep 10 15:51:38.148507 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.148476 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f58867a1-fbad-4c6f-887c-0e5cb9c4db06" containerName="extract" Sep 10 15:51:38.148507 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.148481 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f58867a1-fbad-4c6f-887c-0e5cb9c4db06" containerName="extract" Sep 10 15:51:38.148507 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.148497 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ead09c28-f449-4729-a4ca-bad6065ac22e" containerName="pull" Sep 10 15:51:38.148507 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.148505 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ead09c28-f449-4729-a4ca-bad6065ac22e" containerName="pull" Sep 10 15:51:38.149184 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.148577 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3f2b693-88d0-44fe-9af3-217a56ce65b3" containerName="extract" Sep 10 15:51:38.149184 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.148589 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f58867a1-fbad-4c6f-887c-0e5cb9c4db06" containerName="extract" Sep 10 15:51:38.149184 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.148603 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ead09c28-f449-4729-a4ca-bad6065ac22e" containerName="extract" Sep 10 15:51:38.149184 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.148609 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="21de073c-385b-41ae-ae3b-3c7c01f66b7c" containerName="extract" Sep 10 15:51:38.151735 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.151720 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f5df8d666-pnkxh" Sep 10 15:51:38.153978 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.153956 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 10 15:51:38.154077 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.153979 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 10 15:51:38.154725 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.154706 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 10 15:51:38.154802 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.154749 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-qfdsq\"" Sep 10 15:51:38.154850 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.154756 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 10 15:51:38.163475 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.163453 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f5df8d666-pnkxh"] Sep 10 15:51:38.167318 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.167261 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:51:38.247067 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.247041 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swxrm\" (UniqueName: \"kubernetes.io/projected/c8de5e93-f9f7-4073-a54f-ad64580068f4-kube-api-access-swxrm\") pod \"maas-api-7f5df8d666-pnkxh\" (UID: \"c8de5e93-f9f7-4073-a54f-ad64580068f4\") " pod="odh-ai-gateway-infra/maas-api-7f5df8d666-pnkxh" Sep 10 15:51:38.247211 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.247088 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8de5e93-f9f7-4073-a54f-ad64580068f4-openshift-service-ca\") pod \"maas-api-7f5df8d666-pnkxh\" (UID: \"c8de5e93-f9f7-4073-a54f-ad64580068f4\") " pod="odh-ai-gateway-infra/maas-api-7f5df8d666-pnkxh" Sep 10 15:51:38.247211 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.247152 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8de5e93-f9f7-4073-a54f-ad64580068f4-metrics-certs\") pod \"maas-api-7f5df8d666-pnkxh\" (UID: \"c8de5e93-f9f7-4073-a54f-ad64580068f4\") " pod="odh-ai-gateway-infra/maas-api-7f5df8d666-pnkxh" Sep 10 15:51:38.247211 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.247191 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8de5e93-f9f7-4073-a54f-ad64580068f4-maas-api-tls\") pod \"maas-api-7f5df8d666-pnkxh\" (UID: \"c8de5e93-f9f7-4073-a54f-ad64580068f4\") " pod="odh-ai-gateway-infra/maas-api-7f5df8d666-pnkxh" Sep 10 15:51:38.348560 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.348498 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swxrm\" (UniqueName: \"kubernetes.io/projected/c8de5e93-f9f7-4073-a54f-ad64580068f4-kube-api-access-swxrm\") pod \"maas-api-7f5df8d666-pnkxh\" (UID: \"c8de5e93-f9f7-4073-a54f-ad64580068f4\") " pod="odh-ai-gateway-infra/maas-api-7f5df8d666-pnkxh" Sep 10 15:51:38.348560 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.348538 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8de5e93-f9f7-4073-a54f-ad64580068f4-openshift-service-ca\") pod \"maas-api-7f5df8d666-pnkxh\" (UID: \"c8de5e93-f9f7-4073-a54f-ad64580068f4\") " pod="odh-ai-gateway-infra/maas-api-7f5df8d666-pnkxh" Sep 10 15:51:38.348747 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.348570 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8de5e93-f9f7-4073-a54f-ad64580068f4-metrics-certs\") pod \"maas-api-7f5df8d666-pnkxh\" (UID: \"c8de5e93-f9f7-4073-a54f-ad64580068f4\") " pod="odh-ai-gateway-infra/maas-api-7f5df8d666-pnkxh" Sep 10 15:51:38.348747 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.348591 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8de5e93-f9f7-4073-a54f-ad64580068f4-maas-api-tls\") pod \"maas-api-7f5df8d666-pnkxh\" (UID: \"c8de5e93-f9f7-4073-a54f-ad64580068f4\") " pod="odh-ai-gateway-infra/maas-api-7f5df8d666-pnkxh" Sep 10 15:51:38.348747 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:51:38.348723 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-metrics-tls: secret "maas-api-metrics-tls" not found Sep 10 15:51:38.348902 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:51:38.348812 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8de5e93-f9f7-4073-a54f-ad64580068f4-metrics-certs podName:c8de5e93-f9f7-4073-a54f-ad64580068f4 nodeName:}" failed. No retries permitted until 2026-09-10 15:51:38.848788634 +0000 UTC m=+575.824048579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c8de5e93-f9f7-4073-a54f-ad64580068f4-metrics-certs") pod "maas-api-7f5df8d666-pnkxh" (UID: "c8de5e93-f9f7-4073-a54f-ad64580068f4") : secret "maas-api-metrics-tls" not found Sep 10 15:51:38.349197 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.349178 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8de5e93-f9f7-4073-a54f-ad64580068f4-openshift-service-ca\") pod \"maas-api-7f5df8d666-pnkxh\" (UID: \"c8de5e93-f9f7-4073-a54f-ad64580068f4\") " pod="odh-ai-gateway-infra/maas-api-7f5df8d666-pnkxh" Sep 10 15:51:38.351061 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.351041 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8de5e93-f9f7-4073-a54f-ad64580068f4-maas-api-tls\") pod \"maas-api-7f5df8d666-pnkxh\" (UID: \"c8de5e93-f9f7-4073-a54f-ad64580068f4\") " pod="odh-ai-gateway-infra/maas-api-7f5df8d666-pnkxh" Sep 10 15:51:38.362664 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.362641 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swxrm\" (UniqueName: \"kubernetes.io/projected/c8de5e93-f9f7-4073-a54f-ad64580068f4-kube-api-access-swxrm\") pod \"maas-api-7f5df8d666-pnkxh\" (UID: \"c8de5e93-f9f7-4073-a54f-ad64580068f4\") " pod="odh-ai-gateway-infra/maas-api-7f5df8d666-pnkxh" Sep 10 15:51:38.402148 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.402125 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:51:38.546917 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.546882 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:51:38.643502 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.643459 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:51:38.852738 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.852706 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8de5e93-f9f7-4073-a54f-ad64580068f4-metrics-certs\") pod \"maas-api-7f5df8d666-pnkxh\" (UID: \"c8de5e93-f9f7-4073-a54f-ad64580068f4\") " pod="odh-ai-gateway-infra/maas-api-7f5df8d666-pnkxh" Sep 10 15:51:38.855016 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:38.854996 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8de5e93-f9f7-4073-a54f-ad64580068f4-metrics-certs\") pod \"maas-api-7f5df8d666-pnkxh\" (UID: \"c8de5e93-f9f7-4073-a54f-ad64580068f4\") " pod="odh-ai-gateway-infra/maas-api-7f5df8d666-pnkxh" Sep 10 15:51:39.064726 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:39.064652 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f5df8d666-pnkxh" Sep 10 15:51:39.336397 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:39.336355 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f5df8d666-pnkxh"] Sep 10 15:51:39.337611 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:51:39.337586 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8de5e93_f9f7_4073_a54f_ad64580068f4.slice/crio-4aa4c691cac2a2bc26b565113f2d038e07fc1adf1d9062ac139b6ccf9b33e9de WatchSource:0}: Error finding container 4aa4c691cac2a2bc26b565113f2d038e07fc1adf1d9062ac139b6ccf9b33e9de: Status 404 returned error can't find the container with id 4aa4c691cac2a2bc26b565113f2d038e07fc1adf1d9062ac139b6ccf9b33e9de Sep 10 15:51:40.308273 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:40.308232 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f5df8d666-pnkxh" event={"ID":"c8de5e93-f9f7-4073-a54f-ad64580068f4","Type":"ContainerStarted","Data":"4aa4c691cac2a2bc26b565113f2d038e07fc1adf1d9062ac139b6ccf9b33e9de"} Sep 10 15:51:42.320577 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:42.320542 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f5df8d666-pnkxh" event={"ID":"c8de5e93-f9f7-4073-a54f-ad64580068f4","Type":"ContainerStarted","Data":"82115493d97d9f91fee8a939efd7d10778ce27d618c83bededd12d93e5ba2f97"} Sep 10 15:51:42.320966 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:42.320680 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7f5df8d666-pnkxh" Sep 10 15:51:42.339016 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:42.338962 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7f5df8d666-pnkxh" podStartSLOduration=2.147892962 podStartE2EDuration="4.33894364s" totalImagesPullingTime="2.191050678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:51:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:51:42.337282267 +0000 UTC m=+579.312542221" watchObservedRunningTime="2026-09-10 15:51:42.33894364 +0000 UTC m=+579.314203574" Sep 10 15:51:48.331263 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:48.331232 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7f5df8d666-pnkxh" Sep 10 15:51:49.299357 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:49.299330 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:51:49.396816 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:51:49.396785 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:52:00.963700 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:00.963665 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d4l8v"] Sep 10 15:52:00.967219 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:00.967197 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d4l8v" Sep 10 15:52:00.969165 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:00.969144 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 10 15:52:00.969277 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:00.969232 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 10 15:52:00.969909 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:00.969887 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 10 15:52:00.970011 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:00.969915 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-h87hn\"" Sep 10 15:52:00.977696 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:00.977676 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d4l8v"] Sep 10 15:52:00.988517 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:00.988489 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:52:01.033072 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:01.033046 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/404be92e-4d9e-4070-a64f-ccf91ba25b8c-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d4l8v\" (UID: \"404be92e-4d9e-4070-a64f-ccf91ba25b8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d4l8v" Sep 10 15:52:01.033190 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:01.033076 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkx4c\" (UniqueName: \"kubernetes.io/projected/404be92e-4d9e-4070-a64f-ccf91ba25b8c-kube-api-access-kkx4c\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d4l8v\" (UID: \"404be92e-4d9e-4070-a64f-ccf91ba25b8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d4l8v" Sep 10 15:52:01.033190 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:01.033108 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/404be92e-4d9e-4070-a64f-ccf91ba25b8c-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d4l8v\" (UID: \"404be92e-4d9e-4070-a64f-ccf91ba25b8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d4l8v" Sep 10 15:52:01.033190 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:01.033127 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/404be92e-4d9e-4070-a64f-ccf91ba25b8c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d4l8v\" (UID: \"404be92e-4d9e-4070-a64f-ccf91ba25b8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d4l8v" Sep 10 15:52:01.033190 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:01.033170 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/404be92e-4d9e-4070-a64f-ccf91ba25b8c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d4l8v\" (UID: \"404be92e-4d9e-4070-a64f-ccf91ba25b8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d4l8v" Sep 10 15:52:01.033190 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:01.033185 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/404be92e-4d9e-4070-a64f-ccf91ba25b8c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d4l8v\" (UID: \"404be92e-4d9e-4070-a64f-ccf91ba25b8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d4l8v" Sep 10 15:52:01.133655 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:01.133618 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/404be92e-4d9e-4070-a64f-ccf91ba25b8c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d4l8v\" (UID: \"404be92e-4d9e-4070-a64f-ccf91ba25b8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d4l8v" Sep 10 15:52:01.133655 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:01.133656 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/404be92e-4d9e-4070-a64f-ccf91ba25b8c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d4l8v\" (UID: \"404be92e-4d9e-4070-a64f-ccf91ba25b8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d4l8v" Sep 10 15:52:01.133868 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:01.133718 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/404be92e-4d9e-4070-a64f-ccf91ba25b8c-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d4l8v\" (UID: \"404be92e-4d9e-4070-a64f-ccf91ba25b8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d4l8v" Sep 10 15:52:01.133868 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:01.133734 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkx4c\" (UniqueName: \"kubernetes.io/projected/404be92e-4d9e-4070-a64f-ccf91ba25b8c-kube-api-access-kkx4c\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d4l8v\" (UID: \"404be92e-4d9e-4070-a64f-ccf91ba25b8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d4l8v" Sep 10 15:52:01.133868 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:01.133769 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/404be92e-4d9e-4070-a64f-ccf91ba25b8c-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d4l8v\" (UID: \"404be92e-4d9e-4070-a64f-ccf91ba25b8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d4l8v" Sep 10 15:52:01.133868 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:01.133791 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/404be92e-4d9e-4070-a64f-ccf91ba25b8c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d4l8v\" (UID: \"404be92e-4d9e-4070-a64f-ccf91ba25b8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d4l8v" Sep 10 15:52:01.134152 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:01.134120 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/404be92e-4d9e-4070-a64f-ccf91ba25b8c-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d4l8v\" (UID: \"404be92e-4d9e-4070-a64f-ccf91ba25b8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d4l8v" Sep 10 15:52:01.134229 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:01.134175 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/404be92e-4d9e-4070-a64f-ccf91ba25b8c-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d4l8v\" (UID: \"404be92e-4d9e-4070-a64f-ccf91ba25b8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d4l8v" Sep 10 15:52:01.134286 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:01.134230 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/404be92e-4d9e-4070-a64f-ccf91ba25b8c-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d4l8v\" (UID: \"404be92e-4d9e-4070-a64f-ccf91ba25b8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d4l8v" Sep 10 15:52:01.135875 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:01.135856 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/404be92e-4d9e-4070-a64f-ccf91ba25b8c-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d4l8v\" (UID: \"404be92e-4d9e-4070-a64f-ccf91ba25b8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d4l8v" Sep 10 15:52:01.136259 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:01.136241 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/404be92e-4d9e-4070-a64f-ccf91ba25b8c-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d4l8v\" (UID: \"404be92e-4d9e-4070-a64f-ccf91ba25b8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d4l8v" Sep 10 15:52:01.141489 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:01.141467 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkx4c\" (UniqueName: \"kubernetes.io/projected/404be92e-4d9e-4070-a64f-ccf91ba25b8c-kube-api-access-kkx4c\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-d4l8v\" (UID: \"404be92e-4d9e-4070-a64f-ccf91ba25b8c\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d4l8v" Sep 10 15:52:01.279214 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:01.279117 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d4l8v" Sep 10 15:52:01.550050 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:01.550025 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d4l8v"] Sep 10 15:52:01.551359 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:52:01.551332 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod404be92e_4d9e_4070_a64f_ccf91ba25b8c.slice/crio-ead6c6adb34658e7adb9327af0d94c4acf5cf367c59e2ab01517ca1cbb956bb9 WatchSource:0}: Error finding container ead6c6adb34658e7adb9327af0d94c4acf5cf367c59e2ab01517ca1cbb956bb9: Status 404 returned error can't find the container with id ead6c6adb34658e7adb9327af0d94c4acf5cf367c59e2ab01517ca1cbb956bb9 Sep 10 15:52:02.428808 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:02.428763 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d4l8v" event={"ID":"404be92e-4d9e-4070-a64f-ccf91ba25b8c","Type":"ContainerStarted","Data":"ead6c6adb34658e7adb9327af0d94c4acf5cf367c59e2ab01517ca1cbb956bb9"} Sep 10 15:52:03.436520 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:03.436485 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d4l8v" event={"ID":"404be92e-4d9e-4070-a64f-ccf91ba25b8c","Type":"ContainerStarted","Data":"1b93424a216ebd703b4932a7282cba9f33c8f02e76acf53bb356b27f44cd0bdf"} Sep 10 15:52:03.455667 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:03.455609 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d4l8v" podStartSLOduration=2.459612753 podStartE2EDuration="3.455591281s" totalImagesPullingTime="995.978528ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:52:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:52:03.452319744 +0000 UTC m=+600.427579698" watchObservedRunningTime="2026-09-10 15:52:03.455591281 +0000 UTC m=+600.430851233" Sep 10 15:52:09.388151 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:09.388113 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:52:11.280083 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:11.280046 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d4l8v" Sep 10 15:52:11.280546 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:11.280118 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d4l8v" Sep 10 15:52:11.292706 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:11.292684 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d4l8v" Sep 10 15:52:11.490861 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:11.490832 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-d4l8v" Sep 10 15:52:16.792532 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:16.792497 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:52:22.012498 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:22.012460 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:52:26.990339 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:26.990263 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:52:54.959794 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:54.959758 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-v4knc"] Sep 10 15:52:54.970628 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:54.970579 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v4knc" Sep 10 15:52:54.972914 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:54.972890 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 10 15:52:54.974076 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:54.974055 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-v4knc"] Sep 10 15:52:55.061083 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:55.061054 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:52:55.080291 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:55.080262 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df2ec437-8304-45bd-80c7-a4f6eaa67e33-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v4knc\" (UID: \"df2ec437-8304-45bd-80c7-a4f6eaa67e33\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v4knc" Sep 10 15:52:55.080459 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:55.080334 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btfqh\" (UniqueName: \"kubernetes.io/projected/df2ec437-8304-45bd-80c7-a4f6eaa67e33-kube-api-access-btfqh\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v4knc\" (UID: \"df2ec437-8304-45bd-80c7-a4f6eaa67e33\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v4knc" Sep 10 15:52:55.080459 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:55.080364 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df2ec437-8304-45bd-80c7-a4f6eaa67e33-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v4knc\" (UID: \"df2ec437-8304-45bd-80c7-a4f6eaa67e33\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v4knc" Sep 10 15:52:55.080459 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:55.080443 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df2ec437-8304-45bd-80c7-a4f6eaa67e33-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v4knc\" (UID: \"df2ec437-8304-45bd-80c7-a4f6eaa67e33\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v4knc" Sep 10 15:52:55.080633 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:55.080517 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df2ec437-8304-45bd-80c7-a4f6eaa67e33-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v4knc\" (UID: \"df2ec437-8304-45bd-80c7-a4f6eaa67e33\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v4knc" Sep 10 15:52:55.080633 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:55.080555 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df2ec437-8304-45bd-80c7-a4f6eaa67e33-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v4knc\" (UID: \"df2ec437-8304-45bd-80c7-a4f6eaa67e33\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v4knc" Sep 10 15:52:55.181202 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:55.181170 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df2ec437-8304-45bd-80c7-a4f6eaa67e33-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v4knc\" (UID: \"df2ec437-8304-45bd-80c7-a4f6eaa67e33\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v4knc" Sep 10 15:52:55.181327 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:55.181238 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btfqh\" (UniqueName: \"kubernetes.io/projected/df2ec437-8304-45bd-80c7-a4f6eaa67e33-kube-api-access-btfqh\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v4knc\" (UID: \"df2ec437-8304-45bd-80c7-a4f6eaa67e33\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v4knc" Sep 10 15:52:55.181327 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:55.181265 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df2ec437-8304-45bd-80c7-a4f6eaa67e33-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v4knc\" (UID: \"df2ec437-8304-45bd-80c7-a4f6eaa67e33\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v4knc" Sep 10 15:52:55.181327 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:55.181301 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df2ec437-8304-45bd-80c7-a4f6eaa67e33-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v4knc\" (UID: \"df2ec437-8304-45bd-80c7-a4f6eaa67e33\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v4knc" Sep 10 15:52:55.181524 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:55.181352 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df2ec437-8304-45bd-80c7-a4f6eaa67e33-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v4knc\" (UID: \"df2ec437-8304-45bd-80c7-a4f6eaa67e33\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v4knc" Sep 10 15:52:55.181524 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:55.181415 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df2ec437-8304-45bd-80c7-a4f6eaa67e33-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v4knc\" (UID: \"df2ec437-8304-45bd-80c7-a4f6eaa67e33\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v4knc" Sep 10 15:52:55.181625 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:55.181590 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df2ec437-8304-45bd-80c7-a4f6eaa67e33-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v4knc\" (UID: \"df2ec437-8304-45bd-80c7-a4f6eaa67e33\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v4knc" Sep 10 15:52:55.181721 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:55.181698 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df2ec437-8304-45bd-80c7-a4f6eaa67e33-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v4knc\" (UID: \"df2ec437-8304-45bd-80c7-a4f6eaa67e33\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v4knc" Sep 10 15:52:55.181804 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:55.181782 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df2ec437-8304-45bd-80c7-a4f6eaa67e33-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v4knc\" (UID: \"df2ec437-8304-45bd-80c7-a4f6eaa67e33\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v4knc" Sep 10 15:52:55.183361 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:55.183341 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df2ec437-8304-45bd-80c7-a4f6eaa67e33-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v4knc\" (UID: \"df2ec437-8304-45bd-80c7-a4f6eaa67e33\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v4knc" Sep 10 15:52:55.183720 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:55.183697 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df2ec437-8304-45bd-80c7-a4f6eaa67e33-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v4knc\" (UID: \"df2ec437-8304-45bd-80c7-a4f6eaa67e33\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v4knc" Sep 10 15:52:55.189113 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:55.189090 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btfqh\" (UniqueName: \"kubernetes.io/projected/df2ec437-8304-45bd-80c7-a4f6eaa67e33-kube-api-access-btfqh\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-v4knc\" (UID: \"df2ec437-8304-45bd-80c7-a4f6eaa67e33\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v4knc" Sep 10 15:52:55.282332 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:55.282256 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v4knc" Sep 10 15:52:55.554017 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:55.553988 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-v4knc"] Sep 10 15:52:55.554901 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:52:55.554879 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf2ec437_8304_45bd_80c7_a4f6eaa67e33.slice/crio-3d4a264ef7db66eecdb77307d3e18ef9baeef5fc38c6565d13b3aa577a710505 WatchSource:0}: Error finding container 3d4a264ef7db66eecdb77307d3e18ef9baeef5fc38c6565d13b3aa577a710505: Status 404 returned error can't find the container with id 3d4a264ef7db66eecdb77307d3e18ef9baeef5fc38c6565d13b3aa577a710505 Sep 10 15:52:55.556117 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:55.556103 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 15:52:55.710023 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:55.709992 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v4knc" event={"ID":"df2ec437-8304-45bd-80c7-a4f6eaa67e33","Type":"ContainerStarted","Data":"3d4a264ef7db66eecdb77307d3e18ef9baeef5fc38c6565d13b3aa577a710505"} Sep 10 15:52:56.716635 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:56.716600 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v4knc" event={"ID":"df2ec437-8304-45bd-80c7-a4f6eaa67e33","Type":"ContainerStarted","Data":"d93016622ba1c1aeb74774eec4db80e585b1829823ce269cda1ed8d164b10a03"} Sep 10 15:52:56.734116 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:56.734068 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v4knc" podStartSLOduration=2.555605987 podStartE2EDuration="2.734051605s" totalImagesPullingTime="178.445618ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:52:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:52:56.733049432 +0000 UTC m=+653.708309384" watchObservedRunningTime="2026-09-10 15:52:56.734051605 +0000 UTC m=+653.709311558" Sep 10 15:52:59.920028 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:52:59.919985 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:53:00.644744 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:53:00.644710 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:53:05.282740 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:53:05.282705 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v4knc" Sep 10 15:53:05.283143 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:53:05.282764 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v4knc" Sep 10 15:53:05.295187 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:53:05.295162 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v4knc" Sep 10 15:53:05.774215 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:53:05.774186 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-v4knc" Sep 10 15:53:14.809164 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:53:14.809118 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:53:25.497535 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:53:25.497499 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:53:52.642726 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:53:52.642690 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:08.367306 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:08.367214 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:08.466902 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:08.466866 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:09.481474 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:09.481425 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:09.582162 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:09.582129 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:27.116172 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:27.116137 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:27.217578 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:27.217551 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:27.404755 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:27.404727 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:27.515218 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:27.515180 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:27.726570 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:27.726478 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:28.124169 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:28.124097 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:28.768305 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:28.768266 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:28.867473 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:28.867439 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:29.274203 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:29.274171 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:31.263076 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:31.263040 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:31.358342 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:31.358306 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:32.669081 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:32.669050 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-83dcaf-9b598689c-gmltw"] Sep 10 15:54:32.673614 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:32.673588 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83dcaf-9b598689c-gmltw" Sep 10 15:54:32.675618 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:32.675599 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-83dcaf-serving-cert\"" Sep 10 15:54:32.680693 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:32.680672 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-83dcaf-9b598689c-gmltw"] Sep 10 15:54:32.696925 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:32.696898 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:32.791954 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:32.791923 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69jvv\" (UniqueName: \"kubernetes.io/projected/beee0095-f836-4de6-990e-8e4515e17a74-kube-api-access-69jvv\") pod \"maas-api-e2e-inf-a-83dcaf-9b598689c-gmltw\" (UID: \"beee0095-f836-4de6-990e-8e4515e17a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83dcaf-9b598689c-gmltw" Sep 10 15:54:32.791954 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:32.791958 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/beee0095-f836-4de6-990e-8e4515e17a74-metrics-certs\") pod \"maas-api-e2e-inf-a-83dcaf-9b598689c-gmltw\" (UID: \"beee0095-f836-4de6-990e-8e4515e17a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83dcaf-9b598689c-gmltw" Sep 10 15:54:32.792198 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:32.792018 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/beee0095-f836-4de6-990e-8e4515e17a74-maas-api-tls\") pod \"maas-api-e2e-inf-a-83dcaf-9b598689c-gmltw\" (UID: \"beee0095-f836-4de6-990e-8e4515e17a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83dcaf-9b598689c-gmltw" Sep 10 15:54:32.792198 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:32.792051 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/beee0095-f836-4de6-990e-8e4515e17a74-openshift-service-ca\") pod \"maas-api-e2e-inf-a-83dcaf-9b598689c-gmltw\" (UID: \"beee0095-f836-4de6-990e-8e4515e17a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83dcaf-9b598689c-gmltw" Sep 10 15:54:32.798916 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:32.798875 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:32.892859 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:32.892832 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69jvv\" (UniqueName: \"kubernetes.io/projected/beee0095-f836-4de6-990e-8e4515e17a74-kube-api-access-69jvv\") pod \"maas-api-e2e-inf-a-83dcaf-9b598689c-gmltw\" (UID: \"beee0095-f836-4de6-990e-8e4515e17a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83dcaf-9b598689c-gmltw" Sep 10 15:54:32.893002 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:32.892863 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/beee0095-f836-4de6-990e-8e4515e17a74-metrics-certs\") pod \"maas-api-e2e-inf-a-83dcaf-9b598689c-gmltw\" (UID: \"beee0095-f836-4de6-990e-8e4515e17a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83dcaf-9b598689c-gmltw" Sep 10 15:54:32.893002 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:32.892938 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/beee0095-f836-4de6-990e-8e4515e17a74-maas-api-tls\") pod \"maas-api-e2e-inf-a-83dcaf-9b598689c-gmltw\" (UID: \"beee0095-f836-4de6-990e-8e4515e17a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83dcaf-9b598689c-gmltw" Sep 10 15:54:32.893002 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:32.892969 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/beee0095-f836-4de6-990e-8e4515e17a74-openshift-service-ca\") pod \"maas-api-e2e-inf-a-83dcaf-9b598689c-gmltw\" (UID: \"beee0095-f836-4de6-990e-8e4515e17a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83dcaf-9b598689c-gmltw" Sep 10 15:54:32.893121 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:54:32.893075 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-83dcaf-serving-cert: secret "maas-api-e2e-inf-a-83dcaf-serving-cert" not found Sep 10 15:54:32.893174 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:54:32.893162 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/beee0095-f836-4de6-990e-8e4515e17a74-maas-api-tls podName:beee0095-f836-4de6-990e-8e4515e17a74 nodeName:}" failed. No retries permitted until 2026-09-10 15:54:33.393139439 +0000 UTC m=+750.368399374 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/beee0095-f836-4de6-990e-8e4515e17a74-maas-api-tls") pod "maas-api-e2e-inf-a-83dcaf-9b598689c-gmltw" (UID: "beee0095-f836-4de6-990e-8e4515e17a74") : secret "maas-api-e2e-inf-a-83dcaf-serving-cert" not found Sep 10 15:54:32.893674 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:32.893653 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/beee0095-f836-4de6-990e-8e4515e17a74-openshift-service-ca\") pod \"maas-api-e2e-inf-a-83dcaf-9b598689c-gmltw\" (UID: \"beee0095-f836-4de6-990e-8e4515e17a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83dcaf-9b598689c-gmltw" Sep 10 15:54:32.895258 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:32.895234 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/beee0095-f836-4de6-990e-8e4515e17a74-metrics-certs\") pod \"maas-api-e2e-inf-a-83dcaf-9b598689c-gmltw\" (UID: \"beee0095-f836-4de6-990e-8e4515e17a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83dcaf-9b598689c-gmltw" Sep 10 15:54:32.901097 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:32.901078 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69jvv\" (UniqueName: \"kubernetes.io/projected/beee0095-f836-4de6-990e-8e4515e17a74-kube-api-access-69jvv\") pod \"maas-api-e2e-inf-a-83dcaf-9b598689c-gmltw\" (UID: \"beee0095-f836-4de6-990e-8e4515e17a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83dcaf-9b598689c-gmltw" Sep 10 15:54:33.186029 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:33.185995 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:33.288960 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:33.288921 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:33.393117 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:33.393088 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:33.398463 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:33.398435 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/beee0095-f836-4de6-990e-8e4515e17a74-maas-api-tls\") pod \"maas-api-e2e-inf-a-83dcaf-9b598689c-gmltw\" (UID: \"beee0095-f836-4de6-990e-8e4515e17a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83dcaf-9b598689c-gmltw" Sep 10 15:54:33.400963 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:33.400911 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/beee0095-f836-4de6-990e-8e4515e17a74-maas-api-tls\") pod \"maas-api-e2e-inf-a-83dcaf-9b598689c-gmltw\" (UID: \"beee0095-f836-4de6-990e-8e4515e17a74\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83dcaf-9b598689c-gmltw" Sep 10 15:54:33.403586 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:33.403563 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-83dcaf-698b75cf54-jz542"] Sep 10 15:54:33.407159 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:33.407144 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-83dcaf-698b75cf54-jz542" Sep 10 15:54:33.409710 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:33.409688 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-83dcaf-dockercfg-q7dnf\"" Sep 10 15:54:33.409934 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:33.409915 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-83dcaf\"" Sep 10 15:54:33.419818 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:33.419795 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-83dcaf-698b75cf54-jz542"] Sep 10 15:54:33.499939 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:33.499862 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwrlr\" (UniqueName: \"kubernetes.io/projected/2d82f2a2-f582-431a-b837-17ffdf28c815-kube-api-access-rwrlr\") pod \"payload-pre-processing-e2e-inf-a-83dcaf-698b75cf54-jz542\" (UID: \"2d82f2a2-f582-431a-b837-17ffdf28c815\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-83dcaf-698b75cf54-jz542" Sep 10 15:54:33.500087 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:33.499983 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d82f2a2-f582-431a-b837-17ffdf28c815-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-83dcaf-698b75cf54-jz542\" (UID: \"2d82f2a2-f582-431a-b837-17ffdf28c815\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-83dcaf-698b75cf54-jz542" Sep 10 15:54:33.587229 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:33.587193 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83dcaf-9b598689c-gmltw" Sep 10 15:54:33.601570 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:33.601539 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwrlr\" (UniqueName: \"kubernetes.io/projected/2d82f2a2-f582-431a-b837-17ffdf28c815-kube-api-access-rwrlr\") pod \"payload-pre-processing-e2e-inf-a-83dcaf-698b75cf54-jz542\" (UID: \"2d82f2a2-f582-431a-b837-17ffdf28c815\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-83dcaf-698b75cf54-jz542" Sep 10 15:54:33.601678 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:33.601643 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d82f2a2-f582-431a-b837-17ffdf28c815-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-83dcaf-698b75cf54-jz542\" (UID: \"2d82f2a2-f582-431a-b837-17ffdf28c815\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-83dcaf-698b75cf54-jz542" Sep 10 15:54:33.602509 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:33.602485 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d82f2a2-f582-431a-b837-17ffdf28c815-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-83dcaf-698b75cf54-jz542\" (UID: \"2d82f2a2-f582-431a-b837-17ffdf28c815\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-83dcaf-698b75cf54-jz542" Sep 10 15:54:33.612670 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:33.612648 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwrlr\" (UniqueName: \"kubernetes.io/projected/2d82f2a2-f582-431a-b837-17ffdf28c815-kube-api-access-rwrlr\") pod \"payload-pre-processing-e2e-inf-a-83dcaf-698b75cf54-jz542\" (UID: \"2d82f2a2-f582-431a-b837-17ffdf28c815\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-83dcaf-698b75cf54-jz542" Sep 10 15:54:33.718961 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:33.718921 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-83dcaf-698b75cf54-jz542" Sep 10 15:54:33.866305 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:33.866271 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-83dcaf-9b598689c-gmltw"] Sep 10 15:54:33.867257 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:54:33.867233 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeee0095_f836_4de6_990e_8e4515e17a74.slice/crio-07c14220ed09f41286220347baad779bf0e466e7af18b2e34041e05f124ed63c WatchSource:0}: Error finding container 07c14220ed09f41286220347baad779bf0e466e7af18b2e34041e05f124ed63c: Status 404 returned error can't find the container with id 07c14220ed09f41286220347baad779bf0e466e7af18b2e34041e05f124ed63c Sep 10 15:54:33.991791 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:54:33.991750 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d82f2a2_f582_431a_b837_17ffdf28c815.slice/crio-a5c97455fc9ea01ab6495cecab9fee05d6675f3ecd13524e53364ad8a5d9b855 WatchSource:0}: Error finding container a5c97455fc9ea01ab6495cecab9fee05d6675f3ecd13524e53364ad8a5d9b855: Status 404 returned error can't find the container with id a5c97455fc9ea01ab6495cecab9fee05d6675f3ecd13524e53364ad8a5d9b855 Sep 10 15:54:33.993029 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:33.993002 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-83dcaf-698b75cf54-jz542"] Sep 10 15:54:34.233154 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:34.233112 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-83dcaf-698b75cf54-jz542" event={"ID":"2d82f2a2-f582-431a-b837-17ffdf28c815","Type":"ContainerStarted","Data":"a5c97455fc9ea01ab6495cecab9fee05d6675f3ecd13524e53364ad8a5d9b855"} Sep 10 15:54:34.234292 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:34.234268 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83dcaf-9b598689c-gmltw" event={"ID":"beee0095-f836-4de6-990e-8e4515e17a74","Type":"ContainerStarted","Data":"07c14220ed09f41286220347baad779bf0e466e7af18b2e34041e05f124ed63c"} Sep 10 15:54:35.243095 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:35.243061 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83dcaf-9b598689c-gmltw" event={"ID":"beee0095-f836-4de6-990e-8e4515e17a74","Type":"ContainerStarted","Data":"965a1350d417fd7faeb8b3f907b9c8662d7cec99043f31bda84ab9e719c2cff2"} Sep 10 15:54:35.243554 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:35.243168 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83dcaf-9b598689c-gmltw" Sep 10 15:54:35.262252 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:35.262203 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83dcaf-9b598689c-gmltw" podStartSLOduration=2.887647733 podStartE2EDuration="3.262188083s" totalImagesPullingTime="374.54035ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:54:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:54:35.259831581 +0000 UTC m=+752.235091535" watchObservedRunningTime="2026-09-10 15:54:35.262188083 +0000 UTC m=+752.237448036" Sep 10 15:54:35.590269 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:35.590169 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:35.687186 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:35.687154 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:35.812643 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:35.812622 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:36.086006 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:36.085959 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:36.095289 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:36.095222 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78"] Sep 10 15:54:36.099691 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:36.099667 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" Sep 10 15:54:36.102325 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:36.102158 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-54da81e9-openshift-default-dockercfg-vc7z4\"" Sep 10 15:54:36.123613 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:36.123586 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78"] Sep 10 15:54:36.186425 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:36.186338 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:36.225803 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:36.225753 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c041ae3-1bbe-4089-9c98-3bc78603692a-istio-podinfo\") pod \"e2e-ait-54da81e9-openshift-default-54494d9994-ttx78\" (UID: \"7c041ae3-1bbe-4089-9c98-3bc78603692a\") " pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" Sep 10 15:54:36.225966 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:36.225815 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c041ae3-1bbe-4089-9c98-3bc78603692a-istiod-ca-cert\") pod \"e2e-ait-54da81e9-openshift-default-54494d9994-ttx78\" (UID: \"7c041ae3-1bbe-4089-9c98-3bc78603692a\") " pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" Sep 10 15:54:36.225966 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:36.225846 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c041ae3-1bbe-4089-9c98-3bc78603692a-workload-socket\") pod \"e2e-ait-54da81e9-openshift-default-54494d9994-ttx78\" (UID: \"7c041ae3-1bbe-4089-9c98-3bc78603692a\") " pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" Sep 10 15:54:36.225966 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:36.225882 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrgnj\" (UniqueName: \"kubernetes.io/projected/7c041ae3-1bbe-4089-9c98-3bc78603692a-kube-api-access-qrgnj\") pod \"e2e-ait-54da81e9-openshift-default-54494d9994-ttx78\" (UID: \"7c041ae3-1bbe-4089-9c98-3bc78603692a\") " pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" Sep 10 15:54:36.225966 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:36.225941 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c041ae3-1bbe-4089-9c98-3bc78603692a-credential-socket\") pod \"e2e-ait-54da81e9-openshift-default-54494d9994-ttx78\" (UID: \"7c041ae3-1bbe-4089-9c98-3bc78603692a\") " pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" Sep 10 15:54:36.226125 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:36.225980 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c041ae3-1bbe-4089-9c98-3bc78603692a-istio-data\") pod \"e2e-ait-54da81e9-openshift-default-54494d9994-ttx78\" (UID: \"7c041ae3-1bbe-4089-9c98-3bc78603692a\") " pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" Sep 10 15:54:36.226125 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:36.226021 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c041ae3-1bbe-4089-9c98-3bc78603692a-istio-token\") pod \"e2e-ait-54da81e9-openshift-default-54494d9994-ttx78\" (UID: \"7c041ae3-1bbe-4089-9c98-3bc78603692a\") " pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" Sep 10 15:54:36.226125 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:36.226041 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c041ae3-1bbe-4089-9c98-3bc78603692a-workload-certs\") pod \"e2e-ait-54da81e9-openshift-default-54494d9994-ttx78\" (UID: \"7c041ae3-1bbe-4089-9c98-3bc78603692a\") " pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" Sep 10 15:54:36.226125 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:36.226101 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c041ae3-1bbe-4089-9c98-3bc78603692a-istio-envoy\") pod \"e2e-ait-54da81e9-openshift-default-54494d9994-ttx78\" (UID: \"7c041ae3-1bbe-4089-9c98-3bc78603692a\") " pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" Sep 10 15:54:36.254728 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:36.254697 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-83dcaf-698b75cf54-jz542" event={"ID":"2d82f2a2-f582-431a-b837-17ffdf28c815","Type":"ContainerStarted","Data":"b354bb49b803898a0fed9d9bc9f98e2d9de05a84d361bf6a64d776a40ef11e64"} Sep 10 15:54:36.255084 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:36.254761 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-83dcaf-698b75cf54-jz542" Sep 10 15:54:36.281363 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:36.281309 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-83dcaf-698b75cf54-jz542" podStartSLOduration=1.495170248 podStartE2EDuration="3.281293846s" totalImagesPullingTime="1.786123598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:54:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:54:36.279298514 +0000 UTC m=+753.254558467" watchObservedRunningTime="2026-09-10 15:54:36.281293846 +0000 UTC m=+753.256553799" Sep 10 15:54:36.327338 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:36.327307 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c041ae3-1bbe-4089-9c98-3bc78603692a-istio-token\") pod \"e2e-ait-54da81e9-openshift-default-54494d9994-ttx78\" (UID: \"7c041ae3-1bbe-4089-9c98-3bc78603692a\") " pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" Sep 10 15:54:36.327505 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:36.327348 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c041ae3-1bbe-4089-9c98-3bc78603692a-workload-certs\") pod \"e2e-ait-54da81e9-openshift-default-54494d9994-ttx78\" (UID: \"7c041ae3-1bbe-4089-9c98-3bc78603692a\") " pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" Sep 10 15:54:36.327505 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:36.327430 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c041ae3-1bbe-4089-9c98-3bc78603692a-istio-envoy\") pod \"e2e-ait-54da81e9-openshift-default-54494d9994-ttx78\" (UID: \"7c041ae3-1bbe-4089-9c98-3bc78603692a\") " pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" Sep 10 15:54:36.327627 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:36.327538 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c041ae3-1bbe-4089-9c98-3bc78603692a-istio-podinfo\") pod \"e2e-ait-54da81e9-openshift-default-54494d9994-ttx78\" (UID: \"7c041ae3-1bbe-4089-9c98-3bc78603692a\") " pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" Sep 10 15:54:36.327627 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:36.327574 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c041ae3-1bbe-4089-9c98-3bc78603692a-istiod-ca-cert\") pod \"e2e-ait-54da81e9-openshift-default-54494d9994-ttx78\" (UID: \"7c041ae3-1bbe-4089-9c98-3bc78603692a\") " pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" Sep 10 15:54:36.327627 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:36.327600 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c041ae3-1bbe-4089-9c98-3bc78603692a-workload-socket\") pod \"e2e-ait-54da81e9-openshift-default-54494d9994-ttx78\" (UID: \"7c041ae3-1bbe-4089-9c98-3bc78603692a\") " pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" Sep 10 15:54:36.327783 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:36.327640 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrgnj\" (UniqueName: \"kubernetes.io/projected/7c041ae3-1bbe-4089-9c98-3bc78603692a-kube-api-access-qrgnj\") pod \"e2e-ait-54da81e9-openshift-default-54494d9994-ttx78\" (UID: \"7c041ae3-1bbe-4089-9c98-3bc78603692a\") " pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" Sep 10 15:54:36.327783 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:36.327719 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c041ae3-1bbe-4089-9c98-3bc78603692a-credential-socket\") pod \"e2e-ait-54da81e9-openshift-default-54494d9994-ttx78\" (UID: \"7c041ae3-1bbe-4089-9c98-3bc78603692a\") " pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" Sep 10 15:54:36.327783 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:36.327757 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c041ae3-1bbe-4089-9c98-3bc78603692a-istio-data\") pod \"e2e-ait-54da81e9-openshift-default-54494d9994-ttx78\" (UID: \"7c041ae3-1bbe-4089-9c98-3bc78603692a\") " pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" Sep 10 15:54:36.327944 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:36.327863 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c041ae3-1bbe-4089-9c98-3bc78603692a-workload-certs\") pod \"e2e-ait-54da81e9-openshift-default-54494d9994-ttx78\" (UID: \"7c041ae3-1bbe-4089-9c98-3bc78603692a\") " pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" Sep 10 15:54:36.328291 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:36.328085 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c041ae3-1bbe-4089-9c98-3bc78603692a-istio-data\") pod \"e2e-ait-54da81e9-openshift-default-54494d9994-ttx78\" (UID: \"7c041ae3-1bbe-4089-9c98-3bc78603692a\") " pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" Sep 10 15:54:36.328471 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:36.328447 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c041ae3-1bbe-4089-9c98-3bc78603692a-credential-socket\") pod \"e2e-ait-54da81e9-openshift-default-54494d9994-ttx78\" (UID: \"7c041ae3-1bbe-4089-9c98-3bc78603692a\") " pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" Sep 10 15:54:36.328554 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:36.328502 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c041ae3-1bbe-4089-9c98-3bc78603692a-workload-socket\") pod \"e2e-ait-54da81e9-openshift-default-54494d9994-ttx78\" (UID: \"7c041ae3-1bbe-4089-9c98-3bc78603692a\") " pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" Sep 10 15:54:36.328554 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:36.328521 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c041ae3-1bbe-4089-9c98-3bc78603692a-istiod-ca-cert\") pod \"e2e-ait-54da81e9-openshift-default-54494d9994-ttx78\" (UID: \"7c041ae3-1bbe-4089-9c98-3bc78603692a\") " pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" Sep 10 15:54:36.329918 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:36.329892 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c041ae3-1bbe-4089-9c98-3bc78603692a-istio-envoy\") pod \"e2e-ait-54da81e9-openshift-default-54494d9994-ttx78\" (UID: \"7c041ae3-1bbe-4089-9c98-3bc78603692a\") " pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" Sep 10 15:54:36.330160 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:36.330141 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c041ae3-1bbe-4089-9c98-3bc78603692a-istio-podinfo\") pod \"e2e-ait-54da81e9-openshift-default-54494d9994-ttx78\" (UID: \"7c041ae3-1bbe-4089-9c98-3bc78603692a\") " pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" Sep 10 15:54:36.340552 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:36.340502 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c041ae3-1bbe-4089-9c98-3bc78603692a-istio-token\") pod \"e2e-ait-54da81e9-openshift-default-54494d9994-ttx78\" (UID: \"7c041ae3-1bbe-4089-9c98-3bc78603692a\") " pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" Sep 10 15:54:36.341930 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:36.341910 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrgnj\" (UniqueName: \"kubernetes.io/projected/7c041ae3-1bbe-4089-9c98-3bc78603692a-kube-api-access-qrgnj\") pod \"e2e-ait-54da81e9-openshift-default-54494d9994-ttx78\" (UID: \"7c041ae3-1bbe-4089-9c98-3bc78603692a\") " pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" Sep 10 15:54:36.381428 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:36.381403 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:36.412437 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:36.412414 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" Sep 10 15:54:36.645310 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:36.645213 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:36.704254 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:36.704227 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78"] Sep 10 15:54:36.705958 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:54:36.705917 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c041ae3_1bbe_4089_9c98_3bc78603692a.slice/crio-42c49452883515a20268050e496c6ef1788b72b2f821d0c9138a183b308d9108 WatchSource:0}: Error finding container 42c49452883515a20268050e496c6ef1788b72b2f821d0c9138a183b308d9108: Status 404 returned error can't find the container with id 42c49452883515a20268050e496c6ef1788b72b2f821d0c9138a183b308d9108 Sep 10 15:54:36.709458 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:36.708766 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 15:54:36.709458 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:36.708848 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 15:54:36.709458 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:36.708887 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 15:54:37.132678 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:37.132649 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:37.261620 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:37.261580 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" event={"ID":"7c041ae3-1bbe-4089-9c98-3bc78603692a","Type":"ContainerStarted","Data":"7e5c643d42091de655d8e836f04628c73c8f482f6bd4381aa82f6ec69e07a2cd"} Sep 10 15:54:37.262053 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:37.261675 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" event={"ID":"7c041ae3-1bbe-4089-9c98-3bc78603692a","Type":"ContainerStarted","Data":"42c49452883515a20268050e496c6ef1788b72b2f821d0c9138a183b308d9108"} Sep 10 15:54:37.284489 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:37.284445 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" podStartSLOduration=1.284430963 podStartE2EDuration="1.284430963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:54:37.281747099 +0000 UTC m=+754.257007063" watchObservedRunningTime="2026-09-10 15:54:37.284430963 +0000 UTC m=+754.259690918" Sep 10 15:54:37.413513 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:37.413437 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" Sep 10 15:54:37.418492 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:37.418461 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" Sep 10 15:54:38.267080 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:38.267045 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" Sep 10 15:54:38.268110 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:38.268092 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" Sep 10 15:54:38.611115 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:38.611042 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:38.699957 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:38.699925 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:39.460819 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:39.460785 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:39.556828 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:39.556797 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:39.795571 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:39.795496 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-54da81e9-5f7bd984f8-n74c4"] Sep 10 15:54:39.799263 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:39.799244 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-54da81e9-5f7bd984f8-n74c4" Sep 10 15:54:39.801657 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:39.801632 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-54da81e9-serving-cert\"" Sep 10 15:54:39.812988 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:39.812967 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-54da81e9-5f7bd984f8-n74c4"] Sep 10 15:54:39.889837 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:39.889807 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:39.963327 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:39.963294 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbcw9\" (UniqueName: \"kubernetes.io/projected/291c4fec-1bda-4599-b9be-4d1d9d7a833d-kube-api-access-cbcw9\") pod \"maas-api-e2e-ait-54da81e9-5f7bd984f8-n74c4\" (UID: \"291c4fec-1bda-4599-b9be-4d1d9d7a833d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-54da81e9-5f7bd984f8-n74c4" Sep 10 15:54:39.963560 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:39.963410 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/291c4fec-1bda-4599-b9be-4d1d9d7a833d-maas-api-tls\") pod \"maas-api-e2e-ait-54da81e9-5f7bd984f8-n74c4\" (UID: \"291c4fec-1bda-4599-b9be-4d1d9d7a833d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-54da81e9-5f7bd984f8-n74c4" Sep 10 15:54:39.963560 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:39.963455 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/291c4fec-1bda-4599-b9be-4d1d9d7a833d-openshift-service-ca\") pod \"maas-api-e2e-ait-54da81e9-5f7bd984f8-n74c4\" (UID: \"291c4fec-1bda-4599-b9be-4d1d9d7a833d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-54da81e9-5f7bd984f8-n74c4" Sep 10 15:54:39.963560 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:39.963482 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/291c4fec-1bda-4599-b9be-4d1d9d7a833d-metrics-certs\") pod \"maas-api-e2e-ait-54da81e9-5f7bd984f8-n74c4\" (UID: \"291c4fec-1bda-4599-b9be-4d1d9d7a833d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-54da81e9-5f7bd984f8-n74c4" Sep 10 15:54:39.988171 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:39.988140 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:40.064649 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:40.064577 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/291c4fec-1bda-4599-b9be-4d1d9d7a833d-maas-api-tls\") pod \"maas-api-e2e-ait-54da81e9-5f7bd984f8-n74c4\" (UID: \"291c4fec-1bda-4599-b9be-4d1d9d7a833d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-54da81e9-5f7bd984f8-n74c4" Sep 10 15:54:40.064649 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:40.064612 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/291c4fec-1bda-4599-b9be-4d1d9d7a833d-openshift-service-ca\") pod \"maas-api-e2e-ait-54da81e9-5f7bd984f8-n74c4\" (UID: \"291c4fec-1bda-4599-b9be-4d1d9d7a833d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-54da81e9-5f7bd984f8-n74c4" Sep 10 15:54:40.064649 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:40.064636 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/291c4fec-1bda-4599-b9be-4d1d9d7a833d-metrics-certs\") pod \"maas-api-e2e-ait-54da81e9-5f7bd984f8-n74c4\" (UID: \"291c4fec-1bda-4599-b9be-4d1d9d7a833d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-54da81e9-5f7bd984f8-n74c4" Sep 10 15:54:40.064906 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:40.064700 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbcw9\" (UniqueName: \"kubernetes.io/projected/291c4fec-1bda-4599-b9be-4d1d9d7a833d-kube-api-access-cbcw9\") pod \"maas-api-e2e-ait-54da81e9-5f7bd984f8-n74c4\" (UID: \"291c4fec-1bda-4599-b9be-4d1d9d7a833d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-54da81e9-5f7bd984f8-n74c4" Sep 10 15:54:40.064906 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:54:40.064745 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-54da81e9-serving-cert: secret "maas-api-e2e-ait-54da81e9-serving-cert" not found Sep 10 15:54:40.064906 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:54:40.064816 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/291c4fec-1bda-4599-b9be-4d1d9d7a833d-maas-api-tls podName:291c4fec-1bda-4599-b9be-4d1d9d7a833d nodeName:}" failed. No retries permitted until 2026-09-10 15:54:40.564794812 +0000 UTC m=+757.540054756 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/291c4fec-1bda-4599-b9be-4d1d9d7a833d-maas-api-tls") pod "maas-api-e2e-ait-54da81e9-5f7bd984f8-n74c4" (UID: "291c4fec-1bda-4599-b9be-4d1d9d7a833d") : secret "maas-api-e2e-ait-54da81e9-serving-cert" not found Sep 10 15:54:40.065285 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:40.065264 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/291c4fec-1bda-4599-b9be-4d1d9d7a833d-openshift-service-ca\") pod \"maas-api-e2e-ait-54da81e9-5f7bd984f8-n74c4\" (UID: \"291c4fec-1bda-4599-b9be-4d1d9d7a833d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-54da81e9-5f7bd984f8-n74c4" Sep 10 15:54:40.067028 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:40.067008 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/291c4fec-1bda-4599-b9be-4d1d9d7a833d-metrics-certs\") pod \"maas-api-e2e-ait-54da81e9-5f7bd984f8-n74c4\" (UID: \"291c4fec-1bda-4599-b9be-4d1d9d7a833d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-54da81e9-5f7bd984f8-n74c4" Sep 10 15:54:40.076539 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:40.076513 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbcw9\" (UniqueName: \"kubernetes.io/projected/291c4fec-1bda-4599-b9be-4d1d9d7a833d-kube-api-access-cbcw9\") pod \"maas-api-e2e-ait-54da81e9-5f7bd984f8-n74c4\" (UID: \"291c4fec-1bda-4599-b9be-4d1d9d7a833d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-54da81e9-5f7bd984f8-n74c4" Sep 10 15:54:40.156083 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:40.156058 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:40.569791 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:40.569758 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/291c4fec-1bda-4599-b9be-4d1d9d7a833d-maas-api-tls\") pod \"maas-api-e2e-ait-54da81e9-5f7bd984f8-n74c4\" (UID: \"291c4fec-1bda-4599-b9be-4d1d9d7a833d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-54da81e9-5f7bd984f8-n74c4" Sep 10 15:54:40.572236 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:40.572214 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/291c4fec-1bda-4599-b9be-4d1d9d7a833d-maas-api-tls\") pod \"maas-api-e2e-ait-54da81e9-5f7bd984f8-n74c4\" (UID: \"291c4fec-1bda-4599-b9be-4d1d9d7a833d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-54da81e9-5f7bd984f8-n74c4" Sep 10 15:54:40.638744 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:40.638716 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:40.709986 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:40.709963 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-54da81e9-5f7bd984f8-n74c4" Sep 10 15:54:40.742727 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:40.742697 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:40.984049 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:40.984025 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-54da81e9-5f7bd984f8-n74c4"] Sep 10 15:54:40.985213 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:54:40.985188 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod291c4fec_1bda_4599_b9be_4d1d9d7a833d.slice/crio-94b5ab8118c3a9e209397f192772e2460f17858958fbc6126d91902482c3edfc WatchSource:0}: Error finding container 94b5ab8118c3a9e209397f192772e2460f17858958fbc6126d91902482c3edfc: Status 404 returned error can't find the container with id 94b5ab8118c3a9e209397f192772e2460f17858958fbc6126d91902482c3edfc Sep 10 15:54:41.078136 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:41.078106 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-87a024-674c46cd56-r7pwq"] Sep 10 15:54:41.084615 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:41.084592 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-87a024-674c46cd56-r7pwq" Sep 10 15:54:41.087204 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:41.087181 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-87a024-serving-cert\"" Sep 10 15:54:41.090494 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:41.090468 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-87a024-674c46cd56-r7pwq"] Sep 10 15:54:41.114407 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:41.114365 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:41.173778 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:41.173753 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:41.176741 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:41.176721 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fff6751f-b96a-42ec-b10f-89ba9ec4c0f2-openshift-service-ca\") pod \"maas-api-e2e-autores-87a024-674c46cd56-r7pwq\" (UID: \"fff6751f-b96a-42ec-b10f-89ba9ec4c0f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-87a024-674c46cd56-r7pwq" Sep 10 15:54:41.176816 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:41.176750 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fff6751f-b96a-42ec-b10f-89ba9ec4c0f2-maas-api-tls\") pod \"maas-api-e2e-autores-87a024-674c46cd56-r7pwq\" (UID: \"fff6751f-b96a-42ec-b10f-89ba9ec4c0f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-87a024-674c46cd56-r7pwq" Sep 10 15:54:41.176855 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:41.176816 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9bh4\" (UniqueName: \"kubernetes.io/projected/fff6751f-b96a-42ec-b10f-89ba9ec4c0f2-kube-api-access-c9bh4\") pod \"maas-api-e2e-autores-87a024-674c46cd56-r7pwq\" (UID: \"fff6751f-b96a-42ec-b10f-89ba9ec4c0f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-87a024-674c46cd56-r7pwq" Sep 10 15:54:41.176901 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:41.176887 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fff6751f-b96a-42ec-b10f-89ba9ec4c0f2-metrics-certs\") pod \"maas-api-e2e-autores-87a024-674c46cd56-r7pwq\" (UID: \"fff6751f-b96a-42ec-b10f-89ba9ec4c0f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-87a024-674c46cd56-r7pwq" Sep 10 15:54:41.261948 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:41.261877 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83dcaf-9b598689c-gmltw" Sep 10 15:54:41.277789 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:41.277758 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fff6751f-b96a-42ec-b10f-89ba9ec4c0f2-metrics-certs\") pod \"maas-api-e2e-autores-87a024-674c46cd56-r7pwq\" (UID: \"fff6751f-b96a-42ec-b10f-89ba9ec4c0f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-87a024-674c46cd56-r7pwq" Sep 10 15:54:41.277914 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:41.277898 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fff6751f-b96a-42ec-b10f-89ba9ec4c0f2-openshift-service-ca\") pod \"maas-api-e2e-autores-87a024-674c46cd56-r7pwq\" (UID: \"fff6751f-b96a-42ec-b10f-89ba9ec4c0f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-87a024-674c46cd56-r7pwq" Sep 10 15:54:41.277954 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:41.277917 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fff6751f-b96a-42ec-b10f-89ba9ec4c0f2-maas-api-tls\") pod \"maas-api-e2e-autores-87a024-674c46cd56-r7pwq\" (UID: \"fff6751f-b96a-42ec-b10f-89ba9ec4c0f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-87a024-674c46cd56-r7pwq" Sep 10 15:54:41.278002 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:41.277954 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9bh4\" (UniqueName: \"kubernetes.io/projected/fff6751f-b96a-42ec-b10f-89ba9ec4c0f2-kube-api-access-c9bh4\") pod \"maas-api-e2e-autores-87a024-674c46cd56-r7pwq\" (UID: \"fff6751f-b96a-42ec-b10f-89ba9ec4c0f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-87a024-674c46cd56-r7pwq" Sep 10 15:54:41.278094 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:54:41.278075 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-87a024-serving-cert: secret "maas-api-e2e-autores-87a024-serving-cert" not found Sep 10 15:54:41.278178 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:54:41.278166 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fff6751f-b96a-42ec-b10f-89ba9ec4c0f2-maas-api-tls podName:fff6751f-b96a-42ec-b10f-89ba9ec4c0f2 nodeName:}" failed. No retries permitted until 2026-09-10 15:54:41.778143884 +0000 UTC m=+758.753403831 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fff6751f-b96a-42ec-b10f-89ba9ec4c0f2-maas-api-tls") pod "maas-api-e2e-autores-87a024-674c46cd56-r7pwq" (UID: "fff6751f-b96a-42ec-b10f-89ba9ec4c0f2") : secret "maas-api-e2e-autores-87a024-serving-cert" not found Sep 10 15:54:41.278700 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:41.278675 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fff6751f-b96a-42ec-b10f-89ba9ec4c0f2-openshift-service-ca\") pod \"maas-api-e2e-autores-87a024-674c46cd56-r7pwq\" (UID: \"fff6751f-b96a-42ec-b10f-89ba9ec4c0f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-87a024-674c46cd56-r7pwq" Sep 10 15:54:41.280812 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:41.280786 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fff6751f-b96a-42ec-b10f-89ba9ec4c0f2-metrics-certs\") pod \"maas-api-e2e-autores-87a024-674c46cd56-r7pwq\" (UID: \"fff6751f-b96a-42ec-b10f-89ba9ec4c0f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-87a024-674c46cd56-r7pwq" Sep 10 15:54:41.287561 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:41.287488 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-54da81e9-5f7bd984f8-n74c4" event={"ID":"291c4fec-1bda-4599-b9be-4d1d9d7a833d","Type":"ContainerStarted","Data":"94b5ab8118c3a9e209397f192772e2460f17858958fbc6126d91902482c3edfc"} Sep 10 15:54:41.287668 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:41.287601 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9bh4\" (UniqueName: \"kubernetes.io/projected/fff6751f-b96a-42ec-b10f-89ba9ec4c0f2-kube-api-access-c9bh4\") pod \"maas-api-e2e-autores-87a024-674c46cd56-r7pwq\" (UID: \"fff6751f-b96a-42ec-b10f-89ba9ec4c0f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-87a024-674c46cd56-r7pwq" Sep 10 15:54:41.530345 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:41.530260 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:41.734188 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:41.734165 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:41.784048 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:41.784014 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fff6751f-b96a-42ec-b10f-89ba9ec4c0f2-maas-api-tls\") pod \"maas-api-e2e-autores-87a024-674c46cd56-r7pwq\" (UID: \"fff6751f-b96a-42ec-b10f-89ba9ec4c0f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-87a024-674c46cd56-r7pwq" Sep 10 15:54:41.786910 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:41.786887 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fff6751f-b96a-42ec-b10f-89ba9ec4c0f2-maas-api-tls\") pod \"maas-api-e2e-autores-87a024-674c46cd56-r7pwq\" (UID: \"fff6751f-b96a-42ec-b10f-89ba9ec4c0f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-87a024-674c46cd56-r7pwq" Sep 10 15:54:41.836089 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:41.836053 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:42.003571 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:42.003538 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-87a024-674c46cd56-r7pwq" Sep 10 15:54:42.208515 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:42.208482 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:42.284914 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:42.284888 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-87a024-674c46cd56-r7pwq"] Sep 10 15:54:42.286121 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:54:42.286096 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfff6751f_b96a_42ec_b10f_89ba9ec4c0f2.slice/crio-254029a98d8f0589e33d74ab74d2d0aa614739dfb9fe35ae3ded8296437c6e7e WatchSource:0}: Error finding container 254029a98d8f0589e33d74ab74d2d0aa614739dfb9fe35ae3ded8296437c6e7e: Status 404 returned error can't find the container with id 254029a98d8f0589e33d74ab74d2d0aa614739dfb9fe35ae3ded8296437c6e7e Sep 10 15:54:42.295698 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:42.295671 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-87a024-674c46cd56-r7pwq" event={"ID":"fff6751f-b96a-42ec-b10f-89ba9ec4c0f2","Type":"ContainerStarted","Data":"254029a98d8f0589e33d74ab74d2d0aa614739dfb9fe35ae3ded8296437c6e7e"} Sep 10 15:54:42.297142 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:42.297119 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-54da81e9-5f7bd984f8-n74c4" event={"ID":"291c4fec-1bda-4599-b9be-4d1d9d7a833d","Type":"ContainerStarted","Data":"94b7edb9f322c18af6c10d23e9ca813bbfcc269ae360eec6d44ac88e20687fb3"} Sep 10 15:54:42.297254 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:42.297245 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-54da81e9-5f7bd984f8-n74c4" Sep 10 15:54:42.320704 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:42.320618 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-54da81e9-5f7bd984f8-n74c4" podStartSLOduration=2.582531409 podStartE2EDuration="3.320604345s" totalImagesPullingTime="738.072936ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:54:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:54:42.317170138 +0000 UTC m=+759.292430088" watchObservedRunningTime="2026-09-10 15:54:42.320604345 +0000 UTC m=+759.295864314" Sep 10 15:54:43.030492 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:43.030419 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:43.304733 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:43.304641 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-87a024-674c46cd56-r7pwq" event={"ID":"fff6751f-b96a-42ec-b10f-89ba9ec4c0f2","Type":"ContainerStarted","Data":"72525ca6f22eacd545dcaefe5218e16604f30353bf556403811b93084dd499c7"} Sep 10 15:54:43.304733 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:43.304699 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-87a024-674c46cd56-r7pwq" Sep 10 15:54:43.322174 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:43.322127 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-87a024-674c46cd56-r7pwq" podStartSLOduration=1.9612423460000001 podStartE2EDuration="2.322112031s" totalImagesPullingTime="360.869685ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:54:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:54:43.320371449 +0000 UTC m=+760.295631426" watchObservedRunningTime="2026-09-10 15:54:43.322112031 +0000 UTC m=+760.297371983" Sep 10 15:54:46.034377 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:46.034343 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:46.326029 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:46.325945 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:46.418685 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:46.418653 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:47.263318 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:47.263282 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-83dcaf-698b75cf54-jz542" Sep 10 15:54:48.310239 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:48.310209 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-54da81e9-5f7bd984f8-n74c4" Sep 10 15:54:49.077737 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:49.077707 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:49.315664 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:49.315632 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-87a024-674c46cd56-r7pwq" Sep 10 15:54:50.968568 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:50.968518 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:51.066807 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:51.066762 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:51.166702 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:51.166669 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:51.282272 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:51.282127 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:53.929701 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:53.929664 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:54.309611 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:54.309521 2583 patch_prober.go:28] interesting pod/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 15:54:54.309750 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:54.309608 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" podUID="7c041ae3-1bbe-4089-9c98-3bc78603692a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 15:54:56.301188 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:56.301157 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:56.379085 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:56.379055 2583 generic.go:356] "Generic (PLEG): container finished" podID="7c041ae3-1bbe-4089-9c98-3bc78603692a" containerID="7e5c643d42091de655d8e836f04628c73c8f482f6bd4381aa82f6ec69e07a2cd" exitCode=0 Sep 10 15:54:56.379202 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:56.379106 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" event={"ID":"7c041ae3-1bbe-4089-9c98-3bc78603692a","Type":"ContainerDied","Data":"7e5c643d42091de655d8e836f04628c73c8f482f6bd4381aa82f6ec69e07a2cd"} Sep 10 15:54:56.379489 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:56.379475 2583 scope.go:117] "RemoveContainer" containerID="7e5c643d42091de655d8e836f04628c73c8f482f6bd4381aa82f6ec69e07a2cd" Sep 10 15:54:56.380631 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:56.380599 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 15:54:56.380697 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:56.380665 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 15:54:56.380734 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:56.380695 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 15:54:56.543730 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:56.543635 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:56.838317 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:56.838279 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv"] Sep 10 15:54:56.841018 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:56.840993 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" Sep 10 15:54:56.843494 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:56.843475 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-607b9a-openshift-default-dockercfg-5sdrn\"" Sep 10 15:54:56.855621 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:56.855596 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv"] Sep 10 15:54:56.894406 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:56.891674 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:56.926894 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:56.926857 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-istio-podinfo\") pod \"e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv\" (UID: \"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc\") " pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" Sep 10 15:54:56.927047 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:56.926912 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-istio-token\") pod \"e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv\" (UID: \"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc\") " pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" Sep 10 15:54:56.927047 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:56.927003 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5mk2\" (UniqueName: \"kubernetes.io/projected/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-kube-api-access-t5mk2\") pod \"e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv\" (UID: \"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc\") " pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" Sep 10 15:54:56.927122 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:56.927073 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-istio-envoy\") pod \"e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv\" (UID: \"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc\") " pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" Sep 10 15:54:56.927122 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:56.927091 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-istiod-ca-cert\") pod \"e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv\" (UID: \"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc\") " pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" Sep 10 15:54:56.927193 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:56.927135 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-workload-socket\") pod \"e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv\" (UID: \"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc\") " pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" Sep 10 15:54:56.927193 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:56.927167 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-credential-socket\") pod \"e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv\" (UID: \"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc\") " pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" Sep 10 15:54:56.927292 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:56.927196 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-workload-certs\") pod \"e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv\" (UID: \"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc\") " pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" Sep 10 15:54:56.927292 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:56.927259 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-istio-data\") pod \"e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv\" (UID: \"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc\") " pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" Sep 10 15:54:57.028294 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:57.028254 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-istio-podinfo\") pod \"e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv\" (UID: \"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc\") " pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" Sep 10 15:54:57.028508 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:57.028300 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-istio-token\") pod \"e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv\" (UID: \"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc\") " pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" Sep 10 15:54:57.028508 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:57.028341 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5mk2\" (UniqueName: \"kubernetes.io/projected/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-kube-api-access-t5mk2\") pod \"e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv\" (UID: \"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc\") " pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" Sep 10 15:54:57.028508 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:57.028423 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-istio-envoy\") pod \"e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv\" (UID: \"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc\") " pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" Sep 10 15:54:57.028508 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:57.028444 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-istiod-ca-cert\") pod \"e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv\" (UID: \"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc\") " pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" Sep 10 15:54:57.028508 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:57.028478 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-workload-socket\") pod \"e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv\" (UID: \"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc\") " pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" Sep 10 15:54:57.028508 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:57.028508 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-credential-socket\") pod \"e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv\" (UID: \"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc\") " pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" Sep 10 15:54:57.028815 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:57.028546 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-workload-certs\") pod \"e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv\" (UID: \"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc\") " pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" Sep 10 15:54:57.028815 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:57.028589 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-istio-data\") pod \"e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv\" (UID: \"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc\") " pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" Sep 10 15:54:57.029054 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:57.029027 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-workload-socket\") pod \"e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv\" (UID: \"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc\") " pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" Sep 10 15:54:57.029369 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:57.029116 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-credential-socket\") pod \"e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv\" (UID: \"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc\") " pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" Sep 10 15:54:57.029369 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:57.029127 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-workload-certs\") pod \"e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv\" (UID: \"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc\") " pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" Sep 10 15:54:57.029369 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:57.029207 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-istio-data\") pod \"e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv\" (UID: \"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc\") " pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" Sep 10 15:54:57.029369 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:57.029322 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-istiod-ca-cert\") pod \"e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv\" (UID: \"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc\") " pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" Sep 10 15:54:57.030920 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:57.030903 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-istio-envoy\") pod \"e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv\" (UID: \"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc\") " pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" Sep 10 15:54:57.031017 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:57.030980 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-istio-podinfo\") pod \"e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv\" (UID: \"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc\") " pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" Sep 10 15:54:57.037621 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:57.037602 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-istio-token\") pod \"e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv\" (UID: \"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc\") " pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" Sep 10 15:54:57.037977 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:57.037959 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5mk2\" (UniqueName: \"kubernetes.io/projected/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-kube-api-access-t5mk2\") pod \"e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv\" (UID: \"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc\") " pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" Sep 10 15:54:57.153838 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:57.153809 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" Sep 10 15:54:57.319066 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:57.319032 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:57.392430 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:57.392131 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" event={"ID":"7c041ae3-1bbe-4089-9c98-3bc78603692a","Type":"ContainerStarted","Data":"1f8b1ef7bb38dfb97437dd464e3ba0b86ac8399b2e9ef67f205c0c90c9b2ffc6"} Sep 10 15:54:57.412774 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:57.412683 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" Sep 10 15:54:57.412979 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:57.412949 2583 patch_prober.go:28] interesting pod/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Sep 10 15:54:57.413085 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:57.413032 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" podUID="7c041ae3-1bbe-4089-9c98-3bc78603692a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Sep 10 15:54:57.456037 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:57.456014 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv"] Sep 10 15:54:57.458106 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:54:57.458078 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bec5d6c_78ee_4ab2_9cb4_1df40dfc13fc.slice/crio-e926c40732434d1a695bd301d0e748d11e215a40de87b5195ffb23c9daabd8c4 WatchSource:0}: Error finding container e926c40732434d1a695bd301d0e748d11e215a40de87b5195ffb23c9daabd8c4: Status 404 returned error can't find the container with id e926c40732434d1a695bd301d0e748d11e215a40de87b5195ffb23c9daabd8c4 Sep 10 15:54:57.459108 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:57.459085 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:57.460422 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:57.460369 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 15:54:57.460503 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:57.460460 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 15:54:57.460503 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:57.460492 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 15:54:58.407163 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:58.407121 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" event={"ID":"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc","Type":"ContainerStarted","Data":"22dc9419dcbf54683906cc31e483dcdbc7c24c0969aae44584d9f64b6cc10535"} Sep 10 15:54:58.407659 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:58.407214 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" event={"ID":"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc","Type":"ContainerStarted","Data":"e926c40732434d1a695bd301d0e748d11e215a40de87b5195ffb23c9daabd8c4"} Sep 10 15:54:58.412852 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:58.412797 2583 patch_prober.go:28] interesting pod/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Sep 10 15:54:58.412852 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:58.412844 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" podUID="7c041ae3-1bbe-4089-9c98-3bc78603692a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Sep 10 15:54:58.436723 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:58.436681 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" podStartSLOduration=2.436668023 podStartE2EDuration="2.436668023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:54:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:54:58.43377001 +0000 UTC m=+775.409029964" watchObservedRunningTime="2026-09-10 15:54:58.436668023 +0000 UTC m=+775.411927987" Sep 10 15:54:58.878991 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:58.878952 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:58.979652 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:58.979600 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:59.074160 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:59.074115 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:59.154092 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:59.154007 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" Sep 10 15:54:59.158778 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:59.158754 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" Sep 10 15:54:59.241244 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:59.241207 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:54:59.413403 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:59.413308 2583 patch_prober.go:28] interesting pod/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Sep 10 15:54:59.413403 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:59.413359 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" podUID="7c041ae3-1bbe-4089-9c98-3bc78603692a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Sep 10 15:54:59.413823 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:59.413574 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" Sep 10 15:54:59.415017 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:54:59.414996 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" Sep 10 15:55:00.413287 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:00.413253 2583 patch_prober.go:28] interesting pod/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Sep 10 15:55:00.413513 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:00.413317 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" podUID="7c041ae3-1bbe-4089-9c98-3bc78603692a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Sep 10 15:55:00.440293 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:00.440268 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:00.532589 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:00.532554 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:01.059695 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:01.059661 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:01.413732 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:01.413698 2583 patch_prober.go:28] interesting pod/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Sep 10 15:55:01.414105 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:01.413751 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" podUID="7c041ae3-1bbe-4089-9c98-3bc78603692a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Sep 10 15:55:01.911956 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:01.911919 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-607b9a-57f97879d-x445g"] Sep 10 15:55:01.915761 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:01.915740 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-607b9a-57f97879d-x445g" Sep 10 15:55:01.917759 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:01.917738 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-607b9a-serving-cert\"" Sep 10 15:55:01.926695 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:01.926462 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-607b9a-57f97879d-x445g"] Sep 10 15:55:01.981115 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:01.981067 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:01.985180 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:01.985142 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24270400-b9ca-4faf-91b1-698df59b5a0a-metrics-certs\") pod \"maas-api-e2e-inf-b-607b9a-57f97879d-x445g\" (UID: \"24270400-b9ca-4faf-91b1-698df59b5a0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-607b9a-57f97879d-x445g" Sep 10 15:55:01.985439 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:01.985417 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jgsd\" (UniqueName: \"kubernetes.io/projected/24270400-b9ca-4faf-91b1-698df59b5a0a-kube-api-access-7jgsd\") pod \"maas-api-e2e-inf-b-607b9a-57f97879d-x445g\" (UID: \"24270400-b9ca-4faf-91b1-698df59b5a0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-607b9a-57f97879d-x445g" Sep 10 15:55:01.985724 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:01.985686 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24270400-b9ca-4faf-91b1-698df59b5a0a-maas-api-tls\") pod \"maas-api-e2e-inf-b-607b9a-57f97879d-x445g\" (UID: \"24270400-b9ca-4faf-91b1-698df59b5a0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-607b9a-57f97879d-x445g" Sep 10 15:55:01.985923 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:01.985888 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24270400-b9ca-4faf-91b1-698df59b5a0a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-607b9a-57f97879d-x445g\" (UID: \"24270400-b9ca-4faf-91b1-698df59b5a0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-607b9a-57f97879d-x445g" Sep 10 15:55:02.088248 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:02.088030 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24270400-b9ca-4faf-91b1-698df59b5a0a-maas-api-tls\") pod \"maas-api-e2e-inf-b-607b9a-57f97879d-x445g\" (UID: \"24270400-b9ca-4faf-91b1-698df59b5a0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-607b9a-57f97879d-x445g" Sep 10 15:55:02.088248 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:02.088106 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24270400-b9ca-4faf-91b1-698df59b5a0a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-607b9a-57f97879d-x445g\" (UID: \"24270400-b9ca-4faf-91b1-698df59b5a0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-607b9a-57f97879d-x445g" Sep 10 15:55:02.088248 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:02.088247 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24270400-b9ca-4faf-91b1-698df59b5a0a-metrics-certs\") pod \"maas-api-e2e-inf-b-607b9a-57f97879d-x445g\" (UID: \"24270400-b9ca-4faf-91b1-698df59b5a0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-607b9a-57f97879d-x445g" Sep 10 15:55:02.088596 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:02.088283 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jgsd\" (UniqueName: \"kubernetes.io/projected/24270400-b9ca-4faf-91b1-698df59b5a0a-kube-api-access-7jgsd\") pod \"maas-api-e2e-inf-b-607b9a-57f97879d-x445g\" (UID: \"24270400-b9ca-4faf-91b1-698df59b5a0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-607b9a-57f97879d-x445g" Sep 10 15:55:02.089195 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:02.089108 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:02.089195 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:02.089118 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24270400-b9ca-4faf-91b1-698df59b5a0a-openshift-service-ca\") pod \"maas-api-e2e-inf-b-607b9a-57f97879d-x445g\" (UID: \"24270400-b9ca-4faf-91b1-698df59b5a0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-607b9a-57f97879d-x445g" Sep 10 15:55:02.089195 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:55:02.089192 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-607b9a-serving-cert: secret "maas-api-e2e-inf-b-607b9a-serving-cert" not found Sep 10 15:55:02.089418 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:55:02.089249 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24270400-b9ca-4faf-91b1-698df59b5a0a-maas-api-tls podName:24270400-b9ca-4faf-91b1-698df59b5a0a nodeName:}" failed. No retries permitted until 2026-09-10 15:55:02.589231311 +0000 UTC m=+779.564491255 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/24270400-b9ca-4faf-91b1-698df59b5a0a-maas-api-tls") pod "maas-api-e2e-inf-b-607b9a-57f97879d-x445g" (UID: "24270400-b9ca-4faf-91b1-698df59b5a0a") : secret "maas-api-e2e-inf-b-607b9a-serving-cert" not found Sep 10 15:55:02.092562 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:02.092532 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24270400-b9ca-4faf-91b1-698df59b5a0a-metrics-certs\") pod \"maas-api-e2e-inf-b-607b9a-57f97879d-x445g\" (UID: \"24270400-b9ca-4faf-91b1-698df59b5a0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-607b9a-57f97879d-x445g" Sep 10 15:55:02.097479 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:02.097454 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jgsd\" (UniqueName: \"kubernetes.io/projected/24270400-b9ca-4faf-91b1-698df59b5a0a-kube-api-access-7jgsd\") pod \"maas-api-e2e-inf-b-607b9a-57f97879d-x445g\" (UID: \"24270400-b9ca-4faf-91b1-698df59b5a0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-607b9a-57f97879d-x445g" Sep 10 15:55:02.291192 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:02.291089 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:02.463169 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:02.463129 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:02.534660 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:02.534627 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" Sep 10 15:55:02.535073 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:02.535051 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" Sep 10 15:55:02.536284 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:02.536261 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" Sep 10 15:55:02.560298 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:02.560214 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:02.593664 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:02.593630 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24270400-b9ca-4faf-91b1-698df59b5a0a-maas-api-tls\") pod \"maas-api-e2e-inf-b-607b9a-57f97879d-x445g\" (UID: \"24270400-b9ca-4faf-91b1-698df59b5a0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-607b9a-57f97879d-x445g" Sep 10 15:55:02.596447 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:02.596414 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24270400-b9ca-4faf-91b1-698df59b5a0a-maas-api-tls\") pod \"maas-api-e2e-inf-b-607b9a-57f97879d-x445g\" (UID: \"24270400-b9ca-4faf-91b1-698df59b5a0a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-607b9a-57f97879d-x445g" Sep 10 15:55:02.831100 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:02.831010 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-607b9a-57f97879d-x445g" Sep 10 15:55:02.927751 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:02.927719 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:03.110364 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:03.110333 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-607b9a-57f97879d-x445g"] Sep 10 15:55:03.110973 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:55:03.110950 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24270400_b9ca_4faf_91b1_698df59b5a0a.slice/crio-ae67ecddd2dc036443f2ea73d38986f0887d4f0d0ef53d5cb97dc807c3aa93a6 WatchSource:0}: Error finding container ae67ecddd2dc036443f2ea73d38986f0887d4f0d0ef53d5cb97dc807c3aa93a6: Status 404 returned error can't find the container with id ae67ecddd2dc036443f2ea73d38986f0887d4f0d0ef53d5cb97dc807c3aa93a6 Sep 10 15:55:03.441417 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:03.441373 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-607b9a-57f97879d-x445g" event={"ID":"24270400-b9ca-4faf-91b1-698df59b5a0a","Type":"ContainerStarted","Data":"ae67ecddd2dc036443f2ea73d38986f0887d4f0d0ef53d5cb97dc807c3aa93a6"} Sep 10 15:55:03.941463 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:03.941430 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:04.449625 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:04.449587 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-607b9a-57f97879d-x445g" event={"ID":"24270400-b9ca-4faf-91b1-698df59b5a0a","Type":"ContainerStarted","Data":"34aad7dd2133a4c7fa8fb21942f6e45582e22b2ed2bdea79b870e34e67c030d5"} Sep 10 15:55:04.449800 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:04.449677 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-607b9a-57f97879d-x445g" Sep 10 15:55:04.469144 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:04.469096 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-607b9a-57f97879d-x445g" podStartSLOduration=3.142519153 podStartE2EDuration="3.469081556s" totalImagesPullingTime="326.562403ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:55:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:55:04.468136331 +0000 UTC m=+781.443396284" watchObservedRunningTime="2026-09-10 15:55:04.469081556 +0000 UTC m=+781.444341508" Sep 10 15:55:04.939429 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:04.939396 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:06.486804 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:06.486769 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:06.698939 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:06.698899 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:06.895168 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:06.895139 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:06.990532 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:06.990493 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:09.445827 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:09.445793 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:10.462849 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:10.462818 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-607b9a-57f97879d-x445g" Sep 10 15:55:12.211868 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:12.211830 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:20.350628 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:20.350593 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:21.021205 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:21.021176 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-54da81e9-5f7bd984f8-n74c4"] Sep 10 15:55:21.021479 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:21.021415 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-54da81e9-5f7bd984f8-n74c4" podUID="291c4fec-1bda-4599-b9be-4d1d9d7a833d" containerName="maas-api" containerID="cri-o://94b7edb9f322c18af6c10d23e9ca813bbfcc269ae360eec6d44ac88e20687fb3" gracePeriod=30 Sep 10 15:55:21.051945 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:21.051920 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:21.275638 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:21.275615 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-54da81e9-5f7bd984f8-n74c4" Sep 10 15:55:21.374368 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:21.374333 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/291c4fec-1bda-4599-b9be-4d1d9d7a833d-maas-api-tls\") pod \"291c4fec-1bda-4599-b9be-4d1d9d7a833d\" (UID: \"291c4fec-1bda-4599-b9be-4d1d9d7a833d\") " Sep 10 15:55:21.374824 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:21.374403 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/291c4fec-1bda-4599-b9be-4d1d9d7a833d-openshift-service-ca\") pod \"291c4fec-1bda-4599-b9be-4d1d9d7a833d\" (UID: \"291c4fec-1bda-4599-b9be-4d1d9d7a833d\") " Sep 10 15:55:21.374824 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:21.374444 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/291c4fec-1bda-4599-b9be-4d1d9d7a833d-metrics-certs\") pod \"291c4fec-1bda-4599-b9be-4d1d9d7a833d\" (UID: \"291c4fec-1bda-4599-b9be-4d1d9d7a833d\") " Sep 10 15:55:21.374824 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:21.374489 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbcw9\" (UniqueName: \"kubernetes.io/projected/291c4fec-1bda-4599-b9be-4d1d9d7a833d-kube-api-access-cbcw9\") pod \"291c4fec-1bda-4599-b9be-4d1d9d7a833d\" (UID: \"291c4fec-1bda-4599-b9be-4d1d9d7a833d\") " Sep 10 15:55:21.374824 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:21.374753 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/291c4fec-1bda-4599-b9be-4d1d9d7a833d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "291c4fec-1bda-4599-b9be-4d1d9d7a833d" (UID: "291c4fec-1bda-4599-b9be-4d1d9d7a833d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:55:21.375021 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:21.374899 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/291c4fec-1bda-4599-b9be-4d1d9d7a833d-openshift-service-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:55:21.376540 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:21.376518 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/291c4fec-1bda-4599-b9be-4d1d9d7a833d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "291c4fec-1bda-4599-b9be-4d1d9d7a833d" (UID: "291c4fec-1bda-4599-b9be-4d1d9d7a833d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:55:21.376650 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:21.376556 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/291c4fec-1bda-4599-b9be-4d1d9d7a833d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "291c4fec-1bda-4599-b9be-4d1d9d7a833d" (UID: "291c4fec-1bda-4599-b9be-4d1d9d7a833d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:55:21.376650 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:21.376564 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/291c4fec-1bda-4599-b9be-4d1d9d7a833d-kube-api-access-cbcw9" (OuterVolumeSpecName: "kube-api-access-cbcw9") pod "291c4fec-1bda-4599-b9be-4d1d9d7a833d" (UID: "291c4fec-1bda-4599-b9be-4d1d9d7a833d"). InnerVolumeSpecName "kube-api-access-cbcw9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:55:21.427085 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:21.427000 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78"] Sep 10 15:55:21.427231 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:21.427206 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" podUID="7c041ae3-1bbe-4089-9c98-3bc78603692a" containerName="istio-proxy" containerID="cri-o://1f8b1ef7bb38dfb97437dd464e3ba0b86ac8399b2e9ef67f205c0c90c9b2ffc6" gracePeriod=30 Sep 10 15:55:21.476173 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:21.476142 2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/291c4fec-1bda-4599-b9be-4d1d9d7a833d-metrics-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:55:21.476173 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:21.476168 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbcw9\" (UniqueName: \"kubernetes.io/projected/291c4fec-1bda-4599-b9be-4d1d9d7a833d-kube-api-access-cbcw9\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:55:21.476322 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:21.476181 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/291c4fec-1bda-4599-b9be-4d1d9d7a833d-maas-api-tls\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:55:21.550563 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:21.550532 2583 generic.go:356] "Generic (PLEG): container finished" podID="3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc" containerID="22dc9419dcbf54683906cc31e483dcdbc7c24c0969aae44584d9f64b6cc10535" exitCode=0 Sep 10 15:55:21.550696 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:21.550600 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" event={"ID":"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc","Type":"ContainerDied","Data":"22dc9419dcbf54683906cc31e483dcdbc7c24c0969aae44584d9f64b6cc10535"} Sep 10 15:55:21.551082 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:21.551060 2583 scope.go:117] "RemoveContainer" containerID="22dc9419dcbf54683906cc31e483dcdbc7c24c0969aae44584d9f64b6cc10535" Sep 10 15:55:21.552127 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:21.552097 2583 generic.go:356] "Generic (PLEG): container finished" podID="291c4fec-1bda-4599-b9be-4d1d9d7a833d" containerID="94b7edb9f322c18af6c10d23e9ca813bbfcc269ae360eec6d44ac88e20687fb3" exitCode=0 Sep 10 15:55:21.552244 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:21.552143 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-54da81e9-5f7bd984f8-n74c4" Sep 10 15:55:21.552244 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:21.552159 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 15:55:21.552366 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:21.552179 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-54da81e9-5f7bd984f8-n74c4" event={"ID":"291c4fec-1bda-4599-b9be-4d1d9d7a833d","Type":"ContainerDied","Data":"94b7edb9f322c18af6c10d23e9ca813bbfcc269ae360eec6d44ac88e20687fb3"} Sep 10 15:55:21.552366 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:21.552258 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 15:55:21.552366 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:21.552290 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-54da81e9-5f7bd984f8-n74c4" event={"ID":"291c4fec-1bda-4599-b9be-4d1d9d7a833d","Type":"ContainerDied","Data":"94b5ab8118c3a9e209397f192772e2460f17858958fbc6126d91902482c3edfc"} Sep 10 15:55:21.552366 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:21.552310 2583 scope.go:117] "RemoveContainer" containerID="94b7edb9f322c18af6c10d23e9ca813bbfcc269ae360eec6d44ac88e20687fb3" Sep 10 15:55:21.552366 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:21.552310 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 15:55:21.564567 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:21.564551 2583 scope.go:117] "RemoveContainer" containerID="94b7edb9f322c18af6c10d23e9ca813bbfcc269ae360eec6d44ac88e20687fb3" Sep 10 15:55:21.564852 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:55:21.564831 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94b7edb9f322c18af6c10d23e9ca813bbfcc269ae360eec6d44ac88e20687fb3\": container with ID starting with 94b7edb9f322c18af6c10d23e9ca813bbfcc269ae360eec6d44ac88e20687fb3 not found: ID does not exist" containerID="94b7edb9f322c18af6c10d23e9ca813bbfcc269ae360eec6d44ac88e20687fb3" Sep 10 15:55:21.564929 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:21.564861 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94b7edb9f322c18af6c10d23e9ca813bbfcc269ae360eec6d44ac88e20687fb3"} err="failed to get container status \"94b7edb9f322c18af6c10d23e9ca813bbfcc269ae360eec6d44ac88e20687fb3\": rpc error: code = NotFound desc = could not find container \"94b7edb9f322c18af6c10d23e9ca813bbfcc269ae360eec6d44ac88e20687fb3\": container with ID starting with 94b7edb9f322c18af6c10d23e9ca813bbfcc269ae360eec6d44ac88e20687fb3 not found: ID does not exist" Sep 10 15:55:21.606853 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:21.606828 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-54da81e9-5f7bd984f8-n74c4"] Sep 10 15:55:21.613405 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:21.613359 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-54da81e9-5f7bd984f8-n74c4"] Sep 10 15:55:21.683240 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:21.683159 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:21.759140 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:21.759100 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:21.933117 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:21.933062 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:22.020080 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:22.020001 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:22.122121 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:22.122086 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:22.437639 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:22.437605 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:22.571451 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:22.571374 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" event={"ID":"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc","Type":"ContainerStarted","Data":"9a44c48efa56498d8b650f0e311000c12e2ddfa8b37aa0a166b40b7089df6f93"} Sep 10 15:55:22.852593 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:22.852507 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:22.956564 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:22.956524 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:23.154594 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:23.154562 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" Sep 10 15:55:23.154773 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:23.154753 2583 patch_prober.go:28] interesting pod/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Sep 10 15:55:23.154824 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:23.154804 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" podUID="3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Sep 10 15:55:23.522890 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:23.522804 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="291c4fec-1bda-4599-b9be-4d1d9d7a833d" path="/var/lib/kubelet/pods/291c4fec-1bda-4599-b9be-4d1d9d7a833d/volumes" Sep 10 15:55:24.155182 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:24.155148 2583 patch_prober.go:28] interesting pod/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Sep 10 15:55:24.155348 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:24.155201 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" podUID="3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Sep 10 15:55:25.154360 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:25.154325 2583 patch_prober.go:28] interesting pod/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Sep 10 15:55:25.154734 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:25.154396 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" podUID="3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Sep 10 15:55:25.311643 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:25.311607 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:25.442220 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:25.442133 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:25.607712 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:25.607668 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-384e8433-db67b4d98-6xxgf"] Sep 10 15:55:25.608415 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:25.608393 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="291c4fec-1bda-4599-b9be-4d1d9d7a833d" containerName="maas-api" Sep 10 15:55:25.608522 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:25.608418 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="291c4fec-1bda-4599-b9be-4d1d9d7a833d" containerName="maas-api" Sep 10 15:55:25.608592 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:25.608551 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="291c4fec-1bda-4599-b9be-4d1d9d7a833d" containerName="maas-api" Sep 10 15:55:25.612408 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:25.611542 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-384e8433-db67b4d98-6xxgf" Sep 10 15:55:25.614761 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:25.614583 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-384e8433-serving-cert\"" Sep 10 15:55:25.615396 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:25.615181 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-384e8433-db67b4d98-6xxgf"] Sep 10 15:55:25.622862 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:25.622835 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1-metrics-certs\") pod \"maas-api-e2e-migrate-384e8433-db67b4d98-6xxgf\" (UID: \"2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-384e8433-db67b4d98-6xxgf" Sep 10 15:55:25.622969 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:25.622897 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7hpb\" (UniqueName: \"kubernetes.io/projected/2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1-kube-api-access-k7hpb\") pod \"maas-api-e2e-migrate-384e8433-db67b4d98-6xxgf\" (UID: \"2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-384e8433-db67b4d98-6xxgf" Sep 10 15:55:25.623085 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:25.623067 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1-openshift-service-ca\") pod \"maas-api-e2e-migrate-384e8433-db67b4d98-6xxgf\" (UID: \"2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-384e8433-db67b4d98-6xxgf" Sep 10 15:55:25.623172 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:25.623153 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1-maas-api-tls\") pod \"maas-api-e2e-migrate-384e8433-db67b4d98-6xxgf\" (UID: \"2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-384e8433-db67b4d98-6xxgf" Sep 10 15:55:25.638107 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:25.638084 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:25.723710 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:25.723624 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1-metrics-certs\") pod \"maas-api-e2e-migrate-384e8433-db67b4d98-6xxgf\" (UID: \"2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-384e8433-db67b4d98-6xxgf" Sep 10 15:55:25.723710 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:25.723684 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7hpb\" (UniqueName: \"kubernetes.io/projected/2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1-kube-api-access-k7hpb\") pod \"maas-api-e2e-migrate-384e8433-db67b4d98-6xxgf\" (UID: \"2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-384e8433-db67b4d98-6xxgf" Sep 10 15:55:25.723899 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:25.723728 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1-openshift-service-ca\") pod \"maas-api-e2e-migrate-384e8433-db67b4d98-6xxgf\" (UID: \"2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-384e8433-db67b4d98-6xxgf" Sep 10 15:55:25.723899 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:25.723758 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1-maas-api-tls\") pod \"maas-api-e2e-migrate-384e8433-db67b4d98-6xxgf\" (UID: \"2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-384e8433-db67b4d98-6xxgf" Sep 10 15:55:25.723899 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:55:25.723876 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-384e8433-serving-cert: secret "maas-api-e2e-migrate-384e8433-serving-cert" not found Sep 10 15:55:25.723994 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:55:25.723923 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1-maas-api-tls podName:2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1 nodeName:}" failed. No retries permitted until 2026-09-10 15:55:26.223908096 +0000 UTC m=+803.199168027 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1-maas-api-tls") pod "maas-api-e2e-migrate-384e8433-db67b4d98-6xxgf" (UID: "2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1") : secret "maas-api-e2e-migrate-384e8433-serving-cert" not found Sep 10 15:55:25.724578 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:25.724552 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1-openshift-service-ca\") pod \"maas-api-e2e-migrate-384e8433-db67b4d98-6xxgf\" (UID: \"2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-384e8433-db67b4d98-6xxgf" Sep 10 15:55:25.726292 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:25.726272 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1-metrics-certs\") pod \"maas-api-e2e-migrate-384e8433-db67b4d98-6xxgf\" (UID: \"2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-384e8433-db67b4d98-6xxgf" Sep 10 15:55:25.738184 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:25.738164 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7hpb\" (UniqueName: \"kubernetes.io/projected/2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1-kube-api-access-k7hpb\") pod \"maas-api-e2e-migrate-384e8433-db67b4d98-6xxgf\" (UID: \"2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-384e8433-db67b4d98-6xxgf" Sep 10 15:55:25.738568 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:25.738526 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:25.868895 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:25.868844 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:26.154690 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:26.154662 2583 patch_prober.go:28] interesting pod/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Sep 10 15:55:26.155050 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:26.154716 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" podUID="3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Sep 10 15:55:26.228908 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:26.228874 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1-maas-api-tls\") pod \"maas-api-e2e-migrate-384e8433-db67b4d98-6xxgf\" (UID: \"2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-384e8433-db67b4d98-6xxgf" Sep 10 15:55:26.231265 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:26.231244 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1-maas-api-tls\") pod \"maas-api-e2e-migrate-384e8433-db67b4d98-6xxgf\" (UID: \"2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-384e8433-db67b4d98-6xxgf" Sep 10 15:55:26.422893 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:26.422787 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:26.526463 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:26.526431 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-384e8433-db67b4d98-6xxgf" Sep 10 15:55:26.555153 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:26.555124 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:26.631018 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:26.630988 2583 generic.go:356] "Generic (PLEG): container finished" podID="7c041ae3-1bbe-4089-9c98-3bc78603692a" containerID="1f8b1ef7bb38dfb97437dd464e3ba0b86ac8399b2e9ef67f205c0c90c9b2ffc6" exitCode=0 Sep 10 15:55:26.631323 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:26.631099 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" event={"ID":"7c041ae3-1bbe-4089-9c98-3bc78603692a","Type":"ContainerDied","Data":"1f8b1ef7bb38dfb97437dd464e3ba0b86ac8399b2e9ef67f205c0c90c9b2ffc6"} Sep 10 15:55:26.631323 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:26.631154 2583 scope.go:117] "RemoveContainer" containerID="7e5c643d42091de655d8e836f04628c73c8f482f6bd4381aa82f6ec69e07a2cd" Sep 10 15:55:26.758895 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:26.758872 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" Sep 10 15:55:26.836678 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:26.836065 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c041ae3-1bbe-4089-9c98-3bc78603692a-istio-envoy\") pod \"7c041ae3-1bbe-4089-9c98-3bc78603692a\" (UID: \"7c041ae3-1bbe-4089-9c98-3bc78603692a\") " Sep 10 15:55:26.836678 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:26.836103 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c041ae3-1bbe-4089-9c98-3bc78603692a-workload-socket\") pod \"7c041ae3-1bbe-4089-9c98-3bc78603692a\" (UID: \"7c041ae3-1bbe-4089-9c98-3bc78603692a\") " Sep 10 15:55:26.836678 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:26.836164 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c041ae3-1bbe-4089-9c98-3bc78603692a-istio-data\") pod \"7c041ae3-1bbe-4089-9c98-3bc78603692a\" (UID: \"7c041ae3-1bbe-4089-9c98-3bc78603692a\") " Sep 10 15:55:26.836678 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:26.836236 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c041ae3-1bbe-4089-9c98-3bc78603692a-istio-token\") pod \"7c041ae3-1bbe-4089-9c98-3bc78603692a\" (UID: \"7c041ae3-1bbe-4089-9c98-3bc78603692a\") " Sep 10 15:55:26.836678 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:26.836267 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c041ae3-1bbe-4089-9c98-3bc78603692a-istiod-ca-cert\") pod \"7c041ae3-1bbe-4089-9c98-3bc78603692a\" (UID: \"7c041ae3-1bbe-4089-9c98-3bc78603692a\") " Sep 10 15:55:26.836678 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:26.836316 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c041ae3-1bbe-4089-9c98-3bc78603692a-istio-podinfo\") pod \"7c041ae3-1bbe-4089-9c98-3bc78603692a\" (UID: \"7c041ae3-1bbe-4089-9c98-3bc78603692a\") " Sep 10 15:55:26.836678 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:26.836368 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c041ae3-1bbe-4089-9c98-3bc78603692a-credential-socket\") pod \"7c041ae3-1bbe-4089-9c98-3bc78603692a\" (UID: \"7c041ae3-1bbe-4089-9c98-3bc78603692a\") " Sep 10 15:55:26.836678 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:26.836435 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c041ae3-1bbe-4089-9c98-3bc78603692a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7c041ae3-1bbe-4089-9c98-3bc78603692a" (UID: "7c041ae3-1bbe-4089-9c98-3bc78603692a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:55:26.836678 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:26.836495 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c041ae3-1bbe-4089-9c98-3bc78603692a-workload-certs\") pod \"7c041ae3-1bbe-4089-9c98-3bc78603692a\" (UID: \"7c041ae3-1bbe-4089-9c98-3bc78603692a\") " Sep 10 15:55:26.836678 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:26.836496 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c041ae3-1bbe-4089-9c98-3bc78603692a-istio-data" (OuterVolumeSpecName: "istio-data") pod "7c041ae3-1bbe-4089-9c98-3bc78603692a" (UID: "7c041ae3-1bbe-4089-9c98-3bc78603692a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:55:26.836678 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:26.836566 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrgnj\" (UniqueName: \"kubernetes.io/projected/7c041ae3-1bbe-4089-9c98-3bc78603692a-kube-api-access-qrgnj\") pod \"7c041ae3-1bbe-4089-9c98-3bc78603692a\" (UID: \"7c041ae3-1bbe-4089-9c98-3bc78603692a\") " Sep 10 15:55:26.837353 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:26.836738 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c041ae3-1bbe-4089-9c98-3bc78603692a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7c041ae3-1bbe-4089-9c98-3bc78603692a" (UID: "7c041ae3-1bbe-4089-9c98-3bc78603692a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:55:26.837353 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:26.836764 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c041ae3-1bbe-4089-9c98-3bc78603692a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7c041ae3-1bbe-4089-9c98-3bc78603692a" (UID: "7c041ae3-1bbe-4089-9c98-3bc78603692a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:55:26.837353 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:26.836863 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c041ae3-1bbe-4089-9c98-3bc78603692a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7c041ae3-1bbe-4089-9c98-3bc78603692a" (UID: "7c041ae3-1bbe-4089-9c98-3bc78603692a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:55:26.837353 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:26.837117 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c041ae3-1bbe-4089-9c98-3bc78603692a-workload-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:55:26.837353 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:26.837139 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c041ae3-1bbe-4089-9c98-3bc78603692a-workload-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:55:26.837353 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:26.837158 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c041ae3-1bbe-4089-9c98-3bc78603692a-istio-data\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:55:26.837353 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:26.837175 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c041ae3-1bbe-4089-9c98-3bc78603692a-istiod-ca-cert\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:55:26.837353 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:26.837187 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c041ae3-1bbe-4089-9c98-3bc78603692a-credential-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:55:26.838728 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:26.838705 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-384e8433-db67b4d98-6xxgf"] Sep 10 15:55:26.839389 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:26.839353 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c041ae3-1bbe-4089-9c98-3bc78603692a-istio-token" (OuterVolumeSpecName: "istio-token") pod "7c041ae3-1bbe-4089-9c98-3bc78603692a" (UID: "7c041ae3-1bbe-4089-9c98-3bc78603692a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:55:26.839496 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:26.839418 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c041ae3-1bbe-4089-9c98-3bc78603692a-kube-api-access-qrgnj" (OuterVolumeSpecName: "kube-api-access-qrgnj") pod "7c041ae3-1bbe-4089-9c98-3bc78603692a" (UID: "7c041ae3-1bbe-4089-9c98-3bc78603692a"). InnerVolumeSpecName "kube-api-access-qrgnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:55:26.839747 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:26.839712 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c041ae3-1bbe-4089-9c98-3bc78603692a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7c041ae3-1bbe-4089-9c98-3bc78603692a" (UID: "7c041ae3-1bbe-4089-9c98-3bc78603692a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 15:55:26.839892 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:26.839876 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c041ae3-1bbe-4089-9c98-3bc78603692a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7c041ae3-1bbe-4089-9c98-3bc78603692a" (UID: "7c041ae3-1bbe-4089-9c98-3bc78603692a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:55:26.841116 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:55:26.841095 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ebdbbc3_f3c4_4bc3_a8ef_c4ab25af2fd1.slice/crio-ecb5c1f20016147a80460eca105a9ec60dde53545da41dae5621446d31bf60c0 WatchSource:0}: Error finding container ecb5c1f20016147a80460eca105a9ec60dde53545da41dae5621446d31bf60c0: Status 404 returned error can't find the container with id ecb5c1f20016147a80460eca105a9ec60dde53545da41dae5621446d31bf60c0 Sep 10 15:55:26.938062 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:26.938033 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrgnj\" (UniqueName: \"kubernetes.io/projected/7c041ae3-1bbe-4089-9c98-3bc78603692a-kube-api-access-qrgnj\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:55:26.938062 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:26.938065 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c041ae3-1bbe-4089-9c98-3bc78603692a-istio-envoy\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:55:26.938242 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:26.938077 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c041ae3-1bbe-4089-9c98-3bc78603692a-istio-token\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:55:26.938242 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:26.938085 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c041ae3-1bbe-4089-9c98-3bc78603692a-istio-podinfo\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:55:27.154943 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:27.154901 2583 patch_prober.go:28] interesting pod/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Sep 10 15:55:27.155329 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:27.154966 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" podUID="3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Sep 10 15:55:27.640448 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:27.640407 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-384e8433-db67b4d98-6xxgf" event={"ID":"2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1","Type":"ContainerStarted","Data":"fc743f150002504f7b1888a8efa1f50eb01505fd6677cc62a7c2f150ece35c4a"} Sep 10 15:55:27.640636 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:27.640468 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-384e8433-db67b4d98-6xxgf" event={"ID":"2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1","Type":"ContainerStarted","Data":"ecb5c1f20016147a80460eca105a9ec60dde53545da41dae5621446d31bf60c0"} Sep 10 15:55:27.640636 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:27.640522 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-384e8433-db67b4d98-6xxgf" Sep 10 15:55:27.641894 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:27.641871 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" event={"ID":"7c041ae3-1bbe-4089-9c98-3bc78603692a","Type":"ContainerDied","Data":"42c49452883515a20268050e496c6ef1788b72b2f821d0c9138a183b308d9108"} Sep 10 15:55:27.642027 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:27.641916 2583 scope.go:117] "RemoveContainer" containerID="1f8b1ef7bb38dfb97437dd464e3ba0b86ac8399b2e9ef67f205c0c90c9b2ffc6" Sep 10 15:55:27.642027 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:27.641925 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78" Sep 10 15:55:27.674305 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:27.674227 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-384e8433-db67b4d98-6xxgf" podStartSLOduration=2.227011937 podStartE2EDuration="2.674207771s" totalImagesPullingTime="447.195834ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:55:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:55:27.671300011 +0000 UTC m=+804.646559958" watchObservedRunningTime="2026-09-10 15:55:27.674207771 +0000 UTC m=+804.649467725" Sep 10 15:55:27.697536 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:27.697497 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78"] Sep 10 15:55:27.708281 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:27.708255 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-54da81e9-openshift-default-54494d9994-ttx78"] Sep 10 15:55:28.158654 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:28.158627 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" Sep 10 15:55:28.159024 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:28.159001 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" Sep 10 15:55:28.159612 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:28.159591 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" Sep 10 15:55:29.516254 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:29.516217 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c041ae3-1bbe-4089-9c98-3bc78603692a" path="/var/lib/kubelet/pods/7c041ae3-1bbe-4089-9c98-3bc78603692a/volumes" Sep 10 15:55:33.653528 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:33.653499 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-384e8433-db67b4d98-6xxgf" Sep 10 15:55:57.099241 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:57.099205 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-384e8433-db67b4d98-6xxgf"] Sep 10 15:55:57.099722 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:57.099595 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-384e8433-db67b4d98-6xxgf" podUID="2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1" containerName="maas-api" containerID="cri-o://fc743f150002504f7b1888a8efa1f50eb01505fd6677cc62a7c2f150ece35c4a" gracePeriod=30 Sep 10 15:55:57.134584 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:57.134555 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:57.359432 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:57.359356 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-384e8433-db67b4d98-6xxgf" Sep 10 15:55:57.506071 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:57.506040 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1-maas-api-tls\") pod \"2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1\" (UID: \"2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1\") " Sep 10 15:55:57.506235 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:57.506118 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1-metrics-certs\") pod \"2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1\" (UID: \"2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1\") " Sep 10 15:55:57.506235 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:57.506146 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7hpb\" (UniqueName: \"kubernetes.io/projected/2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1-kube-api-access-k7hpb\") pod \"2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1\" (UID: \"2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1\") " Sep 10 15:55:57.506235 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:57.506183 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1-openshift-service-ca\") pod \"2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1\" (UID: \"2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1\") " Sep 10 15:55:57.506672 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:57.506635 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1" (UID: "2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:55:57.508449 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:57.508418 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1" (UID: "2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:55:57.508551 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:57.508487 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1" (UID: "2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:55:57.508643 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:57.508622 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1-kube-api-access-k7hpb" (OuterVolumeSpecName: "kube-api-access-k7hpb") pod "2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1" (UID: "2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1"). InnerVolumeSpecName "kube-api-access-k7hpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:55:57.607182 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:57.607146 2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1-metrics-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:55:57.607182 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:57.607176 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7hpb\" (UniqueName: \"kubernetes.io/projected/2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1-kube-api-access-k7hpb\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:55:57.607362 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:57.607190 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1-openshift-service-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:55:57.607362 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:57.607205 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1-maas-api-tls\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:55:57.830700 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:57.830663 2583 generic.go:356] "Generic (PLEG): container finished" podID="2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1" containerID="fc743f150002504f7b1888a8efa1f50eb01505fd6677cc62a7c2f150ece35c4a" exitCode=0 Sep 10 15:55:57.830872 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:57.830711 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-384e8433-db67b4d98-6xxgf" Sep 10 15:55:57.830872 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:57.830710 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-384e8433-db67b4d98-6xxgf" event={"ID":"2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1","Type":"ContainerDied","Data":"fc743f150002504f7b1888a8efa1f50eb01505fd6677cc62a7c2f150ece35c4a"} Sep 10 15:55:57.830872 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:57.830764 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-384e8433-db67b4d98-6xxgf" event={"ID":"2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1","Type":"ContainerDied","Data":"ecb5c1f20016147a80460eca105a9ec60dde53545da41dae5621446d31bf60c0"} Sep 10 15:55:57.830872 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:57.830795 2583 scope.go:117] "RemoveContainer" containerID="fc743f150002504f7b1888a8efa1f50eb01505fd6677cc62a7c2f150ece35c4a" Sep 10 15:55:57.840294 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:57.840274 2583 scope.go:117] "RemoveContainer" containerID="fc743f150002504f7b1888a8efa1f50eb01505fd6677cc62a7c2f150ece35c4a" Sep 10 15:55:57.840573 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:55:57.840551 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc743f150002504f7b1888a8efa1f50eb01505fd6677cc62a7c2f150ece35c4a\": container with ID starting with fc743f150002504f7b1888a8efa1f50eb01505fd6677cc62a7c2f150ece35c4a not found: ID does not exist" containerID="fc743f150002504f7b1888a8efa1f50eb01505fd6677cc62a7c2f150ece35c4a" Sep 10 15:55:57.840634 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:57.840580 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc743f150002504f7b1888a8efa1f50eb01505fd6677cc62a7c2f150ece35c4a"} err="failed to get container status \"fc743f150002504f7b1888a8efa1f50eb01505fd6677cc62a7c2f150ece35c4a\": rpc error: code = NotFound desc = could not find container \"fc743f150002504f7b1888a8efa1f50eb01505fd6677cc62a7c2f150ece35c4a\": container with ID starting with fc743f150002504f7b1888a8efa1f50eb01505fd6677cc62a7c2f150ece35c4a not found: ID does not exist" Sep 10 15:55:57.853367 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:57.853345 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-384e8433-db67b4d98-6xxgf"] Sep 10 15:55:57.865901 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:57.865879 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-384e8433-db67b4d98-6xxgf"] Sep 10 15:55:58.289800 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:58.289767 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22"] Sep 10 15:55:58.290461 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:58.290440 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c041ae3-1bbe-4089-9c98-3bc78603692a" containerName="istio-proxy" Sep 10 15:55:58.290510 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:58.290464 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c041ae3-1bbe-4089-9c98-3bc78603692a" containerName="istio-proxy" Sep 10 15:55:58.290510 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:58.290479 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c041ae3-1bbe-4089-9c98-3bc78603692a" containerName="istio-proxy" Sep 10 15:55:58.290510 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:58.290488 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c041ae3-1bbe-4089-9c98-3bc78603692a" containerName="istio-proxy" Sep 10 15:55:58.290510 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:58.290500 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1" containerName="maas-api" Sep 10 15:55:58.290510 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:58.290508 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1" containerName="maas-api" Sep 10 15:55:58.290676 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:58.290623 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1" containerName="maas-api" Sep 10 15:55:58.290676 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:58.290632 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c041ae3-1bbe-4089-9c98-3bc78603692a" containerName="istio-proxy" Sep 10 15:55:58.290676 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:58.290641 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c041ae3-1bbe-4089-9c98-3bc78603692a" containerName="istio-proxy" Sep 10 15:55:58.294950 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:58.294922 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" Sep 10 15:55:58.297267 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:58.297251 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-2dacd235-openshift-default-dockercfg-nw829\"" Sep 10 15:55:58.305600 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:58.305575 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22"] Sep 10 15:55:58.414755 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:58.414720 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3043223-0816-40f8-b9cc-2bc5830c61ff-workload-socket\") pod \"e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22\" (UID: \"e3043223-0816-40f8-b9cc-2bc5830c61ff\") " pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" Sep 10 15:55:58.414960 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:58.414779 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3043223-0816-40f8-b9cc-2bc5830c61ff-istio-token\") pod \"e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22\" (UID: \"e3043223-0816-40f8-b9cc-2bc5830c61ff\") " pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" Sep 10 15:55:58.414960 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:58.414839 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3043223-0816-40f8-b9cc-2bc5830c61ff-istio-data\") pod \"e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22\" (UID: \"e3043223-0816-40f8-b9cc-2bc5830c61ff\") " pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" Sep 10 15:55:58.414960 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:58.414915 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3043223-0816-40f8-b9cc-2bc5830c61ff-istiod-ca-cert\") pod \"e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22\" (UID: \"e3043223-0816-40f8-b9cc-2bc5830c61ff\") " pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" Sep 10 15:55:58.414960 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:58.414954 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfm4b\" (UniqueName: \"kubernetes.io/projected/e3043223-0816-40f8-b9cc-2bc5830c61ff-kube-api-access-kfm4b\") pod \"e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22\" (UID: \"e3043223-0816-40f8-b9cc-2bc5830c61ff\") " pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" Sep 10 15:55:58.415188 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:58.414990 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3043223-0816-40f8-b9cc-2bc5830c61ff-istio-podinfo\") pod \"e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22\" (UID: \"e3043223-0816-40f8-b9cc-2bc5830c61ff\") " pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" Sep 10 15:55:58.415188 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:58.415043 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3043223-0816-40f8-b9cc-2bc5830c61ff-credential-socket\") pod \"e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22\" (UID: \"e3043223-0816-40f8-b9cc-2bc5830c61ff\") " pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" Sep 10 15:55:58.415188 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:58.415060 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3043223-0816-40f8-b9cc-2bc5830c61ff-istio-envoy\") pod \"e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22\" (UID: \"e3043223-0816-40f8-b9cc-2bc5830c61ff\") " pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" Sep 10 15:55:58.415188 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:58.415108 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3043223-0816-40f8-b9cc-2bc5830c61ff-workload-certs\") pod \"e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22\" (UID: \"e3043223-0816-40f8-b9cc-2bc5830c61ff\") " pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" Sep 10 15:55:58.515823 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:58.515791 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfm4b\" (UniqueName: \"kubernetes.io/projected/e3043223-0816-40f8-b9cc-2bc5830c61ff-kube-api-access-kfm4b\") pod \"e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22\" (UID: \"e3043223-0816-40f8-b9cc-2bc5830c61ff\") " pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" Sep 10 15:55:58.515980 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:58.515835 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3043223-0816-40f8-b9cc-2bc5830c61ff-istio-podinfo\") pod \"e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22\" (UID: \"e3043223-0816-40f8-b9cc-2bc5830c61ff\") " pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" Sep 10 15:55:58.515980 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:58.515859 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3043223-0816-40f8-b9cc-2bc5830c61ff-credential-socket\") pod \"e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22\" (UID: \"e3043223-0816-40f8-b9cc-2bc5830c61ff\") " pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" Sep 10 15:55:58.515980 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:58.515878 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3043223-0816-40f8-b9cc-2bc5830c61ff-istio-envoy\") pod \"e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22\" (UID: \"e3043223-0816-40f8-b9cc-2bc5830c61ff\") " pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" Sep 10 15:55:58.515980 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:58.515914 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3043223-0816-40f8-b9cc-2bc5830c61ff-workload-certs\") pod \"e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22\" (UID: \"e3043223-0816-40f8-b9cc-2bc5830c61ff\") " pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" Sep 10 15:55:58.515980 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:58.515963 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3043223-0816-40f8-b9cc-2bc5830c61ff-workload-socket\") pod \"e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22\" (UID: \"e3043223-0816-40f8-b9cc-2bc5830c61ff\") " pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" Sep 10 15:55:58.516262 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:58.516007 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3043223-0816-40f8-b9cc-2bc5830c61ff-istio-token\") pod \"e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22\" (UID: \"e3043223-0816-40f8-b9cc-2bc5830c61ff\") " pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" Sep 10 15:55:58.516262 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:58.516222 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3043223-0816-40f8-b9cc-2bc5830c61ff-credential-socket\") pod \"e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22\" (UID: \"e3043223-0816-40f8-b9cc-2bc5830c61ff\") " pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" Sep 10 15:55:58.516262 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:58.516226 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3043223-0816-40f8-b9cc-2bc5830c61ff-istio-data\") pod \"e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22\" (UID: \"e3043223-0816-40f8-b9cc-2bc5830c61ff\") " pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" Sep 10 15:55:58.516470 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:58.516266 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3043223-0816-40f8-b9cc-2bc5830c61ff-workload-certs\") pod \"e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22\" (UID: \"e3043223-0816-40f8-b9cc-2bc5830c61ff\") " pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" Sep 10 15:55:58.516470 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:58.516324 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3043223-0816-40f8-b9cc-2bc5830c61ff-istiod-ca-cert\") pod \"e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22\" (UID: \"e3043223-0816-40f8-b9cc-2bc5830c61ff\") " pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" Sep 10 15:55:58.516470 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:58.516355 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3043223-0816-40f8-b9cc-2bc5830c61ff-workload-socket\") pod \"e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22\" (UID: \"e3043223-0816-40f8-b9cc-2bc5830c61ff\") " pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" Sep 10 15:55:58.516637 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:58.516556 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3043223-0816-40f8-b9cc-2bc5830c61ff-istio-data\") pod \"e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22\" (UID: \"e3043223-0816-40f8-b9cc-2bc5830c61ff\") " pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" Sep 10 15:55:58.516857 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:58.516836 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3043223-0816-40f8-b9cc-2bc5830c61ff-istiod-ca-cert\") pod \"e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22\" (UID: \"e3043223-0816-40f8-b9cc-2bc5830c61ff\") " pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" Sep 10 15:55:58.518239 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:58.518218 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3043223-0816-40f8-b9cc-2bc5830c61ff-istio-envoy\") pod \"e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22\" (UID: \"e3043223-0816-40f8-b9cc-2bc5830c61ff\") " pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" Sep 10 15:55:58.518459 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:58.518443 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3043223-0816-40f8-b9cc-2bc5830c61ff-istio-podinfo\") pod \"e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22\" (UID: \"e3043223-0816-40f8-b9cc-2bc5830c61ff\") " pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" Sep 10 15:55:58.528122 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:58.528103 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3043223-0816-40f8-b9cc-2bc5830c61ff-istio-token\") pod \"e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22\" (UID: \"e3043223-0816-40f8-b9cc-2bc5830c61ff\") " pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" Sep 10 15:55:58.528210 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:58.528180 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfm4b\" (UniqueName: \"kubernetes.io/projected/e3043223-0816-40f8-b9cc-2bc5830c61ff-kube-api-access-kfm4b\") pod \"e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22\" (UID: \"e3043223-0816-40f8-b9cc-2bc5830c61ff\") " pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" Sep 10 15:55:58.606702 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:58.606639 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" Sep 10 15:55:58.843913 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:58.843870 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:55:58.891952 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:58.891929 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22"] Sep 10 15:55:58.894273 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:55:58.894242 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3043223_0816_40f8_b9cc_2bc5830c61ff.slice/crio-56121c894fd5068ff684fac022077a9d6131e17a4ffb219c874bdbd487ce35d1 WatchSource:0}: Error finding container 56121c894fd5068ff684fac022077a9d6131e17a4ffb219c874bdbd487ce35d1: Status 404 returned error can't find the container with id 56121c894fd5068ff684fac022077a9d6131e17a4ffb219c874bdbd487ce35d1 Sep 10 15:55:58.896331 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:58.896303 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 15:55:58.896464 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:58.896364 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 15:55:58.896464 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:58.896411 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 15:55:59.515229 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:59.515197 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1" path="/var/lib/kubelet/pods/2ebdbbc3-f3c4-4bc3-a8ef-c4ab25af2fd1/volumes" Sep 10 15:55:59.845879 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:59.845787 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" event={"ID":"e3043223-0816-40f8-b9cc-2bc5830c61ff","Type":"ContainerStarted","Data":"4dca16ddd8854673dd9b20e9af1388c17e09a7124f9f1601221743406fb3e790"} Sep 10 15:55:59.845879 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:59.845850 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" event={"ID":"e3043223-0816-40f8-b9cc-2bc5830c61ff","Type":"ContainerStarted","Data":"56121c894fd5068ff684fac022077a9d6131e17a4ffb219c874bdbd487ce35d1"} Sep 10 15:55:59.890860 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:55:59.890815 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" podStartSLOduration=1.890800305 podStartE2EDuration="1.890800305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:55:59.890670149 +0000 UTC m=+836.865930124" watchObservedRunningTime="2026-09-10 15:55:59.890800305 +0000 UTC m=+836.866060258" Sep 10 15:56:00.607288 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:00.607251 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" Sep 10 15:56:00.612845 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:00.612820 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" Sep 10 15:56:00.851826 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:00.851800 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" Sep 10 15:56:00.852531 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:00.852514 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" Sep 10 15:56:01.324657 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:01.324614 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:01.417341 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:01.417307 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:02.455256 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:02.455221 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2dacd235-55c8bfdb47-wgfwn"] Sep 10 15:56:02.458774 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:02.458754 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dacd235-55c8bfdb47-wgfwn" Sep 10 15:56:02.461116 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:02.461092 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-2dacd235-serving-cert\"" Sep 10 15:56:02.472414 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:02.472391 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2dacd235-55c8bfdb47-wgfwn"] Sep 10 15:56:02.529039 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:02.529000 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:02.557251 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:02.557220 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b85fe6b-ec23-41f9-8684-8f1a432140f0-openshift-service-ca\") pod \"maas-api-e2e-ait-2dacd235-55c8bfdb47-wgfwn\" (UID: \"4b85fe6b-ec23-41f9-8684-8f1a432140f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dacd235-55c8bfdb47-wgfwn" Sep 10 15:56:02.557461 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:02.557283 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b85fe6b-ec23-41f9-8684-8f1a432140f0-maas-api-tls\") pod \"maas-api-e2e-ait-2dacd235-55c8bfdb47-wgfwn\" (UID: \"4b85fe6b-ec23-41f9-8684-8f1a432140f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dacd235-55c8bfdb47-wgfwn" Sep 10 15:56:02.557461 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:02.557334 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b85fe6b-ec23-41f9-8684-8f1a432140f0-metrics-certs\") pod \"maas-api-e2e-ait-2dacd235-55c8bfdb47-wgfwn\" (UID: \"4b85fe6b-ec23-41f9-8684-8f1a432140f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dacd235-55c8bfdb47-wgfwn" Sep 10 15:56:02.557461 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:02.557445 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d69s6\" (UniqueName: \"kubernetes.io/projected/4b85fe6b-ec23-41f9-8684-8f1a432140f0-kube-api-access-d69s6\") pod \"maas-api-e2e-ait-2dacd235-55c8bfdb47-wgfwn\" (UID: \"4b85fe6b-ec23-41f9-8684-8f1a432140f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dacd235-55c8bfdb47-wgfwn" Sep 10 15:56:02.627684 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:02.627656 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:02.658102 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:02.658073 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b85fe6b-ec23-41f9-8684-8f1a432140f0-openshift-service-ca\") pod \"maas-api-e2e-ait-2dacd235-55c8bfdb47-wgfwn\" (UID: \"4b85fe6b-ec23-41f9-8684-8f1a432140f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dacd235-55c8bfdb47-wgfwn" Sep 10 15:56:02.658299 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:02.658135 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b85fe6b-ec23-41f9-8684-8f1a432140f0-maas-api-tls\") pod \"maas-api-e2e-ait-2dacd235-55c8bfdb47-wgfwn\" (UID: \"4b85fe6b-ec23-41f9-8684-8f1a432140f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dacd235-55c8bfdb47-wgfwn" Sep 10 15:56:02.658299 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:02.658154 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b85fe6b-ec23-41f9-8684-8f1a432140f0-metrics-certs\") pod \"maas-api-e2e-ait-2dacd235-55c8bfdb47-wgfwn\" (UID: \"4b85fe6b-ec23-41f9-8684-8f1a432140f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dacd235-55c8bfdb47-wgfwn" Sep 10 15:56:02.658299 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:02.658199 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d69s6\" (UniqueName: \"kubernetes.io/projected/4b85fe6b-ec23-41f9-8684-8f1a432140f0-kube-api-access-d69s6\") pod \"maas-api-e2e-ait-2dacd235-55c8bfdb47-wgfwn\" (UID: \"4b85fe6b-ec23-41f9-8684-8f1a432140f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dacd235-55c8bfdb47-wgfwn" Sep 10 15:56:02.658478 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:56:02.658300 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-2dacd235-serving-cert: secret "maas-api-e2e-ait-2dacd235-serving-cert" not found Sep 10 15:56:02.658478 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:56:02.658394 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b85fe6b-ec23-41f9-8684-8f1a432140f0-maas-api-tls podName:4b85fe6b-ec23-41f9-8684-8f1a432140f0 nodeName:}" failed. No retries permitted until 2026-09-10 15:56:03.158358765 +0000 UTC m=+840.133618700 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4b85fe6b-ec23-41f9-8684-8f1a432140f0-maas-api-tls") pod "maas-api-e2e-ait-2dacd235-55c8bfdb47-wgfwn" (UID: "4b85fe6b-ec23-41f9-8684-8f1a432140f0") : secret "maas-api-e2e-ait-2dacd235-serving-cert" not found Sep 10 15:56:02.658847 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:02.658818 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b85fe6b-ec23-41f9-8684-8f1a432140f0-openshift-service-ca\") pod \"maas-api-e2e-ait-2dacd235-55c8bfdb47-wgfwn\" (UID: \"4b85fe6b-ec23-41f9-8684-8f1a432140f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dacd235-55c8bfdb47-wgfwn" Sep 10 15:56:02.660837 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:02.660815 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b85fe6b-ec23-41f9-8684-8f1a432140f0-metrics-certs\") pod \"maas-api-e2e-ait-2dacd235-55c8bfdb47-wgfwn\" (UID: \"4b85fe6b-ec23-41f9-8684-8f1a432140f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dacd235-55c8bfdb47-wgfwn" Sep 10 15:56:02.667727 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:02.667704 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d69s6\" (UniqueName: \"kubernetes.io/projected/4b85fe6b-ec23-41f9-8684-8f1a432140f0-kube-api-access-d69s6\") pod \"maas-api-e2e-ait-2dacd235-55c8bfdb47-wgfwn\" (UID: \"4b85fe6b-ec23-41f9-8684-8f1a432140f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dacd235-55c8bfdb47-wgfwn" Sep 10 15:56:02.727292 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:02.727206 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:03.019433 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:03.019336 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:03.164500 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:03.164454 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b85fe6b-ec23-41f9-8684-8f1a432140f0-maas-api-tls\") pod \"maas-api-e2e-ait-2dacd235-55c8bfdb47-wgfwn\" (UID: \"4b85fe6b-ec23-41f9-8684-8f1a432140f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dacd235-55c8bfdb47-wgfwn" Sep 10 15:56:03.166777 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:03.166757 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b85fe6b-ec23-41f9-8684-8f1a432140f0-maas-api-tls\") pod \"maas-api-e2e-ait-2dacd235-55c8bfdb47-wgfwn\" (UID: \"4b85fe6b-ec23-41f9-8684-8f1a432140f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dacd235-55c8bfdb47-wgfwn" Sep 10 15:56:03.185474 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:03.185448 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:03.373114 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:03.373034 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dacd235-55c8bfdb47-wgfwn" Sep 10 15:56:03.649680 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:03.649655 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2dacd235-55c8bfdb47-wgfwn"] Sep 10 15:56:03.651017 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:56:03.650990 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b85fe6b_ec23_41f9_8684_8f1a432140f0.slice/crio-535ef92c7dfa398b55b9f56dd99bdfec61df988611d9a6fa856caf15573d1102 WatchSource:0}: Error finding container 535ef92c7dfa398b55b9f56dd99bdfec61df988611d9a6fa856caf15573d1102: Status 404 returned error can't find the container with id 535ef92c7dfa398b55b9f56dd99bdfec61df988611d9a6fa856caf15573d1102 Sep 10 15:56:03.871791 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:03.871759 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dacd235-55c8bfdb47-wgfwn" event={"ID":"4b85fe6b-ec23-41f9-8684-8f1a432140f0","Type":"ContainerStarted","Data":"535ef92c7dfa398b55b9f56dd99bdfec61df988611d9a6fa856caf15573d1102"} Sep 10 15:56:04.879253 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:04.879221 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dacd235-55c8bfdb47-wgfwn" event={"ID":"4b85fe6b-ec23-41f9-8684-8f1a432140f0","Type":"ContainerStarted","Data":"742c882cc010cffbf4da22e2d78d7760e38010e0ee2d41f9a19ac79111e5bf1b"} Sep 10 15:56:04.879642 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:04.879338 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dacd235-55c8bfdb47-wgfwn" Sep 10 15:56:04.902000 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:04.901958 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dacd235-55c8bfdb47-wgfwn" podStartSLOduration=2.445688904 podStartE2EDuration="2.901946025s" totalImagesPullingTime="456.257121ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:56:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:56:04.899335599 +0000 UTC m=+841.874595572" watchObservedRunningTime="2026-09-10 15:56:04.901946025 +0000 UTC m=+841.877205978" Sep 10 15:56:04.957996 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:04.957966 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:10.894262 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:10.894229 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dacd235-55c8bfdb47-wgfwn" Sep 10 15:56:16.893513 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:16.893473 2583 patch_prober.go:28] interesting pod/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 10 15:56:16.893952 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:16.893528 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" podUID="e3043223-0816-40f8-b9cc-2bc5830c61ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 10 15:56:18.966126 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:18.966093 2583 generic.go:356] "Generic (PLEG): container finished" podID="e3043223-0816-40f8-b9cc-2bc5830c61ff" containerID="4dca16ddd8854673dd9b20e9af1388c17e09a7124f9f1601221743406fb3e790" exitCode=0 Sep 10 15:56:18.966546 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:18.966166 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" event={"ID":"e3043223-0816-40f8-b9cc-2bc5830c61ff","Type":"ContainerDied","Data":"4dca16ddd8854673dd9b20e9af1388c17e09a7124f9f1601221743406fb3e790"} Sep 10 15:56:18.966590 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:18.966550 2583 scope.go:117] "RemoveContainer" containerID="4dca16ddd8854673dd9b20e9af1388c17e09a7124f9f1601221743406fb3e790" Sep 10 15:56:18.967733 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:18.967704 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 15:56:18.967801 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:18.967773 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 15:56:18.967842 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:18.967813 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 15:56:19.978280 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:19.978238 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" event={"ID":"e3043223-0816-40f8-b9cc-2bc5830c61ff","Type":"ContainerStarted","Data":"896c0fea88f4351c31b74c617bf0f8cda8fbe50e3c8886406d1dfb4e63bd1400"} Sep 10 15:56:20.607681 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:20.607644 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" Sep 10 15:56:20.607878 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:20.607836 2583 patch_prober.go:28] interesting pod/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Sep 10 15:56:20.607958 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:20.607908 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" podUID="e3043223-0816-40f8-b9cc-2bc5830c61ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Sep 10 15:56:21.607939 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:21.607903 2583 patch_prober.go:28] interesting pod/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Sep 10 15:56:21.608310 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:21.607956 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" podUID="e3043223-0816-40f8-b9cc-2bc5830c61ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Sep 10 15:56:22.607473 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:22.607438 2583 patch_prober.go:28] interesting pod/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Sep 10 15:56:22.607643 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:22.607491 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" podUID="e3043223-0816-40f8-b9cc-2bc5830c61ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Sep 10 15:56:23.607208 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:23.607173 2583 patch_prober.go:28] interesting pod/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Sep 10 15:56:23.607618 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:23.607227 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" podUID="e3043223-0816-40f8-b9cc-2bc5830c61ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Sep 10 15:56:24.607200 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:24.607161 2583 patch_prober.go:28] interesting pod/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" start-of-body= Sep 10 15:56:24.607374 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:24.607224 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" podUID="e3043223-0816-40f8-b9cc-2bc5830c61ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.49:15021/healthz/ready\": dial tcp 10.134.0.49:15021: connect: connection refused" Sep 10 15:56:25.611165 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:25.611136 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" Sep 10 15:56:25.611607 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:25.611406 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" Sep 10 15:56:25.612015 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:25.611997 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" Sep 10 15:56:33.295210 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:33.295175 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2dacd235-55c8bfdb47-wgfwn"] Sep 10 15:56:33.295705 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:33.295431 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dacd235-55c8bfdb47-wgfwn" podUID="4b85fe6b-ec23-41f9-8684-8f1a432140f0" containerName="maas-api" containerID="cri-o://742c882cc010cffbf4da22e2d78d7760e38010e0ee2d41f9a19ac79111e5bf1b" gracePeriod=30 Sep 10 15:56:33.328104 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:33.328051 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:33.428354 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:33.428323 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:33.551868 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:33.551811 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dacd235-55c8bfdb47-wgfwn" Sep 10 15:56:33.568096 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:33.568028 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:33.649650 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:33.649624 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b85fe6b-ec23-41f9-8684-8f1a432140f0-metrics-certs\") pod \"4b85fe6b-ec23-41f9-8684-8f1a432140f0\" (UID: \"4b85fe6b-ec23-41f9-8684-8f1a432140f0\") " Sep 10 15:56:33.649829 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:33.649806 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d69s6\" (UniqueName: \"kubernetes.io/projected/4b85fe6b-ec23-41f9-8684-8f1a432140f0-kube-api-access-d69s6\") pod \"4b85fe6b-ec23-41f9-8684-8f1a432140f0\" (UID: \"4b85fe6b-ec23-41f9-8684-8f1a432140f0\") " Sep 10 15:56:33.649880 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:33.649840 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b85fe6b-ec23-41f9-8684-8f1a432140f0-maas-api-tls\") pod \"4b85fe6b-ec23-41f9-8684-8f1a432140f0\" (UID: \"4b85fe6b-ec23-41f9-8684-8f1a432140f0\") " Sep 10 15:56:33.649880 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:33.649864 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b85fe6b-ec23-41f9-8684-8f1a432140f0-openshift-service-ca\") pod \"4b85fe6b-ec23-41f9-8684-8f1a432140f0\" (UID: \"4b85fe6b-ec23-41f9-8684-8f1a432140f0\") " Sep 10 15:56:33.650258 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:33.650224 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b85fe6b-ec23-41f9-8684-8f1a432140f0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4b85fe6b-ec23-41f9-8684-8f1a432140f0" (UID: "4b85fe6b-ec23-41f9-8684-8f1a432140f0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:56:33.651812 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:33.651784 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b85fe6b-ec23-41f9-8684-8f1a432140f0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4b85fe6b-ec23-41f9-8684-8f1a432140f0" (UID: "4b85fe6b-ec23-41f9-8684-8f1a432140f0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:56:33.651915 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:33.651829 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b85fe6b-ec23-41f9-8684-8f1a432140f0-kube-api-access-d69s6" (OuterVolumeSpecName: "kube-api-access-d69s6") pod "4b85fe6b-ec23-41f9-8684-8f1a432140f0" (UID: "4b85fe6b-ec23-41f9-8684-8f1a432140f0"). InnerVolumeSpecName "kube-api-access-d69s6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:56:33.651915 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:33.651826 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b85fe6b-ec23-41f9-8684-8f1a432140f0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4b85fe6b-ec23-41f9-8684-8f1a432140f0" (UID: "4b85fe6b-ec23-41f9-8684-8f1a432140f0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:56:33.654699 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:33.654678 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:33.751483 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:33.751456 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d69s6\" (UniqueName: \"kubernetes.io/projected/4b85fe6b-ec23-41f9-8684-8f1a432140f0-kube-api-access-d69s6\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:56:33.751483 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:33.751481 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b85fe6b-ec23-41f9-8684-8f1a432140f0-maas-api-tls\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:56:33.751650 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:33.751491 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b85fe6b-ec23-41f9-8684-8f1a432140f0-openshift-service-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:56:33.751650 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:33.751500 2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b85fe6b-ec23-41f9-8684-8f1a432140f0-metrics-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:56:33.753269 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:33.753245 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:34.067002 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:34.066957 2583 generic.go:356] "Generic (PLEG): container finished" podID="4b85fe6b-ec23-41f9-8684-8f1a432140f0" containerID="742c882cc010cffbf4da22e2d78d7760e38010e0ee2d41f9a19ac79111e5bf1b" exitCode=0 Sep 10 15:56:34.067184 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:34.067007 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dacd235-55c8bfdb47-wgfwn" Sep 10 15:56:34.067184 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:34.067036 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dacd235-55c8bfdb47-wgfwn" event={"ID":"4b85fe6b-ec23-41f9-8684-8f1a432140f0","Type":"ContainerDied","Data":"742c882cc010cffbf4da22e2d78d7760e38010e0ee2d41f9a19ac79111e5bf1b"} Sep 10 15:56:34.067184 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:34.067088 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2dacd235-55c8bfdb47-wgfwn" event={"ID":"4b85fe6b-ec23-41f9-8684-8f1a432140f0","Type":"ContainerDied","Data":"535ef92c7dfa398b55b9f56dd99bdfec61df988611d9a6fa856caf15573d1102"} Sep 10 15:56:34.067184 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:34.067113 2583 scope.go:117] "RemoveContainer" containerID="742c882cc010cffbf4da22e2d78d7760e38010e0ee2d41f9a19ac79111e5bf1b" Sep 10 15:56:34.078679 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:34.078654 2583 scope.go:117] "RemoveContainer" containerID="742c882cc010cffbf4da22e2d78d7760e38010e0ee2d41f9a19ac79111e5bf1b" Sep 10 15:56:34.080526 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:56:34.080452 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"742c882cc010cffbf4da22e2d78d7760e38010e0ee2d41f9a19ac79111e5bf1b\": container with ID starting with 742c882cc010cffbf4da22e2d78d7760e38010e0ee2d41f9a19ac79111e5bf1b not found: ID does not exist" containerID="742c882cc010cffbf4da22e2d78d7760e38010e0ee2d41f9a19ac79111e5bf1b" Sep 10 15:56:34.080526 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:34.080495 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"742c882cc010cffbf4da22e2d78d7760e38010e0ee2d41f9a19ac79111e5bf1b"} err="failed to get container status \"742c882cc010cffbf4da22e2d78d7760e38010e0ee2d41f9a19ac79111e5bf1b\": rpc error: code = NotFound desc = could not find container \"742c882cc010cffbf4da22e2d78d7760e38010e0ee2d41f9a19ac79111e5bf1b\": container with ID starting with 742c882cc010cffbf4da22e2d78d7760e38010e0ee2d41f9a19ac79111e5bf1b not found: ID does not exist" Sep 10 15:56:34.100446 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:34.100421 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2dacd235-55c8bfdb47-wgfwn"] Sep 10 15:56:34.106824 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:34.106800 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2dacd235-55c8bfdb47-wgfwn"] Sep 10 15:56:35.160660 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:35.160630 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22"] Sep 10 15:56:35.161047 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:35.160846 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" podUID="e3043223-0816-40f8-b9cc-2bc5830c61ff" containerName="istio-proxy" containerID="cri-o://896c0fea88f4351c31b74c617bf0f8cda8fbe50e3c8886406d1dfb4e63bd1400" gracePeriod=30 Sep 10 15:56:35.176341 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:35.176313 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:35.515751 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:35.515673 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b85fe6b-ec23-41f9-8684-8f1a432140f0" path="/var/lib/kubelet/pods/4b85fe6b-ec23-41f9-8684-8f1a432140f0/volumes" Sep 10 15:56:35.612426 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:35.612397 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:36.125313 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:36.125281 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:36.326162 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:36.326127 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:38.809279 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:38.809247 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:38.937220 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:38.937190 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:40.206079 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.206049 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8d528235-54d696866f-rthbp"] Sep 10 15:56:40.206751 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.206730 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b85fe6b-ec23-41f9-8684-8f1a432140f0" containerName="maas-api" Sep 10 15:56:40.206818 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.206755 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b85fe6b-ec23-41f9-8684-8f1a432140f0" containerName="maas-api" Sep 10 15:56:40.206898 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.206887 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b85fe6b-ec23-41f9-8684-8f1a432140f0" containerName="maas-api" Sep 10 15:56:40.210135 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.210102 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8d528235-54d696866f-rthbp" Sep 10 15:56:40.218530 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.218499 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-8d528235-serving-cert\"" Sep 10 15:56:40.286243 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.286212 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8d528235-54d696866f-rthbp"] Sep 10 15:56:40.292745 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.292704 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:40.311540 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.311303 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvt2j\" (UniqueName: \"kubernetes.io/projected/62e2229c-7d88-4c58-9246-7f063cebe2d7-kube-api-access-gvt2j\") pod \"maas-api-e2e-derive-8d528235-54d696866f-rthbp\" (UID: \"62e2229c-7d88-4c58-9246-7f063cebe2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8d528235-54d696866f-rthbp" Sep 10 15:56:40.311540 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.311372 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62e2229c-7d88-4c58-9246-7f063cebe2d7-metrics-certs\") pod \"maas-api-e2e-derive-8d528235-54d696866f-rthbp\" (UID: \"62e2229c-7d88-4c58-9246-7f063cebe2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8d528235-54d696866f-rthbp" Sep 10 15:56:40.311540 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.311525 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62e2229c-7d88-4c58-9246-7f063cebe2d7-openshift-service-ca\") pod \"maas-api-e2e-derive-8d528235-54d696866f-rthbp\" (UID: \"62e2229c-7d88-4c58-9246-7f063cebe2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8d528235-54d696866f-rthbp" Sep 10 15:56:40.311794 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.311621 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62e2229c-7d88-4c58-9246-7f063cebe2d7-maas-api-tls\") pod \"maas-api-e2e-derive-8d528235-54d696866f-rthbp\" (UID: \"62e2229c-7d88-4c58-9246-7f063cebe2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8d528235-54d696866f-rthbp" Sep 10 15:56:40.406357 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.406333 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:40.412466 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.412443 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62e2229c-7d88-4c58-9246-7f063cebe2d7-openshift-service-ca\") pod \"maas-api-e2e-derive-8d528235-54d696866f-rthbp\" (UID: \"62e2229c-7d88-4c58-9246-7f063cebe2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8d528235-54d696866f-rthbp" Sep 10 15:56:40.412545 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.412482 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62e2229c-7d88-4c58-9246-7f063cebe2d7-maas-api-tls\") pod \"maas-api-e2e-derive-8d528235-54d696866f-rthbp\" (UID: \"62e2229c-7d88-4c58-9246-7f063cebe2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8d528235-54d696866f-rthbp" Sep 10 15:56:40.412545 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.412507 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvt2j\" (UniqueName: \"kubernetes.io/projected/62e2229c-7d88-4c58-9246-7f063cebe2d7-kube-api-access-gvt2j\") pod \"maas-api-e2e-derive-8d528235-54d696866f-rthbp\" (UID: \"62e2229c-7d88-4c58-9246-7f063cebe2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8d528235-54d696866f-rthbp" Sep 10 15:56:40.412545 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.412541 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62e2229c-7d88-4c58-9246-7f063cebe2d7-metrics-certs\") pod \"maas-api-e2e-derive-8d528235-54d696866f-rthbp\" (UID: \"62e2229c-7d88-4c58-9246-7f063cebe2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8d528235-54d696866f-rthbp" Sep 10 15:56:40.413065 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.413046 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62e2229c-7d88-4c58-9246-7f063cebe2d7-openshift-service-ca\") pod \"maas-api-e2e-derive-8d528235-54d696866f-rthbp\" (UID: \"62e2229c-7d88-4c58-9246-7f063cebe2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8d528235-54d696866f-rthbp" Sep 10 15:56:40.414860 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.414842 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62e2229c-7d88-4c58-9246-7f063cebe2d7-metrics-certs\") pod \"maas-api-e2e-derive-8d528235-54d696866f-rthbp\" (UID: \"62e2229c-7d88-4c58-9246-7f063cebe2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8d528235-54d696866f-rthbp" Sep 10 15:56:40.415024 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.415003 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62e2229c-7d88-4c58-9246-7f063cebe2d7-maas-api-tls\") pod \"maas-api-e2e-derive-8d528235-54d696866f-rthbp\" (UID: \"62e2229c-7d88-4c58-9246-7f063cebe2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8d528235-54d696866f-rthbp" Sep 10 15:56:40.448316 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.448254 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" Sep 10 15:56:40.454666 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.454629 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvt2j\" (UniqueName: \"kubernetes.io/projected/62e2229c-7d88-4c58-9246-7f063cebe2d7-kube-api-access-gvt2j\") pod \"maas-api-e2e-derive-8d528235-54d696866f-rthbp\" (UID: \"62e2229c-7d88-4c58-9246-7f063cebe2d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-8d528235-54d696866f-rthbp" Sep 10 15:56:40.532394 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.532352 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:40.549538 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.549514 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8d528235-54d696866f-rthbp" Sep 10 15:56:40.614441 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.614415 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3043223-0816-40f8-b9cc-2bc5830c61ff-credential-socket\") pod \"e3043223-0816-40f8-b9cc-2bc5830c61ff\" (UID: \"e3043223-0816-40f8-b9cc-2bc5830c61ff\") " Sep 10 15:56:40.614577 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.614493 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfm4b\" (UniqueName: \"kubernetes.io/projected/e3043223-0816-40f8-b9cc-2bc5830c61ff-kube-api-access-kfm4b\") pod \"e3043223-0816-40f8-b9cc-2bc5830c61ff\" (UID: \"e3043223-0816-40f8-b9cc-2bc5830c61ff\") " Sep 10 15:56:40.614577 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.614522 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3043223-0816-40f8-b9cc-2bc5830c61ff-istio-token\") pod \"e3043223-0816-40f8-b9cc-2bc5830c61ff\" (UID: \"e3043223-0816-40f8-b9cc-2bc5830c61ff\") " Sep 10 15:56:40.614577 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.614546 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3043223-0816-40f8-b9cc-2bc5830c61ff-istio-podinfo\") pod \"e3043223-0816-40f8-b9cc-2bc5830c61ff\" (UID: \"e3043223-0816-40f8-b9cc-2bc5830c61ff\") " Sep 10 15:56:40.614734 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.614583 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3043223-0816-40f8-b9cc-2bc5830c61ff-workload-certs\") pod \"e3043223-0816-40f8-b9cc-2bc5830c61ff\" (UID: \"e3043223-0816-40f8-b9cc-2bc5830c61ff\") " Sep 10 15:56:40.614734 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.614646 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3043223-0816-40f8-b9cc-2bc5830c61ff-istio-envoy\") pod \"e3043223-0816-40f8-b9cc-2bc5830c61ff\" (UID: \"e3043223-0816-40f8-b9cc-2bc5830c61ff\") " Sep 10 15:56:40.614734 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.614702 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3043223-0816-40f8-b9cc-2bc5830c61ff-istio-data\") pod \"e3043223-0816-40f8-b9cc-2bc5830c61ff\" (UID: \"e3043223-0816-40f8-b9cc-2bc5830c61ff\") " Sep 10 15:56:40.614734 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.614707 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3043223-0816-40f8-b9cc-2bc5830c61ff-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e3043223-0816-40f8-b9cc-2bc5830c61ff" (UID: "e3043223-0816-40f8-b9cc-2bc5830c61ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:56:40.614734 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.614727 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3043223-0816-40f8-b9cc-2bc5830c61ff-istiod-ca-cert\") pod \"e3043223-0816-40f8-b9cc-2bc5830c61ff\" (UID: \"e3043223-0816-40f8-b9cc-2bc5830c61ff\") " Sep 10 15:56:40.614980 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.614794 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3043223-0816-40f8-b9cc-2bc5830c61ff-workload-socket\") pod \"e3043223-0816-40f8-b9cc-2bc5830c61ff\" (UID: \"e3043223-0816-40f8-b9cc-2bc5830c61ff\") " Sep 10 15:56:40.614980 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.614909 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3043223-0816-40f8-b9cc-2bc5830c61ff-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e3043223-0816-40f8-b9cc-2bc5830c61ff" (UID: "e3043223-0816-40f8-b9cc-2bc5830c61ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:56:40.615278 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.615152 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3043223-0816-40f8-b9cc-2bc5830c61ff-credential-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:56:40.615278 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.615158 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3043223-0816-40f8-b9cc-2bc5830c61ff-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e3043223-0816-40f8-b9cc-2bc5830c61ff" (UID: "e3043223-0816-40f8-b9cc-2bc5830c61ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:56:40.615278 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.615169 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3043223-0816-40f8-b9cc-2bc5830c61ff-workload-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:56:40.615717 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.615669 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3043223-0816-40f8-b9cc-2bc5830c61ff-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e3043223-0816-40f8-b9cc-2bc5830c61ff" (UID: "e3043223-0816-40f8-b9cc-2bc5830c61ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:56:40.615985 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.615947 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3043223-0816-40f8-b9cc-2bc5830c61ff-istio-data" (OuterVolumeSpecName: "istio-data") pod "e3043223-0816-40f8-b9cc-2bc5830c61ff" (UID: "e3043223-0816-40f8-b9cc-2bc5830c61ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:56:40.617684 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.617650 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3043223-0816-40f8-b9cc-2bc5830c61ff-kube-api-access-kfm4b" (OuterVolumeSpecName: "kube-api-access-kfm4b") pod "e3043223-0816-40f8-b9cc-2bc5830c61ff" (UID: "e3043223-0816-40f8-b9cc-2bc5830c61ff"). InnerVolumeSpecName "kube-api-access-kfm4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:56:40.617684 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.617675 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3043223-0816-40f8-b9cc-2bc5830c61ff-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e3043223-0816-40f8-b9cc-2bc5830c61ff" (UID: "e3043223-0816-40f8-b9cc-2bc5830c61ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 15:56:40.617890 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.617866 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3043223-0816-40f8-b9cc-2bc5830c61ff-istio-token" (OuterVolumeSpecName: "istio-token") pod "e3043223-0816-40f8-b9cc-2bc5830c61ff" (UID: "e3043223-0816-40f8-b9cc-2bc5830c61ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:56:40.618812 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.618791 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3043223-0816-40f8-b9cc-2bc5830c61ff-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e3043223-0816-40f8-b9cc-2bc5830c61ff" (UID: "e3043223-0816-40f8-b9cc-2bc5830c61ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:56:40.716564 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.716502 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3043223-0816-40f8-b9cc-2bc5830c61ff-istio-data\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:56:40.716564 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.716525 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3043223-0816-40f8-b9cc-2bc5830c61ff-istiod-ca-cert\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:56:40.716564 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.716533 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3043223-0816-40f8-b9cc-2bc5830c61ff-workload-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:56:40.716564 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.716542 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfm4b\" (UniqueName: \"kubernetes.io/projected/e3043223-0816-40f8-b9cc-2bc5830c61ff-kube-api-access-kfm4b\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:56:40.716564 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.716551 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3043223-0816-40f8-b9cc-2bc5830c61ff-istio-token\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:56:40.716564 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.716559 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3043223-0816-40f8-b9cc-2bc5830c61ff-istio-podinfo\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:56:40.716564 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.716567 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3043223-0816-40f8-b9cc-2bc5830c61ff-istio-envoy\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:56:40.863719 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:40.863695 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8d528235-54d696866f-rthbp"] Sep 10 15:56:40.865540 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:56:40.865514 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62e2229c_7d88_4c58_9246_7f063cebe2d7.slice/crio-4199ac6f23aa3f21ff7853b86da28b55e5bb8d2f8036299f83a686270ccae0b2 WatchSource:0}: Error finding container 4199ac6f23aa3f21ff7853b86da28b55e5bb8d2f8036299f83a686270ccae0b2: Status 404 returned error can't find the container with id 4199ac6f23aa3f21ff7853b86da28b55e5bb8d2f8036299f83a686270ccae0b2 Sep 10 15:56:41.108958 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:41.108863 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8d528235-54d696866f-rthbp" event={"ID":"62e2229c-7d88-4c58-9246-7f063cebe2d7","Type":"ContainerStarted","Data":"4199ac6f23aa3f21ff7853b86da28b55e5bb8d2f8036299f83a686270ccae0b2"} Sep 10 15:56:41.110695 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:41.110664 2583 generic.go:356] "Generic (PLEG): container finished" podID="e3043223-0816-40f8-b9cc-2bc5830c61ff" containerID="896c0fea88f4351c31b74c617bf0f8cda8fbe50e3c8886406d1dfb4e63bd1400" exitCode=0 Sep 10 15:56:41.110819 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:41.110714 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" Sep 10 15:56:41.110819 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:41.110720 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" event={"ID":"e3043223-0816-40f8-b9cc-2bc5830c61ff","Type":"ContainerDied","Data":"896c0fea88f4351c31b74c617bf0f8cda8fbe50e3c8886406d1dfb4e63bd1400"} Sep 10 15:56:41.110819 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:41.110767 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22" event={"ID":"e3043223-0816-40f8-b9cc-2bc5830c61ff","Type":"ContainerDied","Data":"56121c894fd5068ff684fac022077a9d6131e17a4ffb219c874bdbd487ce35d1"} Sep 10 15:56:41.110819 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:41.110793 2583 scope.go:117] "RemoveContainer" containerID="896c0fea88f4351c31b74c617bf0f8cda8fbe50e3c8886406d1dfb4e63bd1400" Sep 10 15:56:41.120712 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:41.120694 2583 scope.go:117] "RemoveContainer" containerID="4dca16ddd8854673dd9b20e9af1388c17e09a7124f9f1601221743406fb3e790" Sep 10 15:56:41.128790 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:41.128773 2583 scope.go:117] "RemoveContainer" containerID="896c0fea88f4351c31b74c617bf0f8cda8fbe50e3c8886406d1dfb4e63bd1400" Sep 10 15:56:41.129049 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:56:41.129031 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"896c0fea88f4351c31b74c617bf0f8cda8fbe50e3c8886406d1dfb4e63bd1400\": container with ID starting with 896c0fea88f4351c31b74c617bf0f8cda8fbe50e3c8886406d1dfb4e63bd1400 not found: ID does not exist" containerID="896c0fea88f4351c31b74c617bf0f8cda8fbe50e3c8886406d1dfb4e63bd1400" Sep 10 15:56:41.129089 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:41.129059 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"896c0fea88f4351c31b74c617bf0f8cda8fbe50e3c8886406d1dfb4e63bd1400"} err="failed to get container status \"896c0fea88f4351c31b74c617bf0f8cda8fbe50e3c8886406d1dfb4e63bd1400\": rpc error: code = NotFound desc = could not find container \"896c0fea88f4351c31b74c617bf0f8cda8fbe50e3c8886406d1dfb4e63bd1400\": container with ID starting with 896c0fea88f4351c31b74c617bf0f8cda8fbe50e3c8886406d1dfb4e63bd1400 not found: ID does not exist" Sep 10 15:56:41.129089 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:41.129075 2583 scope.go:117] "RemoveContainer" containerID="4dca16ddd8854673dd9b20e9af1388c17e09a7124f9f1601221743406fb3e790" Sep 10 15:56:41.129315 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:56:41.129299 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dca16ddd8854673dd9b20e9af1388c17e09a7124f9f1601221743406fb3e790\": container with ID starting with 4dca16ddd8854673dd9b20e9af1388c17e09a7124f9f1601221743406fb3e790 not found: ID does not exist" containerID="4dca16ddd8854673dd9b20e9af1388c17e09a7124f9f1601221743406fb3e790" Sep 10 15:56:41.129351 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:41.129321 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dca16ddd8854673dd9b20e9af1388c17e09a7124f9f1601221743406fb3e790"} err="failed to get container status \"4dca16ddd8854673dd9b20e9af1388c17e09a7124f9f1601221743406fb3e790\": rpc error: code = NotFound desc = could not find container \"4dca16ddd8854673dd9b20e9af1388c17e09a7124f9f1601221743406fb3e790\": container with ID starting with 4dca16ddd8854673dd9b20e9af1388c17e09a7124f9f1601221743406fb3e790 not found: ID does not exist" Sep 10 15:56:41.145084 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:41.145057 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22"] Sep 10 15:56:41.148798 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:41.148776 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-2dacd235-openshift-default-74b49478c6-9bc22"] Sep 10 15:56:41.410396 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:41.410345 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:41.518581 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:41.518542 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3043223-0816-40f8-b9cc-2bc5830c61ff" path="/var/lib/kubelet/pods/e3043223-0816-40f8-b9cc-2bc5830c61ff/volumes" Sep 10 15:56:41.599441 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:41.599374 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:56:41.897871 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:41.897839 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8d528235-5687bc88c8-s7jnp"] Sep 10 15:56:41.898590 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:41.898570 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3043223-0816-40f8-b9cc-2bc5830c61ff" containerName="istio-proxy" Sep 10 15:56:41.898642 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:41.898594 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3043223-0816-40f8-b9cc-2bc5830c61ff" containerName="istio-proxy" Sep 10 15:56:41.898749 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:41.898737 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3043223-0816-40f8-b9cc-2bc5830c61ff" containerName="istio-proxy" Sep 10 15:56:41.898784 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:41.898756 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3043223-0816-40f8-b9cc-2bc5830c61ff" containerName="istio-proxy" Sep 10 15:56:41.902607 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:41.902584 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-8d528235-5687bc88c8-s7jnp" Sep 10 15:56:41.905251 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:41.905232 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-8d528235-dockercfg-bhtml\"" Sep 10 15:56:41.905369 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:41.905327 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-8d528235\"" Sep 10 15:56:41.937406 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:41.937354 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8d528235-5687bc88c8-s7jnp"] Sep 10 15:56:42.029265 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:42.029238 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/354e46a3-31a2-4b72-8cdc-d80f08a315b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-8d528235-5687bc88c8-s7jnp\" (UID: \"354e46a3-31a2-4b72-8cdc-d80f08a315b6\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8d528235-5687bc88c8-s7jnp" Sep 10 15:56:42.029435 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:42.029270 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zhx6\" (UniqueName: \"kubernetes.io/projected/354e46a3-31a2-4b72-8cdc-d80f08a315b6-kube-api-access-2zhx6\") pod \"payload-pre-processing-e2e-derive-8d528235-5687bc88c8-s7jnp\" (UID: \"354e46a3-31a2-4b72-8cdc-d80f08a315b6\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8d528235-5687bc88c8-s7jnp" Sep 10 15:56:42.117753 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:42.117723 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8d528235-54d696866f-rthbp" event={"ID":"62e2229c-7d88-4c58-9246-7f063cebe2d7","Type":"ContainerStarted","Data":"5f8692a3eb438230e67b30f55b1142659378994f8f1de9aa245bb4a0e3246524"} Sep 10 15:56:42.117895 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:42.117777 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8d528235-54d696866f-rthbp" Sep 10 15:56:42.129831 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:42.129805 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/354e46a3-31a2-4b72-8cdc-d80f08a315b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-8d528235-5687bc88c8-s7jnp\" (UID: \"354e46a3-31a2-4b72-8cdc-d80f08a315b6\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8d528235-5687bc88c8-s7jnp" Sep 10 15:56:42.129964 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:42.129836 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zhx6\" (UniqueName: \"kubernetes.io/projected/354e46a3-31a2-4b72-8cdc-d80f08a315b6-kube-api-access-2zhx6\") pod \"payload-pre-processing-e2e-derive-8d528235-5687bc88c8-s7jnp\" (UID: \"354e46a3-31a2-4b72-8cdc-d80f08a315b6\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8d528235-5687bc88c8-s7jnp" Sep 10 15:56:42.130513 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:42.130493 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/354e46a3-31a2-4b72-8cdc-d80f08a315b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-8d528235-5687bc88c8-s7jnp\" (UID: \"354e46a3-31a2-4b72-8cdc-d80f08a315b6\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8d528235-5687bc88c8-s7jnp" Sep 10 15:56:42.150646 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:42.150589 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zhx6\" (UniqueName: \"kubernetes.io/projected/354e46a3-31a2-4b72-8cdc-d80f08a315b6-kube-api-access-2zhx6\") pod \"payload-pre-processing-e2e-derive-8d528235-5687bc88c8-s7jnp\" (UID: \"354e46a3-31a2-4b72-8cdc-d80f08a315b6\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-8d528235-5687bc88c8-s7jnp" Sep 10 15:56:42.160857 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:42.160814 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8d528235-54d696866f-rthbp" podStartSLOduration=1.6651072679999999 podStartE2EDuration="2.160800629s" totalImagesPullingTime="495.693361ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:56:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:56:42.157727386 +0000 UTC m=+879.132987329" watchObservedRunningTime="2026-09-10 15:56:42.160800629 +0000 UTC m=+879.136060581" Sep 10 15:56:42.212706 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:42.212685 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-8d528235-5687bc88c8-s7jnp" Sep 10 15:56:42.501262 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:42.501238 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8d528235-5687bc88c8-s7jnp"] Sep 10 15:56:42.501856 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:56:42.501828 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod354e46a3_31a2_4b72_8cdc_d80f08a315b6.slice/crio-497190818d1e2733ad765f81d32c5836c1ad800c3ac6db977958d481dfe800ae WatchSource:0}: Error finding container 497190818d1e2733ad765f81d32c5836c1ad800c3ac6db977958d481dfe800ae: Status 404 returned error can't find the container with id 497190818d1e2733ad765f81d32c5836c1ad800c3ac6db977958d481dfe800ae Sep 10 15:56:43.127702 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:43.127667 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-8d528235-5687bc88c8-s7jnp" event={"ID":"354e46a3-31a2-4b72-8cdc-d80f08a315b6","Type":"ContainerStarted","Data":"40c93bd659c2b23afd0ac7ebd01b031aa6a167371013fb0fe1468d6d2d2032e2"} Sep 10 15:56:43.127863 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:43.127808 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-8d528235-5687bc88c8-s7jnp" Sep 10 15:56:43.127863 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:43.127832 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-8d528235-5687bc88c8-s7jnp" event={"ID":"354e46a3-31a2-4b72-8cdc-d80f08a315b6","Type":"ContainerStarted","Data":"497190818d1e2733ad765f81d32c5836c1ad800c3ac6db977958d481dfe800ae"} Sep 10 15:56:43.151520 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:43.151475 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-8d528235-5687bc88c8-s7jnp" podStartSLOduration=2.151461034 podStartE2EDuration="2.151461034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:56:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:56:43.149707464 +0000 UTC m=+880.124967418" watchObservedRunningTime="2026-09-10 15:56:43.151461034 +0000 UTC m=+880.126720986" Sep 10 15:56:48.132851 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:48.132821 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8d528235-54d696866f-rthbp" Sep 10 15:56:54.143686 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:54.143659 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-8d528235-5687bc88c8-s7jnp" Sep 10 15:56:57.667748 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:56:57.667665 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:01.447177 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:01.447147 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:01.656561 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:01.656527 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:02.937524 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:02.937495 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:08.508300 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:08.508265 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:10.919449 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:10.919419 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:11.599058 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:11.599028 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:13.707120 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:13.707086 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:13.945219 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:13.945173 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:20.476141 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:20.476107 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:21.248024 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:21.247986 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8d528235-54d696866f-rthbp"] Sep 10 15:57:21.248278 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:21.248235 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8d528235-54d696866f-rthbp" podUID="62e2229c-7d88-4c58-9246-7f063cebe2d7" containerName="maas-api" containerID="cri-o://5f8692a3eb438230e67b30f55b1142659378994f8f1de9aa245bb4a0e3246524" gracePeriod=30 Sep 10 15:57:21.278196 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:21.278140 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:21.325263 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:21.325224 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8d528235-5687bc88c8-s7jnp"] Sep 10 15:57:21.325507 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:21.325483 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-8d528235-5687bc88c8-s7jnp" podUID="354e46a3-31a2-4b72-8cdc-d80f08a315b6" containerName="payload-pre-processing" containerID="cri-o://40c93bd659c2b23afd0ac7ebd01b031aa6a167371013fb0fe1468d6d2d2032e2" gracePeriod=30 Sep 10 15:57:21.350174 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:21.350148 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:21.452031 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:21.451981 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:21.521019 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:21.520948 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:21.582085 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:21.579590 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8d528235-54d696866f-rthbp" Sep 10 15:57:21.610180 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:21.610157 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-8d528235-5687bc88c8-s7jnp" Sep 10 15:57:21.680811 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:21.680783 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62e2229c-7d88-4c58-9246-7f063cebe2d7-openshift-service-ca\") pod \"62e2229c-7d88-4c58-9246-7f063cebe2d7\" (UID: \"62e2229c-7d88-4c58-9246-7f063cebe2d7\") " Sep 10 15:57:21.681515 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:21.681486 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62e2229c-7d88-4c58-9246-7f063cebe2d7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "62e2229c-7d88-4c58-9246-7f063cebe2d7" (UID: "62e2229c-7d88-4c58-9246-7f063cebe2d7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:57:21.681515 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:21.681498 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62e2229c-7d88-4c58-9246-7f063cebe2d7-maas-api-tls\") pod \"62e2229c-7d88-4c58-9246-7f063cebe2d7\" (UID: \"62e2229c-7d88-4c58-9246-7f063cebe2d7\") " Sep 10 15:57:21.681723 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:21.681556 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/354e46a3-31a2-4b72-8cdc-d80f08a315b6-plugins-config-volume\") pod \"354e46a3-31a2-4b72-8cdc-d80f08a315b6\" (UID: \"354e46a3-31a2-4b72-8cdc-d80f08a315b6\") " Sep 10 15:57:21.681723 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:21.681593 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvt2j\" (UniqueName: \"kubernetes.io/projected/62e2229c-7d88-4c58-9246-7f063cebe2d7-kube-api-access-gvt2j\") pod \"62e2229c-7d88-4c58-9246-7f063cebe2d7\" (UID: \"62e2229c-7d88-4c58-9246-7f063cebe2d7\") " Sep 10 15:57:21.681723 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:21.681641 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zhx6\" (UniqueName: \"kubernetes.io/projected/354e46a3-31a2-4b72-8cdc-d80f08a315b6-kube-api-access-2zhx6\") pod \"354e46a3-31a2-4b72-8cdc-d80f08a315b6\" (UID: \"354e46a3-31a2-4b72-8cdc-d80f08a315b6\") " Sep 10 15:57:21.681723 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:21.681687 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62e2229c-7d88-4c58-9246-7f063cebe2d7-metrics-certs\") pod \"62e2229c-7d88-4c58-9246-7f063cebe2d7\" (UID: \"62e2229c-7d88-4c58-9246-7f063cebe2d7\") " Sep 10 15:57:21.682208 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:21.682177 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/354e46a3-31a2-4b72-8cdc-d80f08a315b6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "354e46a3-31a2-4b72-8cdc-d80f08a315b6" (UID: "354e46a3-31a2-4b72-8cdc-d80f08a315b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:57:21.682635 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:21.682611 2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/354e46a3-31a2-4b72-8cdc-d80f08a315b6-plugins-config-volume\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:57:21.682753 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:21.682641 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62e2229c-7d88-4c58-9246-7f063cebe2d7-openshift-service-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:57:21.683883 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:21.683860 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62e2229c-7d88-4c58-9246-7f063cebe2d7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "62e2229c-7d88-4c58-9246-7f063cebe2d7" (UID: "62e2229c-7d88-4c58-9246-7f063cebe2d7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:57:21.684326 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:21.684303 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/354e46a3-31a2-4b72-8cdc-d80f08a315b6-kube-api-access-2zhx6" (OuterVolumeSpecName: "kube-api-access-2zhx6") pod "354e46a3-31a2-4b72-8cdc-d80f08a315b6" (UID: "354e46a3-31a2-4b72-8cdc-d80f08a315b6"). InnerVolumeSpecName "kube-api-access-2zhx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:57:21.684455 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:21.684430 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62e2229c-7d88-4c58-9246-7f063cebe2d7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "62e2229c-7d88-4c58-9246-7f063cebe2d7" (UID: "62e2229c-7d88-4c58-9246-7f063cebe2d7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:57:21.684718 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:21.684697 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62e2229c-7d88-4c58-9246-7f063cebe2d7-kube-api-access-gvt2j" (OuterVolumeSpecName: "kube-api-access-gvt2j") pod "62e2229c-7d88-4c58-9246-7f063cebe2d7" (UID: "62e2229c-7d88-4c58-9246-7f063cebe2d7"). InnerVolumeSpecName "kube-api-access-gvt2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:57:21.718128 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:21.718108 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:21.783732 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:21.783666 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62e2229c-7d88-4c58-9246-7f063cebe2d7-maas-api-tls\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:57:21.783732 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:21.783687 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvt2j\" (UniqueName: \"kubernetes.io/projected/62e2229c-7d88-4c58-9246-7f063cebe2d7-kube-api-access-gvt2j\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:57:21.783732 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:21.783697 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zhx6\" (UniqueName: \"kubernetes.io/projected/354e46a3-31a2-4b72-8cdc-d80f08a315b6-kube-api-access-2zhx6\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:57:21.783732 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:21.783706 2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/62e2229c-7d88-4c58-9246-7f063cebe2d7-metrics-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:57:22.359272 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:22.359237 2583 generic.go:356] "Generic (PLEG): container finished" podID="62e2229c-7d88-4c58-9246-7f063cebe2d7" containerID="5f8692a3eb438230e67b30f55b1142659378994f8f1de9aa245bb4a0e3246524" exitCode=0 Sep 10 15:57:22.359433 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:22.359278 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8d528235-54d696866f-rthbp" event={"ID":"62e2229c-7d88-4c58-9246-7f063cebe2d7","Type":"ContainerDied","Data":"5f8692a3eb438230e67b30f55b1142659378994f8f1de9aa245bb4a0e3246524"} Sep 10 15:57:22.359433 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:22.359334 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8d528235-54d696866f-rthbp" event={"ID":"62e2229c-7d88-4c58-9246-7f063cebe2d7","Type":"ContainerDied","Data":"4199ac6f23aa3f21ff7853b86da28b55e5bb8d2f8036299f83a686270ccae0b2"} Sep 10 15:57:22.359433 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:22.359355 2583 scope.go:117] "RemoveContainer" containerID="5f8692a3eb438230e67b30f55b1142659378994f8f1de9aa245bb4a0e3246524" Sep 10 15:57:22.359433 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:22.359288 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-8d528235-54d696866f-rthbp" Sep 10 15:57:22.361022 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:22.360990 2583 generic.go:356] "Generic (PLEG): container finished" podID="354e46a3-31a2-4b72-8cdc-d80f08a315b6" containerID="40c93bd659c2b23afd0ac7ebd01b031aa6a167371013fb0fe1468d6d2d2032e2" exitCode=0 Sep 10 15:57:22.361105 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:22.361051 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-8d528235-5687bc88c8-s7jnp" Sep 10 15:57:22.361105 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:22.361069 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-8d528235-5687bc88c8-s7jnp" event={"ID":"354e46a3-31a2-4b72-8cdc-d80f08a315b6","Type":"ContainerDied","Data":"40c93bd659c2b23afd0ac7ebd01b031aa6a167371013fb0fe1468d6d2d2032e2"} Sep 10 15:57:22.361178 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:22.361133 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-8d528235-5687bc88c8-s7jnp" event={"ID":"354e46a3-31a2-4b72-8cdc-d80f08a315b6","Type":"ContainerDied","Data":"497190818d1e2733ad765f81d32c5836c1ad800c3ac6db977958d481dfe800ae"} Sep 10 15:57:22.369530 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:22.369511 2583 scope.go:117] "RemoveContainer" containerID="5f8692a3eb438230e67b30f55b1142659378994f8f1de9aa245bb4a0e3246524" Sep 10 15:57:22.369930 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:57:22.369787 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f8692a3eb438230e67b30f55b1142659378994f8f1de9aa245bb4a0e3246524\": container with ID starting with 5f8692a3eb438230e67b30f55b1142659378994f8f1de9aa245bb4a0e3246524 not found: ID does not exist" containerID="5f8692a3eb438230e67b30f55b1142659378994f8f1de9aa245bb4a0e3246524" Sep 10 15:57:22.369930 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:22.369814 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f8692a3eb438230e67b30f55b1142659378994f8f1de9aa245bb4a0e3246524"} err="failed to get container status \"5f8692a3eb438230e67b30f55b1142659378994f8f1de9aa245bb4a0e3246524\": rpc error: code = NotFound desc = could not find container \"5f8692a3eb438230e67b30f55b1142659378994f8f1de9aa245bb4a0e3246524\": container with ID starting with 5f8692a3eb438230e67b30f55b1142659378994f8f1de9aa245bb4a0e3246524 not found: ID does not exist" Sep 10 15:57:22.369930 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:22.369830 2583 scope.go:117] "RemoveContainer" containerID="40c93bd659c2b23afd0ac7ebd01b031aa6a167371013fb0fe1468d6d2d2032e2" Sep 10 15:57:22.378600 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:22.378583 2583 scope.go:117] "RemoveContainer" containerID="40c93bd659c2b23afd0ac7ebd01b031aa6a167371013fb0fe1468d6d2d2032e2" Sep 10 15:57:22.378855 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:57:22.378840 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40c93bd659c2b23afd0ac7ebd01b031aa6a167371013fb0fe1468d6d2d2032e2\": container with ID starting with 40c93bd659c2b23afd0ac7ebd01b031aa6a167371013fb0fe1468d6d2d2032e2 not found: ID does not exist" containerID="40c93bd659c2b23afd0ac7ebd01b031aa6a167371013fb0fe1468d6d2d2032e2" Sep 10 15:57:22.378902 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:22.378860 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40c93bd659c2b23afd0ac7ebd01b031aa6a167371013fb0fe1468d6d2d2032e2"} err="failed to get container status \"40c93bd659c2b23afd0ac7ebd01b031aa6a167371013fb0fe1468d6d2d2032e2\": rpc error: code = NotFound desc = could not find container \"40c93bd659c2b23afd0ac7ebd01b031aa6a167371013fb0fe1468d6d2d2032e2\": container with ID starting with 40c93bd659c2b23afd0ac7ebd01b031aa6a167371013fb0fe1468d6d2d2032e2 not found: ID does not exist" Sep 10 15:57:22.440514 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:22.440484 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:22.441476 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:22.441450 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8d528235-5687bc88c8-s7jnp"] Sep 10 15:57:22.446244 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:22.446220 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-8d528235-5687bc88c8-s7jnp"] Sep 10 15:57:22.469426 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:22.469400 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8d528235-54d696866f-rthbp"] Sep 10 15:57:22.473315 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:22.473294 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-8d528235-54d696866f-rthbp"] Sep 10 15:57:22.627450 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:22.627363 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:23.515446 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:23.515419 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="354e46a3-31a2-4b72-8cdc-d80f08a315b6" path="/var/lib/kubelet/pods/354e46a3-31a2-4b72-8cdc-d80f08a315b6/volumes" Sep 10 15:57:23.515826 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:23.515812 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62e2229c-7d88-4c58-9246-7f063cebe2d7" path="/var/lib/kubelet/pods/62e2229c-7d88-4c58-9246-7f063cebe2d7/volumes" Sep 10 15:57:24.019342 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:24.019309 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:24.119057 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:24.119028 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:24.672479 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:24.672447 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-87a024-674c46cd56-r7pwq"] Sep 10 15:57:24.672678 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:24.672654 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-87a024-674c46cd56-r7pwq" podUID="fff6751f-b96a-42ec-b10f-89ba9ec4c0f2" containerName="maas-api" containerID="cri-o://72525ca6f22eacd545dcaefe5218e16604f30353bf556403811b93084dd499c7" gracePeriod=30 Sep 10 15:57:24.703239 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:24.703215 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:24.798486 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:24.798449 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:24.879575 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:24.879533 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:24.922938 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:24.922882 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-87a024-674c46cd56-r7pwq" Sep 10 15:57:24.979284 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:24.979250 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:25.012460 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:25.012429 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fff6751f-b96a-42ec-b10f-89ba9ec4c0f2-openshift-service-ca\") pod \"fff6751f-b96a-42ec-b10f-89ba9ec4c0f2\" (UID: \"fff6751f-b96a-42ec-b10f-89ba9ec4c0f2\") " Sep 10 15:57:25.012641 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:25.012471 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fff6751f-b96a-42ec-b10f-89ba9ec4c0f2-metrics-certs\") pod \"fff6751f-b96a-42ec-b10f-89ba9ec4c0f2\" (UID: \"fff6751f-b96a-42ec-b10f-89ba9ec4c0f2\") " Sep 10 15:57:25.012641 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:25.012555 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fff6751f-b96a-42ec-b10f-89ba9ec4c0f2-maas-api-tls\") pod \"fff6751f-b96a-42ec-b10f-89ba9ec4c0f2\" (UID: \"fff6751f-b96a-42ec-b10f-89ba9ec4c0f2\") " Sep 10 15:57:25.012641 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:25.012610 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9bh4\" (UniqueName: \"kubernetes.io/projected/fff6751f-b96a-42ec-b10f-89ba9ec4c0f2-kube-api-access-c9bh4\") pod \"fff6751f-b96a-42ec-b10f-89ba9ec4c0f2\" (UID: \"fff6751f-b96a-42ec-b10f-89ba9ec4c0f2\") " Sep 10 15:57:25.012930 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:25.012897 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fff6751f-b96a-42ec-b10f-89ba9ec4c0f2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fff6751f-b96a-42ec-b10f-89ba9ec4c0f2" (UID: "fff6751f-b96a-42ec-b10f-89ba9ec4c0f2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:57:25.014827 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:25.014798 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fff6751f-b96a-42ec-b10f-89ba9ec4c0f2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fff6751f-b96a-42ec-b10f-89ba9ec4c0f2" (UID: "fff6751f-b96a-42ec-b10f-89ba9ec4c0f2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:57:25.014827 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:25.014814 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fff6751f-b96a-42ec-b10f-89ba9ec4c0f2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fff6751f-b96a-42ec-b10f-89ba9ec4c0f2" (UID: "fff6751f-b96a-42ec-b10f-89ba9ec4c0f2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:57:25.014958 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:25.014836 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fff6751f-b96a-42ec-b10f-89ba9ec4c0f2-kube-api-access-c9bh4" (OuterVolumeSpecName: "kube-api-access-c9bh4") pod "fff6751f-b96a-42ec-b10f-89ba9ec4c0f2" (UID: "fff6751f-b96a-42ec-b10f-89ba9ec4c0f2"). InnerVolumeSpecName "kube-api-access-c9bh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:57:25.050163 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:25.050134 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:25.113913 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:25.113881 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fff6751f-b96a-42ec-b10f-89ba9ec4c0f2-maas-api-tls\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:57:25.113913 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:25.113909 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9bh4\" (UniqueName: \"kubernetes.io/projected/fff6751f-b96a-42ec-b10f-89ba9ec4c0f2-kube-api-access-c9bh4\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:57:25.113913 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:25.113918 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fff6751f-b96a-42ec-b10f-89ba9ec4c0f2-openshift-service-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:57:25.114122 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:25.113927 2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fff6751f-b96a-42ec-b10f-89ba9ec4c0f2-metrics-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:57:25.380152 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:25.380118 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:25.381950 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:25.381920 2583 generic.go:356] "Generic (PLEG): container finished" podID="fff6751f-b96a-42ec-b10f-89ba9ec4c0f2" containerID="72525ca6f22eacd545dcaefe5218e16604f30353bf556403811b93084dd499c7" exitCode=0 Sep 10 15:57:25.382056 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:25.381968 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-87a024-674c46cd56-r7pwq" Sep 10 15:57:25.382056 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:25.381999 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-87a024-674c46cd56-r7pwq" event={"ID":"fff6751f-b96a-42ec-b10f-89ba9ec4c0f2","Type":"ContainerDied","Data":"72525ca6f22eacd545dcaefe5218e16604f30353bf556403811b93084dd499c7"} Sep 10 15:57:25.382056 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:25.382048 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-87a024-674c46cd56-r7pwq" event={"ID":"fff6751f-b96a-42ec-b10f-89ba9ec4c0f2","Type":"ContainerDied","Data":"254029a98d8f0589e33d74ab74d2d0aa614739dfb9fe35ae3ded8296437c6e7e"} Sep 10 15:57:25.382169 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:25.382071 2583 scope.go:117] "RemoveContainer" containerID="72525ca6f22eacd545dcaefe5218e16604f30353bf556403811b93084dd499c7" Sep 10 15:57:25.391732 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:25.391716 2583 scope.go:117] "RemoveContainer" containerID="72525ca6f22eacd545dcaefe5218e16604f30353bf556403811b93084dd499c7" Sep 10 15:57:25.392001 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:57:25.391982 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72525ca6f22eacd545dcaefe5218e16604f30353bf556403811b93084dd499c7\": container with ID starting with 72525ca6f22eacd545dcaefe5218e16604f30353bf556403811b93084dd499c7 not found: ID does not exist" containerID="72525ca6f22eacd545dcaefe5218e16604f30353bf556403811b93084dd499c7" Sep 10 15:57:25.392061 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:25.392012 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72525ca6f22eacd545dcaefe5218e16604f30353bf556403811b93084dd499c7"} err="failed to get container status \"72525ca6f22eacd545dcaefe5218e16604f30353bf556403811b93084dd499c7\": rpc error: code = NotFound desc = could not find container \"72525ca6f22eacd545dcaefe5218e16604f30353bf556403811b93084dd499c7\": container with ID starting with 72525ca6f22eacd545dcaefe5218e16604f30353bf556403811b93084dd499c7 not found: ID does not exist" Sep 10 15:57:25.410485 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:25.410458 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-87a024-674c46cd56-r7pwq"] Sep 10 15:57:25.415201 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:25.415181 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-87a024-674c46cd56-r7pwq"] Sep 10 15:57:25.515923 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:25.515901 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fff6751f-b96a-42ec-b10f-89ba9ec4c0f2" path="/var/lib/kubelet/pods/fff6751f-b96a-42ec-b10f-89ba9ec4c0f2/volumes" Sep 10 15:57:30.508894 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:30.508854 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:30.611126 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:30.611096 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:30.711986 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:30.711955 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:30.813483 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:30.813408 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:31.235156 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:31.235125 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:32.769224 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:32.769189 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:32.869587 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:32.869547 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:33.063682 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:33.063596 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:33.491704 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:33.491664 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:33.784772 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:33.784682 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:33.958305 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:33.958277 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:36.608060 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:36.608025 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:36.706095 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:36.706068 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:37.868359 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:37.868327 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-61fba2-55559789c5-bx75h"] Sep 10 15:57:37.868974 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:37.868951 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="354e46a3-31a2-4b72-8cdc-d80f08a315b6" containerName="payload-pre-processing" Sep 10 15:57:37.869076 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:37.868977 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="354e46a3-31a2-4b72-8cdc-d80f08a315b6" containerName="payload-pre-processing" Sep 10 15:57:37.869076 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:37.868990 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3043223-0816-40f8-b9cc-2bc5830c61ff" containerName="istio-proxy" Sep 10 15:57:37.869076 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:37.868998 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3043223-0816-40f8-b9cc-2bc5830c61ff" containerName="istio-proxy" Sep 10 15:57:37.869076 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:37.869009 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62e2229c-7d88-4c58-9246-7f063cebe2d7" containerName="maas-api" Sep 10 15:57:37.869076 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:37.869017 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="62e2229c-7d88-4c58-9246-7f063cebe2d7" containerName="maas-api" Sep 10 15:57:37.869076 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:37.869028 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fff6751f-b96a-42ec-b10f-89ba9ec4c0f2" containerName="maas-api" Sep 10 15:57:37.869076 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:37.869036 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fff6751f-b96a-42ec-b10f-89ba9ec4c0f2" containerName="maas-api" Sep 10 15:57:37.869434 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:37.869143 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="354e46a3-31a2-4b72-8cdc-d80f08a315b6" containerName="payload-pre-processing" Sep 10 15:57:37.869434 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:37.869163 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="62e2229c-7d88-4c58-9246-7f063cebe2d7" containerName="maas-api" Sep 10 15:57:37.869434 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:37.869173 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fff6751f-b96a-42ec-b10f-89ba9ec4c0f2" containerName="maas-api" Sep 10 15:57:37.872589 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:37.872567 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-61fba2-55559789c5-bx75h" Sep 10 15:57:37.875666 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:37.875641 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-61fba2-serving-cert\"" Sep 10 15:57:37.886314 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:37.886260 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-61fba2-55559789c5-bx75h"] Sep 10 15:57:37.916211 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:37.916180 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:37.931440 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:37.931414 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a-openshift-service-ca\") pod \"maas-api-e2e-explicit-61fba2-55559789c5-bx75h\" (UID: \"628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-61fba2-55559789c5-bx75h" Sep 10 15:57:37.931555 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:37.931445 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a-maas-api-tls\") pod \"maas-api-e2e-explicit-61fba2-55559789c5-bx75h\" (UID: \"628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-61fba2-55559789c5-bx75h" Sep 10 15:57:37.931555 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:37.931487 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j2vm\" (UniqueName: \"kubernetes.io/projected/628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a-kube-api-access-6j2vm\") pod \"maas-api-e2e-explicit-61fba2-55559789c5-bx75h\" (UID: \"628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-61fba2-55559789c5-bx75h" Sep 10 15:57:37.931555 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:37.931505 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a-metrics-certs\") pod \"maas-api-e2e-explicit-61fba2-55559789c5-bx75h\" (UID: \"628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-61fba2-55559789c5-bx75h" Sep 10 15:57:37.989509 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:37.989479 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:38.032792 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:38.032761 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a-openshift-service-ca\") pod \"maas-api-e2e-explicit-61fba2-55559789c5-bx75h\" (UID: \"628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-61fba2-55559789c5-bx75h" Sep 10 15:57:38.032937 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:38.032798 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a-maas-api-tls\") pod \"maas-api-e2e-explicit-61fba2-55559789c5-bx75h\" (UID: \"628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-61fba2-55559789c5-bx75h" Sep 10 15:57:38.032937 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:38.032851 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j2vm\" (UniqueName: \"kubernetes.io/projected/628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a-kube-api-access-6j2vm\") pod \"maas-api-e2e-explicit-61fba2-55559789c5-bx75h\" (UID: \"628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-61fba2-55559789c5-bx75h" Sep 10 15:57:38.032937 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:38.032870 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a-metrics-certs\") pod \"maas-api-e2e-explicit-61fba2-55559789c5-bx75h\" (UID: \"628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-61fba2-55559789c5-bx75h" Sep 10 15:57:38.033067 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:57:38.032992 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-61fba2-serving-cert: secret "maas-api-e2e-explicit-61fba2-serving-cert" not found Sep 10 15:57:38.033067 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:57:38.033064 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a-maas-api-tls podName:628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a nodeName:}" failed. No retries permitted until 2026-09-10 15:57:38.533040698 +0000 UTC m=+935.508300652 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a-maas-api-tls") pod "maas-api-e2e-explicit-61fba2-55559789c5-bx75h" (UID: "628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a") : secret "maas-api-e2e-explicit-61fba2-serving-cert" not found Sep 10 15:57:38.033452 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:38.033426 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a-openshift-service-ca\") pod \"maas-api-e2e-explicit-61fba2-55559789c5-bx75h\" (UID: \"628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-61fba2-55559789c5-bx75h" Sep 10 15:57:38.035452 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:38.035426 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a-metrics-certs\") pod \"maas-api-e2e-explicit-61fba2-55559789c5-bx75h\" (UID: \"628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-61fba2-55559789c5-bx75h" Sep 10 15:57:38.043135 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:38.043107 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j2vm\" (UniqueName: \"kubernetes.io/projected/628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a-kube-api-access-6j2vm\") pod \"maas-api-e2e-explicit-61fba2-55559789c5-bx75h\" (UID: \"628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-61fba2-55559789c5-bx75h" Sep 10 15:57:38.486353 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:38.486321 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:38.537170 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:38.537141 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a-maas-api-tls\") pod \"maas-api-e2e-explicit-61fba2-55559789c5-bx75h\" (UID: \"628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-61fba2-55559789c5-bx75h" Sep 10 15:57:38.545631 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:38.541055 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a-maas-api-tls\") pod \"maas-api-e2e-explicit-61fba2-55559789c5-bx75h\" (UID: \"628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-61fba2-55559789c5-bx75h" Sep 10 15:57:38.590899 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:38.590857 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:38.782645 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:38.782566 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-61fba2-8567f5f97-58w4g"] Sep 10 15:57:38.786555 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:38.786538 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-61fba2-8567f5f97-58w4g" Sep 10 15:57:38.786749 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:38.786729 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-61fba2-55559789c5-bx75h" Sep 10 15:57:38.790250 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:38.790231 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-61fba2-dockercfg-zcz8c\"" Sep 10 15:57:38.790897 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:38.790879 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-61fba2\"" Sep 10 15:57:38.800755 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:38.800732 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-61fba2-8567f5f97-58w4g"] Sep 10 15:57:38.840871 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:38.840837 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwcsn\" (UniqueName: \"kubernetes.io/projected/4d98af8b-06f8-46a7-aa44-1e29d3b81f4f-kube-api-access-rwcsn\") pod \"payload-processing-e2e-explicit-61fba2-8567f5f97-58w4g\" (UID: \"4d98af8b-06f8-46a7-aa44-1e29d3b81f4f\") " pod="openshift-ingress/payload-processing-e2e-explicit-61fba2-8567f5f97-58w4g" Sep 10 15:57:38.841143 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:38.841118 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d98af8b-06f8-46a7-aa44-1e29d3b81f4f-plugins-config-volume\") pod \"payload-processing-e2e-explicit-61fba2-8567f5f97-58w4g\" (UID: \"4d98af8b-06f8-46a7-aa44-1e29d3b81f4f\") " pod="openshift-ingress/payload-processing-e2e-explicit-61fba2-8567f5f97-58w4g" Sep 10 15:57:38.942896 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:38.942865 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwcsn\" (UniqueName: \"kubernetes.io/projected/4d98af8b-06f8-46a7-aa44-1e29d3b81f4f-kube-api-access-rwcsn\") pod \"payload-processing-e2e-explicit-61fba2-8567f5f97-58w4g\" (UID: \"4d98af8b-06f8-46a7-aa44-1e29d3b81f4f\") " pod="openshift-ingress/payload-processing-e2e-explicit-61fba2-8567f5f97-58w4g" Sep 10 15:57:38.943335 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:38.942927 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d98af8b-06f8-46a7-aa44-1e29d3b81f4f-plugins-config-volume\") pod \"payload-processing-e2e-explicit-61fba2-8567f5f97-58w4g\" (UID: \"4d98af8b-06f8-46a7-aa44-1e29d3b81f4f\") " pod="openshift-ingress/payload-processing-e2e-explicit-61fba2-8567f5f97-58w4g" Sep 10 15:57:38.943811 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:38.943787 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d98af8b-06f8-46a7-aa44-1e29d3b81f4f-plugins-config-volume\") pod \"payload-processing-e2e-explicit-61fba2-8567f5f97-58w4g\" (UID: \"4d98af8b-06f8-46a7-aa44-1e29d3b81f4f\") " pod="openshift-ingress/payload-processing-e2e-explicit-61fba2-8567f5f97-58w4g" Sep 10 15:57:38.952294 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:38.952274 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwcsn\" (UniqueName: \"kubernetes.io/projected/4d98af8b-06f8-46a7-aa44-1e29d3b81f4f-kube-api-access-rwcsn\") pod \"payload-processing-e2e-explicit-61fba2-8567f5f97-58w4g\" (UID: \"4d98af8b-06f8-46a7-aa44-1e29d3b81f4f\") " pod="openshift-ingress/payload-processing-e2e-explicit-61fba2-8567f5f97-58w4g" Sep 10 15:57:39.075175 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:39.075143 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-61fba2-55559789c5-bx75h"] Sep 10 15:57:39.075599 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:57:39.075574 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod628c62cd_cf25_4f0c_857e_2dbc5d1c1a6a.slice/crio-3ace4c658ce18893e7077aaaa37b2408af06e8d9d8d64f59a786117a1b31e00f WatchSource:0}: Error finding container 3ace4c658ce18893e7077aaaa37b2408af06e8d9d8d64f59a786117a1b31e00f: Status 404 returned error can't find the container with id 3ace4c658ce18893e7077aaaa37b2408af06e8d9d8d64f59a786117a1b31e00f Sep 10 15:57:39.098248 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:39.098226 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-61fba2-8567f5f97-58w4g" Sep 10 15:57:39.381471 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:39.381444 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-61fba2-8567f5f97-58w4g"] Sep 10 15:57:39.382862 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:57:39.382835 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d98af8b_06f8_46a7_aa44_1e29d3b81f4f.slice/crio-48e7c052d67bc62d83b73097d9b12d658f6a33ec1c877aa7c0777285c6e98862 WatchSource:0}: Error finding container 48e7c052d67bc62d83b73097d9b12d658f6a33ec1c877aa7c0777285c6e98862: Status 404 returned error can't find the container with id 48e7c052d67bc62d83b73097d9b12d658f6a33ec1c877aa7c0777285c6e98862 Sep 10 15:57:39.470282 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:39.470234 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-61fba2-55559789c5-bx75h" event={"ID":"628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a","Type":"ContainerStarted","Data":"3ace4c658ce18893e7077aaaa37b2408af06e8d9d8d64f59a786117a1b31e00f"} Sep 10 15:57:39.471964 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:39.471938 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-61fba2-8567f5f97-58w4g" event={"ID":"4d98af8b-06f8-46a7-aa44-1e29d3b81f4f","Type":"ContainerStarted","Data":"c5ee29fe23bf6acaa290ac28d095895a96309a64567fd9878c4bcf97ec7d4ed4"} Sep 10 15:57:39.472070 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:39.471985 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-61fba2-8567f5f97-58w4g" event={"ID":"4d98af8b-06f8-46a7-aa44-1e29d3b81f4f","Type":"ContainerStarted","Data":"48e7c052d67bc62d83b73097d9b12d658f6a33ec1c877aa7c0777285c6e98862"} Sep 10 15:57:39.472070 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:39.472056 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-61fba2-8567f5f97-58w4g" Sep 10 15:57:39.491446 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:39.491407 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-61fba2-8567f5f97-58w4g" podStartSLOduration=1.491369945 podStartE2EDuration="1.491369945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:57:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:57:39.490498653 +0000 UTC m=+936.465758607" watchObservedRunningTime="2026-09-10 15:57:39.491369945 +0000 UTC m=+936.466629925" Sep 10 15:57:40.480699 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:40.480662 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-61fba2-55559789c5-bx75h" event={"ID":"628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a","Type":"ContainerStarted","Data":"8247e91a947b1799d92901d383c4bfd8ad55c55375bf54ea3c24c3cb3f9f7740"} Sep 10 15:57:40.481120 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:40.480896 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-61fba2-55559789c5-bx75h" Sep 10 15:57:40.503271 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:40.503222 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-61fba2-55559789c5-bx75h" podStartSLOduration=2.958119602 podStartE2EDuration="3.503208291s" totalImagesPullingTime="545.088689ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:57:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:57:40.500644058 +0000 UTC m=+937.475904011" watchObservedRunningTime="2026-09-10 15:57:40.503208291 +0000 UTC m=+937.478468243" Sep 10 15:57:43.222617 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:43.222583 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:43.326718 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:43.326685 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:46.492221 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:46.492193 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-61fba2-55559789c5-bx75h" Sep 10 15:57:50.483517 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:50.483483 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-61fba2-8567f5f97-58w4g" Sep 10 15:57:51.286711 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:51.286675 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:57:57.160698 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:57:57.160635 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:01.584146 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:01.584106 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:04.596225 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:04.596188 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:04.679508 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:04.679474 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:04.792362 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:04.792328 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:13.589799 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:13.589764 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:20.755699 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:20.755664 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:27.163846 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:27.163768 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:27.277696 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:27.277671 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:27.364778 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:27.364747 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:29.269737 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:29.269706 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:34.025979 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:34.025941 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:37.885922 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:37.885858 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:38.705249 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:38.705210 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:44.650915 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:44.650880 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-61fba2-55559789c5-bx75h"] Sep 10 15:58:44.651397 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:44.651152 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-61fba2-55559789c5-bx75h" podUID="628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a" containerName="maas-api" containerID="cri-o://8247e91a947b1799d92901d383c4bfd8ad55c55375bf54ea3c24c3cb3f9f7740" gracePeriod=30 Sep 10 15:58:44.683824 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:44.683741 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:44.716096 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:44.716068 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-61fba2-8567f5f97-58w4g"] Sep 10 15:58:44.716332 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:44.716278 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-61fba2-8567f5f97-58w4g" podUID="4d98af8b-06f8-46a7-aa44-1e29d3b81f4f" containerName="payload-processing" containerID="cri-o://c5ee29fe23bf6acaa290ac28d095895a96309a64567fd9878c4bcf97ec7d4ed4" gracePeriod=30 Sep 10 15:58:44.784428 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:44.784397 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:44.852609 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:44.852571 2583 generic.go:356] "Generic (PLEG): container finished" podID="628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a" containerID="8247e91a947b1799d92901d383c4bfd8ad55c55375bf54ea3c24c3cb3f9f7740" exitCode=0 Sep 10 15:58:44.852785 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:44.852723 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-61fba2-55559789c5-bx75h" event={"ID":"628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a","Type":"ContainerDied","Data":"8247e91a947b1799d92901d383c4bfd8ad55c55375bf54ea3c24c3cb3f9f7740"} Sep 10 15:58:44.855130 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:44.855002 2583 generic.go:356] "Generic (PLEG): container finished" podID="4d98af8b-06f8-46a7-aa44-1e29d3b81f4f" containerID="c5ee29fe23bf6acaa290ac28d095895a96309a64567fd9878c4bcf97ec7d4ed4" exitCode=0 Sep 10 15:58:44.855130 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:44.855071 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-61fba2-8567f5f97-58w4g" event={"ID":"4d98af8b-06f8-46a7-aa44-1e29d3b81f4f","Type":"ContainerDied","Data":"c5ee29fe23bf6acaa290ac28d095895a96309a64567fd9878c4bcf97ec7d4ed4"} Sep 10 15:58:44.885114 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:44.885084 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:44.956568 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:44.956538 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:44.982793 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:44.982769 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-61fba2-55559789c5-bx75h" Sep 10 15:58:45.003043 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:45.002968 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-61fba2-8567f5f97-58w4g" Sep 10 15:58:45.024867 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:45.024450 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6j2vm\" (UniqueName: \"kubernetes.io/projected/628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a-kube-api-access-6j2vm\") pod \"628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a\" (UID: \"628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a\") " Sep 10 15:58:45.025031 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:45.024955 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a-openshift-service-ca\") pod \"628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a\" (UID: \"628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a\") " Sep 10 15:58:45.025104 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:45.025035 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a-metrics-certs\") pod \"628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a\" (UID: \"628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a\") " Sep 10 15:58:45.026798 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:45.025334 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a-maas-api-tls\") pod \"628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a\" (UID: \"628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a\") " Sep 10 15:58:45.028243 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:45.027978 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a" (UID: "628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:58:45.032422 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:45.029754 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a-kube-api-access-6j2vm" (OuterVolumeSpecName: "kube-api-access-6j2vm") pod "628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a" (UID: "628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a"). InnerVolumeSpecName "kube-api-access-6j2vm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:58:45.036614 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:45.036582 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a" (UID: "628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:58:45.036699 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:45.036605 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a" (UID: "628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:58:45.128133 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:45.128110 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwcsn\" (UniqueName: \"kubernetes.io/projected/4d98af8b-06f8-46a7-aa44-1e29d3b81f4f-kube-api-access-rwcsn\") pod \"4d98af8b-06f8-46a7-aa44-1e29d3b81f4f\" (UID: \"4d98af8b-06f8-46a7-aa44-1e29d3b81f4f\") " Sep 10 15:58:45.128241 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:45.128211 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d98af8b-06f8-46a7-aa44-1e29d3b81f4f-plugins-config-volume\") pod \"4d98af8b-06f8-46a7-aa44-1e29d3b81f4f\" (UID: \"4d98af8b-06f8-46a7-aa44-1e29d3b81f4f\") " Sep 10 15:58:45.128561 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:45.128539 2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a-metrics-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:58:45.128561 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:45.128559 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a-maas-api-tls\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:58:45.128709 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:45.128568 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j2vm\" (UniqueName: \"kubernetes.io/projected/628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a-kube-api-access-6j2vm\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:58:45.128709 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:45.128578 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a-openshift-service-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:58:45.128709 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:45.128581 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d98af8b-06f8-46a7-aa44-1e29d3b81f4f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4d98af8b-06f8-46a7-aa44-1e29d3b81f4f" (UID: "4d98af8b-06f8-46a7-aa44-1e29d3b81f4f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:58:45.130081 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:45.130057 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d98af8b-06f8-46a7-aa44-1e29d3b81f4f-kube-api-access-rwcsn" (OuterVolumeSpecName: "kube-api-access-rwcsn") pod "4d98af8b-06f8-46a7-aa44-1e29d3b81f4f" (UID: "4d98af8b-06f8-46a7-aa44-1e29d3b81f4f"). InnerVolumeSpecName "kube-api-access-rwcsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:58:45.229845 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:45.229774 2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d98af8b-06f8-46a7-aa44-1e29d3b81f4f-plugins-config-volume\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:58:45.229845 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:45.229800 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwcsn\" (UniqueName: \"kubernetes.io/projected/4d98af8b-06f8-46a7-aa44-1e29d3b81f4f-kube-api-access-rwcsn\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:58:45.299739 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:45.299697 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:45.480021 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:45.479944 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:45.581276 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:45.581249 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:45.678862 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:45.678836 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:45.783724 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:45.783662 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:45.862120 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:45.862098 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-61fba2-8567f5f97-58w4g" Sep 10 15:58:45.862237 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:45.862128 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-61fba2-8567f5f97-58w4g" event={"ID":"4d98af8b-06f8-46a7-aa44-1e29d3b81f4f","Type":"ContainerDied","Data":"48e7c052d67bc62d83b73097d9b12d658f6a33ec1c877aa7c0777285c6e98862"} Sep 10 15:58:45.862237 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:45.862189 2583 scope.go:117] "RemoveContainer" containerID="c5ee29fe23bf6acaa290ac28d095895a96309a64567fd9878c4bcf97ec7d4ed4" Sep 10 15:58:45.863738 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:45.863716 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-61fba2-55559789c5-bx75h" event={"ID":"628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a","Type":"ContainerDied","Data":"3ace4c658ce18893e7077aaaa37b2408af06e8d9d8d64f59a786117a1b31e00f"} Sep 10 15:58:45.863836 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:45.863746 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-61fba2-55559789c5-bx75h" Sep 10 15:58:45.873474 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:45.873459 2583 scope.go:117] "RemoveContainer" containerID="8247e91a947b1799d92901d383c4bfd8ad55c55375bf54ea3c24c3cb3f9f7740" Sep 10 15:58:45.885436 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:45.885412 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-61fba2-8567f5f97-58w4g"] Sep 10 15:58:45.888044 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:45.888019 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-61fba2-8567f5f97-58w4g"] Sep 10 15:58:45.904050 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:45.904029 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-61fba2-55559789c5-bx75h"] Sep 10 15:58:45.909350 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:45.909331 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-61fba2-55559789c5-bx75h"] Sep 10 15:58:46.450307 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:46.450276 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:46.550752 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:46.550723 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:46.650501 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:46.650467 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:47.515183 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:47.515147 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d98af8b-06f8-46a7-aa44-1e29d3b81f4f" path="/var/lib/kubelet/pods/4d98af8b-06f8-46a7-aa44-1e29d3b81f4f/volumes" Sep 10 15:58:47.515591 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:47.515575 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a" path="/var/lib/kubelet/pods/628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a/volumes" Sep 10 15:58:48.206429 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:48.206392 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:52.096372 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:52.096334 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:52.187255 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:52.187228 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:52.384717 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:52.384688 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:52.809416 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:52.809324 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:53.117963 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:53.117886 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x"] Sep 10 15:58:53.118513 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:53.118496 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d98af8b-06f8-46a7-aa44-1e29d3b81f4f" containerName="payload-processing" Sep 10 15:58:53.118563 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:53.118518 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d98af8b-06f8-46a7-aa44-1e29d3b81f4f" containerName="payload-processing" Sep 10 15:58:53.118599 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:53.118565 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a" containerName="maas-api" Sep 10 15:58:53.118599 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:53.118574 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a" containerName="maas-api" Sep 10 15:58:53.118674 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:53.118664 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="628c62cd-cf25-4f0c-857e-2dbc5d1c1a6a" containerName="maas-api" Sep 10 15:58:53.118713 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:53.118680 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d98af8b-06f8-46a7-aa44-1e29d3b81f4f" containerName="payload-processing" Sep 10 15:58:53.124034 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:53.124012 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x" Sep 10 15:58:53.126190 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:53.126168 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-17d9a5-openshift-default-dockercfg-g8xbf\"" Sep 10 15:58:53.135611 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:53.135579 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x"] Sep 10 15:58:53.151337 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:53.151313 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:53.196625 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:53.196588 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/141cb709-5da4-4878-b1ce-733402bae3aa-istio-token\") pod \"e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x\" (UID: \"141cb709-5da4-4878-b1ce-733402bae3aa\") " pod="openshift-ingress/e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x" Sep 10 15:58:53.196796 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:53.196632 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/141cb709-5da4-4878-b1ce-733402bae3aa-workload-socket\") pod \"e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x\" (UID: \"141cb709-5da4-4878-b1ce-733402bae3aa\") " pod="openshift-ingress/e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x" Sep 10 15:58:53.196796 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:53.196674 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/141cb709-5da4-4878-b1ce-733402bae3aa-istio-envoy\") pod \"e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x\" (UID: \"141cb709-5da4-4878-b1ce-733402bae3aa\") " pod="openshift-ingress/e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x" Sep 10 15:58:53.196796 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:53.196701 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/141cb709-5da4-4878-b1ce-733402bae3aa-istio-data\") pod \"e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x\" (UID: \"141cb709-5da4-4878-b1ce-733402bae3aa\") " pod="openshift-ingress/e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x" Sep 10 15:58:53.196970 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:53.196849 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/141cb709-5da4-4878-b1ce-733402bae3aa-workload-certs\") pod \"e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x\" (UID: \"141cb709-5da4-4878-b1ce-733402bae3aa\") " pod="openshift-ingress/e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x" Sep 10 15:58:53.196970 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:53.196893 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/141cb709-5da4-4878-b1ce-733402bae3aa-istiod-ca-cert\") pod \"e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x\" (UID: \"141cb709-5da4-4878-b1ce-733402bae3aa\") " pod="openshift-ingress/e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x" Sep 10 15:58:53.196970 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:53.196919 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/141cb709-5da4-4878-b1ce-733402bae3aa-istio-podinfo\") pod \"e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x\" (UID: \"141cb709-5da4-4878-b1ce-733402bae3aa\") " pod="openshift-ingress/e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x" Sep 10 15:58:53.196970 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:53.196951 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwpcn\" (UniqueName: \"kubernetes.io/projected/141cb709-5da4-4878-b1ce-733402bae3aa-kube-api-access-wwpcn\") pod \"e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x\" (UID: \"141cb709-5da4-4878-b1ce-733402bae3aa\") " pod="openshift-ingress/e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x" Sep 10 15:58:53.197179 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:53.197009 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/141cb709-5da4-4878-b1ce-733402bae3aa-credential-socket\") pod \"e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x\" (UID: \"141cb709-5da4-4878-b1ce-733402bae3aa\") " pod="openshift-ingress/e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x" Sep 10 15:58:53.278748 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:53.278710 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:53.297673 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:53.297645 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/141cb709-5da4-4878-b1ce-733402bae3aa-istio-token\") pod \"e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x\" (UID: \"141cb709-5da4-4878-b1ce-733402bae3aa\") " pod="openshift-ingress/e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x" Sep 10 15:58:53.297673 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:53.297674 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/141cb709-5da4-4878-b1ce-733402bae3aa-workload-socket\") pod \"e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x\" (UID: \"141cb709-5da4-4878-b1ce-733402bae3aa\") " pod="openshift-ingress/e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x" Sep 10 15:58:53.297883 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:53.297810 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/141cb709-5da4-4878-b1ce-733402bae3aa-istio-envoy\") pod \"e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x\" (UID: \"141cb709-5da4-4878-b1ce-733402bae3aa\") " pod="openshift-ingress/e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x" Sep 10 15:58:53.297883 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:53.297855 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/141cb709-5da4-4878-b1ce-733402bae3aa-istio-data\") pod \"e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x\" (UID: \"141cb709-5da4-4878-b1ce-733402bae3aa\") " pod="openshift-ingress/e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x" Sep 10 15:58:53.297996 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:53.297931 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/141cb709-5da4-4878-b1ce-733402bae3aa-workload-certs\") pod \"e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x\" (UID: \"141cb709-5da4-4878-b1ce-733402bae3aa\") " pod="openshift-ingress/e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x" Sep 10 15:58:53.297996 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:53.297984 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/141cb709-5da4-4878-b1ce-733402bae3aa-workload-socket\") pod \"e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x\" (UID: \"141cb709-5da4-4878-b1ce-733402bae3aa\") " pod="openshift-ingress/e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x" Sep 10 15:58:53.298106 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:53.297995 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/141cb709-5da4-4878-b1ce-733402bae3aa-istiod-ca-cert\") pod \"e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x\" (UID: \"141cb709-5da4-4878-b1ce-733402bae3aa\") " pod="openshift-ingress/e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x" Sep 10 15:58:53.298106 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:53.298037 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/141cb709-5da4-4878-b1ce-733402bae3aa-istio-podinfo\") pod \"e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x\" (UID: \"141cb709-5da4-4878-b1ce-733402bae3aa\") " pod="openshift-ingress/e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x" Sep 10 15:58:53.298106 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:53.298072 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwpcn\" (UniqueName: \"kubernetes.io/projected/141cb709-5da4-4878-b1ce-733402bae3aa-kube-api-access-wwpcn\") pod \"e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x\" (UID: \"141cb709-5da4-4878-b1ce-733402bae3aa\") " pod="openshift-ingress/e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x" Sep 10 15:58:53.298270 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:53.298206 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/141cb709-5da4-4878-b1ce-733402bae3aa-istio-data\") pod \"e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x\" (UID: \"141cb709-5da4-4878-b1ce-733402bae3aa\") " pod="openshift-ingress/e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x" Sep 10 15:58:53.298270 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:53.298233 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/141cb709-5da4-4878-b1ce-733402bae3aa-workload-certs\") pod \"e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x\" (UID: \"141cb709-5da4-4878-b1ce-733402bae3aa\") " pod="openshift-ingress/e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x" Sep 10 15:58:53.298374 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:53.298261 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/141cb709-5da4-4878-b1ce-733402bae3aa-credential-socket\") pod \"e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x\" (UID: \"141cb709-5da4-4878-b1ce-733402bae3aa\") " pod="openshift-ingress/e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x" Sep 10 15:58:53.298492 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:53.298470 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/141cb709-5da4-4878-b1ce-733402bae3aa-credential-socket\") pod \"e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x\" (UID: \"141cb709-5da4-4878-b1ce-733402bae3aa\") " pod="openshift-ingress/e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x" Sep 10 15:58:53.298668 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:53.298649 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/141cb709-5da4-4878-b1ce-733402bae3aa-istiod-ca-cert\") pod \"e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x\" (UID: \"141cb709-5da4-4878-b1ce-733402bae3aa\") " pod="openshift-ingress/e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x" Sep 10 15:58:53.300145 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:53.300127 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/141cb709-5da4-4878-b1ce-733402bae3aa-istio-envoy\") pod \"e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x\" (UID: \"141cb709-5da4-4878-b1ce-733402bae3aa\") " pod="openshift-ingress/e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x" Sep 10 15:58:53.300455 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:53.300435 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/141cb709-5da4-4878-b1ce-733402bae3aa-istio-podinfo\") pod \"e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x\" (UID: \"141cb709-5da4-4878-b1ce-733402bae3aa\") " pod="openshift-ingress/e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x" Sep 10 15:58:53.309716 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:53.309688 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/141cb709-5da4-4878-b1ce-733402bae3aa-istio-token\") pod \"e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x\" (UID: \"141cb709-5da4-4878-b1ce-733402bae3aa\") " pod="openshift-ingress/e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x" Sep 10 15:58:53.309806 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:53.309792 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwpcn\" (UniqueName: \"kubernetes.io/projected/141cb709-5da4-4878-b1ce-733402bae3aa-kube-api-access-wwpcn\") pod \"e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x\" (UID: \"141cb709-5da4-4878-b1ce-733402bae3aa\") " pod="openshift-ingress/e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x" Sep 10 15:58:53.435571 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:53.435535 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x" Sep 10 15:58:53.712062 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:53.712037 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x"] Sep 10 15:58:53.713622 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:58:53.713595 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod141cb709_5da4_4878_b1ce_733402bae3aa.slice/crio-0a1ae2730c937669f83af367d14cbbd69495410702f1239fd0f52cb465b62ed7 WatchSource:0}: Error finding container 0a1ae2730c937669f83af367d14cbbd69495410702f1239fd0f52cb465b62ed7: Status 404 returned error can't find the container with id 0a1ae2730c937669f83af367d14cbbd69495410702f1239fd0f52cb465b62ed7 Sep 10 15:58:53.715767 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:53.715735 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 15:58:53.715846 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:53.715800 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 15:58:53.715846 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:53.715827 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 15:58:53.914482 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:53.914444 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x" event={"ID":"141cb709-5da4-4878-b1ce-733402bae3aa","Type":"ContainerStarted","Data":"62edf9521f03425d98c44c1f486cccf786e479aabde54bd66a0d81272ca0d75e"} Sep 10 15:58:53.914639 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:53.914515 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x" event={"ID":"141cb709-5da4-4878-b1ce-733402bae3aa","Type":"ContainerStarted","Data":"0a1ae2730c937669f83af367d14cbbd69495410702f1239fd0f52cb465b62ed7"} Sep 10 15:58:53.942763 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:53.942706 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x" podStartSLOduration=0.942685186 podStartE2EDuration="942.685186ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:58:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:58:53.940980397 +0000 UTC m=+1010.916240358" watchObservedRunningTime="2026-09-10 15:58:53.942685186 +0000 UTC m=+1010.917945133" Sep 10 15:58:54.316662 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:54.316633 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:54.430530 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:54.430497 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:54.436483 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:54.436457 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x" Sep 10 15:58:54.441986 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:54.441964 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x" Sep 10 15:58:54.529342 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:54.529316 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:54.920374 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:54.920339 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x" Sep 10 15:58:54.921310 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:54.921291 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x" Sep 10 15:58:55.746650 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:55.746612 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:55.850893 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:55.850844 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:58:56.463187 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:58:56.463149 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:02.820208 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:02.820177 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:02.950225 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:02.950191 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:03.065518 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:03.065488 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:03.435293 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:03.435257 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:03.624695 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:03.624666 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:06.180517 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:06.180482 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:06.278094 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:06.278054 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:07.318101 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:07.318067 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9a70cb1a-764b99f6f4-f9scl"] Sep 10 15:59:07.323270 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:07.323248 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a70cb1a-764b99f6f4-f9scl" Sep 10 15:59:07.325882 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:07.325861 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-9a70cb1a-serving-cert\"" Sep 10 15:59:07.333952 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:07.333931 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9a70cb1a-764b99f6f4-f9scl"] Sep 10 15:59:07.340529 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:07.340499 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e9e4f69-35d9-40e7-b4a9-65f9fa847512-maas-api-tls\") pod \"maas-api-e2e-mt-9a70cb1a-764b99f6f4-f9scl\" (UID: \"6e9e4f69-35d9-40e7-b4a9-65f9fa847512\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a70cb1a-764b99f6f4-f9scl" Sep 10 15:59:07.340644 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:07.340622 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8ljj\" (UniqueName: \"kubernetes.io/projected/6e9e4f69-35d9-40e7-b4a9-65f9fa847512-kube-api-access-h8ljj\") pod \"maas-api-e2e-mt-9a70cb1a-764b99f6f4-f9scl\" (UID: \"6e9e4f69-35d9-40e7-b4a9-65f9fa847512\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a70cb1a-764b99f6f4-f9scl" Sep 10 15:59:07.340741 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:07.340716 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e9e4f69-35d9-40e7-b4a9-65f9fa847512-openshift-service-ca\") pod \"maas-api-e2e-mt-9a70cb1a-764b99f6f4-f9scl\" (UID: \"6e9e4f69-35d9-40e7-b4a9-65f9fa847512\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a70cb1a-764b99f6f4-f9scl" Sep 10 15:59:07.340795 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:07.340776 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e9e4f69-35d9-40e7-b4a9-65f9fa847512-metrics-certs\") pod \"maas-api-e2e-mt-9a70cb1a-764b99f6f4-f9scl\" (UID: \"6e9e4f69-35d9-40e7-b4a9-65f9fa847512\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a70cb1a-764b99f6f4-f9scl" Sep 10 15:59:07.358494 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:07.358467 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:07.441570 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:07.441537 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8ljj\" (UniqueName: \"kubernetes.io/projected/6e9e4f69-35d9-40e7-b4a9-65f9fa847512-kube-api-access-h8ljj\") pod \"maas-api-e2e-mt-9a70cb1a-764b99f6f4-f9scl\" (UID: \"6e9e4f69-35d9-40e7-b4a9-65f9fa847512\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a70cb1a-764b99f6f4-f9scl" Sep 10 15:59:07.441732 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:07.441595 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e9e4f69-35d9-40e7-b4a9-65f9fa847512-openshift-service-ca\") pod \"maas-api-e2e-mt-9a70cb1a-764b99f6f4-f9scl\" (UID: \"6e9e4f69-35d9-40e7-b4a9-65f9fa847512\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a70cb1a-764b99f6f4-f9scl" Sep 10 15:59:07.441789 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:07.441736 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e9e4f69-35d9-40e7-b4a9-65f9fa847512-metrics-certs\") pod \"maas-api-e2e-mt-9a70cb1a-764b99f6f4-f9scl\" (UID: \"6e9e4f69-35d9-40e7-b4a9-65f9fa847512\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a70cb1a-764b99f6f4-f9scl" Sep 10 15:59:07.441853 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:07.441832 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e9e4f69-35d9-40e7-b4a9-65f9fa847512-maas-api-tls\") pod \"maas-api-e2e-mt-9a70cb1a-764b99f6f4-f9scl\" (UID: \"6e9e4f69-35d9-40e7-b4a9-65f9fa847512\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a70cb1a-764b99f6f4-f9scl" Sep 10 15:59:07.442078 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:59:07.442056 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-9a70cb1a-serving-cert: secret "maas-api-e2e-mt-9a70cb1a-serving-cert" not found Sep 10 15:59:07.442193 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:59:07.442138 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e9e4f69-35d9-40e7-b4a9-65f9fa847512-maas-api-tls podName:6e9e4f69-35d9-40e7-b4a9-65f9fa847512 nodeName:}" failed. No retries permitted until 2026-09-10 15:59:07.942113855 +0000 UTC m=+1024.917373791 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6e9e4f69-35d9-40e7-b4a9-65f9fa847512-maas-api-tls") pod "maas-api-e2e-mt-9a70cb1a-764b99f6f4-f9scl" (UID: "6e9e4f69-35d9-40e7-b4a9-65f9fa847512") : secret "maas-api-e2e-mt-9a70cb1a-serving-cert" not found Sep 10 15:59:07.442525 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:07.442500 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e9e4f69-35d9-40e7-b4a9-65f9fa847512-openshift-service-ca\") pod \"maas-api-e2e-mt-9a70cb1a-764b99f6f4-f9scl\" (UID: \"6e9e4f69-35d9-40e7-b4a9-65f9fa847512\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a70cb1a-764b99f6f4-f9scl" Sep 10 15:59:07.444486 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:07.444466 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e9e4f69-35d9-40e7-b4a9-65f9fa847512-metrics-certs\") pod \"maas-api-e2e-mt-9a70cb1a-764b99f6f4-f9scl\" (UID: \"6e9e4f69-35d9-40e7-b4a9-65f9fa847512\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a70cb1a-764b99f6f4-f9scl" Sep 10 15:59:07.451870 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:07.451848 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8ljj\" (UniqueName: \"kubernetes.io/projected/6e9e4f69-35d9-40e7-b4a9-65f9fa847512-kube-api-access-h8ljj\") pod \"maas-api-e2e-mt-9a70cb1a-764b99f6f4-f9scl\" (UID: \"6e9e4f69-35d9-40e7-b4a9-65f9fa847512\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a70cb1a-764b99f6f4-f9scl" Sep 10 15:59:07.465605 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:07.465581 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:07.732887 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:07.732853 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:07.946722 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:07.946695 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e9e4f69-35d9-40e7-b4a9-65f9fa847512-maas-api-tls\") pod \"maas-api-e2e-mt-9a70cb1a-764b99f6f4-f9scl\" (UID: \"6e9e4f69-35d9-40e7-b4a9-65f9fa847512\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a70cb1a-764b99f6f4-f9scl" Sep 10 15:59:07.948957 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:07.948931 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e9e4f69-35d9-40e7-b4a9-65f9fa847512-maas-api-tls\") pod \"maas-api-e2e-mt-9a70cb1a-764b99f6f4-f9scl\" (UID: \"6e9e4f69-35d9-40e7-b4a9-65f9fa847512\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a70cb1a-764b99f6f4-f9scl" Sep 10 15:59:08.000680 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:08.000609 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:08.098504 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:08.098474 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:08.236605 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:08.235934 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a70cb1a-764b99f6f4-f9scl" Sep 10 15:59:08.519634 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:08.519612 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9a70cb1a-764b99f6f4-f9scl"] Sep 10 15:59:08.520563 ip-10-0-139-107 kubenswrapper[2583]: W0910 15:59:08.520538 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e9e4f69_35d9_40e7_b4a9_65f9fa847512.slice/crio-465cf07c3f5d4c2ad764ca4df30689737ba5d1208e2a501fdf7c4fe6e7f90818 WatchSource:0}: Error finding container 465cf07c3f5d4c2ad764ca4df30689737ba5d1208e2a501fdf7c4fe6e7f90818: Status 404 returned error can't find the container with id 465cf07c3f5d4c2ad764ca4df30689737ba5d1208e2a501fdf7c4fe6e7f90818 Sep 10 15:59:08.521886 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:08.521869 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 15:59:09.003177 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:09.003140 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a70cb1a-764b99f6f4-f9scl" event={"ID":"6e9e4f69-35d9-40e7-b4a9-65f9fa847512","Type":"ContainerStarted","Data":"465cf07c3f5d4c2ad764ca4df30689737ba5d1208e2a501fdf7c4fe6e7f90818"} Sep 10 15:59:10.010397 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:10.010352 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a70cb1a-764b99f6f4-f9scl" event={"ID":"6e9e4f69-35d9-40e7-b4a9-65f9fa847512","Type":"ContainerStarted","Data":"a0ca44c2f7777f9ec0393528f809fb84625c618248af083c5ed5a06168b02c44"} Sep 10 15:59:10.010834 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:10.010474 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a70cb1a-764b99f6f4-f9scl" Sep 10 15:59:16.021808 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:16.021780 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a70cb1a-764b99f6f4-f9scl" Sep 10 15:59:16.042999 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:16.042939 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a70cb1a-764b99f6f4-f9scl" podStartSLOduration=8.448351291 podStartE2EDuration="9.042921149s" totalImagesPullingTime="594.569858ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:59:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:59:10.048119291 +0000 UTC m=+1027.023379243" watchObservedRunningTime="2026-09-10 15:59:16.042921149 +0000 UTC m=+1033.018181103" Sep 10 15:59:22.477234 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:22.477199 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:26.493243 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:26.493215 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:29.127490 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:29.127458 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:31.174230 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:31.174195 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:31.274948 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:31.274913 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:31.883214 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:31.883180 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:32.033049 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:32.033013 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:32.313658 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:32.313583 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:38.809205 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:38.809173 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:38.936977 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:38.936943 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:41.659242 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:41.659211 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:41.758927 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:41.758895 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:41.829898 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:41.829872 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:42.585687 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:42.585654 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:42.919651 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:42.919616 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:43.778748 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:43.778716 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:43.873128 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:43.873067 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x"] Sep 10 15:59:43.873370 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:43.873318 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x" podUID="141cb709-5da4-4878-b1ce-733402bae3aa" containerName="istio-proxy" containerID="cri-o://62edf9521f03425d98c44c1f486cccf786e479aabde54bd66a0d81272ca0d75e" gracePeriod=30 Sep 10 15:59:43.904169 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:43.904140 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:44.128671 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:44.128646 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:44.251032 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:44.250994 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:49.130719 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:49.130661 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x" Sep 10 15:59:49.234270 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:49.234238 2583 generic.go:356] "Generic (PLEG): container finished" podID="141cb709-5da4-4878-b1ce-733402bae3aa" containerID="62edf9521f03425d98c44c1f486cccf786e479aabde54bd66a0d81272ca0d75e" exitCode=0 Sep 10 15:59:49.234425 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:49.234282 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x" Sep 10 15:59:49.234425 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:49.234324 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x" event={"ID":"141cb709-5da4-4878-b1ce-733402bae3aa","Type":"ContainerDied","Data":"62edf9521f03425d98c44c1f486cccf786e479aabde54bd66a0d81272ca0d75e"} Sep 10 15:59:49.234425 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:49.234393 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x" event={"ID":"141cb709-5da4-4878-b1ce-733402bae3aa","Type":"ContainerDied","Data":"0a1ae2730c937669f83af367d14cbbd69495410702f1239fd0f52cb465b62ed7"} Sep 10 15:59:49.234425 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:49.234399 2583 scope.go:117] "RemoveContainer" containerID="62edf9521f03425d98c44c1f486cccf786e479aabde54bd66a0d81272ca0d75e" Sep 10 15:59:49.243525 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:49.243506 2583 scope.go:117] "RemoveContainer" containerID="62edf9521f03425d98c44c1f486cccf786e479aabde54bd66a0d81272ca0d75e" Sep 10 15:59:49.243795 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:59:49.243779 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62edf9521f03425d98c44c1f486cccf786e479aabde54bd66a0d81272ca0d75e\": container with ID starting with 62edf9521f03425d98c44c1f486cccf786e479aabde54bd66a0d81272ca0d75e not found: ID does not exist" containerID="62edf9521f03425d98c44c1f486cccf786e479aabde54bd66a0d81272ca0d75e" Sep 10 15:59:49.243834 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:49.243804 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62edf9521f03425d98c44c1f486cccf786e479aabde54bd66a0d81272ca0d75e"} err="failed to get container status \"62edf9521f03425d98c44c1f486cccf786e479aabde54bd66a0d81272ca0d75e\": rpc error: code = NotFound desc = could not find container \"62edf9521f03425d98c44c1f486cccf786e479aabde54bd66a0d81272ca0d75e\": container with ID starting with 62edf9521f03425d98c44c1f486cccf786e479aabde54bd66a0d81272ca0d75e not found: ID does not exist" Sep 10 15:59:49.319301 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:49.319277 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/141cb709-5da4-4878-b1ce-733402bae3aa-istio-envoy\") pod \"141cb709-5da4-4878-b1ce-733402bae3aa\" (UID: \"141cb709-5da4-4878-b1ce-733402bae3aa\") " Sep 10 15:59:49.319459 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:49.319315 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/141cb709-5da4-4878-b1ce-733402bae3aa-workload-certs\") pod \"141cb709-5da4-4878-b1ce-733402bae3aa\" (UID: \"141cb709-5da4-4878-b1ce-733402bae3aa\") " Sep 10 15:59:49.319459 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:49.319346 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/141cb709-5da4-4878-b1ce-733402bae3aa-workload-socket\") pod \"141cb709-5da4-4878-b1ce-733402bae3aa\" (UID: \"141cb709-5da4-4878-b1ce-733402bae3aa\") " Sep 10 15:59:49.319459 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:49.319415 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/141cb709-5da4-4878-b1ce-733402bae3aa-istiod-ca-cert\") pod \"141cb709-5da4-4878-b1ce-733402bae3aa\" (UID: \"141cb709-5da4-4878-b1ce-733402bae3aa\") " Sep 10 15:59:49.319459 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:49.319440 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/141cb709-5da4-4878-b1ce-733402bae3aa-credential-socket\") pod \"141cb709-5da4-4878-b1ce-733402bae3aa\" (UID: \"141cb709-5da4-4878-b1ce-733402bae3aa\") " Sep 10 15:59:49.319685 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:49.319484 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwpcn\" (UniqueName: \"kubernetes.io/projected/141cb709-5da4-4878-b1ce-733402bae3aa-kube-api-access-wwpcn\") pod \"141cb709-5da4-4878-b1ce-733402bae3aa\" (UID: \"141cb709-5da4-4878-b1ce-733402bae3aa\") " Sep 10 15:59:49.319685 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:49.319562 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/141cb709-5da4-4878-b1ce-733402bae3aa-istio-data\") pod \"141cb709-5da4-4878-b1ce-733402bae3aa\" (UID: \"141cb709-5da4-4878-b1ce-733402bae3aa\") " Sep 10 15:59:49.319685 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:49.319557 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/141cb709-5da4-4878-b1ce-733402bae3aa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "141cb709-5da4-4878-b1ce-733402bae3aa" (UID: "141cb709-5da4-4878-b1ce-733402bae3aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:59:49.319685 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:49.319604 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/141cb709-5da4-4878-b1ce-733402bae3aa-istio-token\") pod \"141cb709-5da4-4878-b1ce-733402bae3aa\" (UID: \"141cb709-5da4-4878-b1ce-733402bae3aa\") " Sep 10 15:59:49.319874 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:49.319704 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/141cb709-5da4-4878-b1ce-733402bae3aa-istio-podinfo\") pod \"141cb709-5da4-4878-b1ce-733402bae3aa\" (UID: \"141cb709-5da4-4878-b1ce-733402bae3aa\") " Sep 10 15:59:49.320098 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:49.320051 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/141cb709-5da4-4878-b1ce-733402bae3aa-workload-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:59:49.320098 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:49.320067 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/141cb709-5da4-4878-b1ce-733402bae3aa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "141cb709-5da4-4878-b1ce-733402bae3aa" (UID: "141cb709-5da4-4878-b1ce-733402bae3aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:59:49.320208 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:49.320144 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/141cb709-5da4-4878-b1ce-733402bae3aa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "141cb709-5da4-4878-b1ce-733402bae3aa" (UID: "141cb709-5da4-4878-b1ce-733402bae3aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:59:49.320208 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:49.320195 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/141cb709-5da4-4878-b1ce-733402bae3aa-istio-data" (OuterVolumeSpecName: "istio-data") pod "141cb709-5da4-4878-b1ce-733402bae3aa" (UID: "141cb709-5da4-4878-b1ce-733402bae3aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:59:49.320290 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:49.320223 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/141cb709-5da4-4878-b1ce-733402bae3aa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "141cb709-5da4-4878-b1ce-733402bae3aa" (UID: "141cb709-5da4-4878-b1ce-733402bae3aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:59:49.321636 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:49.321605 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/141cb709-5da4-4878-b1ce-733402bae3aa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "141cb709-5da4-4878-b1ce-733402bae3aa" (UID: "141cb709-5da4-4878-b1ce-733402bae3aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 15:59:49.322067 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:49.322043 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/141cb709-5da4-4878-b1ce-733402bae3aa-kube-api-access-wwpcn" (OuterVolumeSpecName: "kube-api-access-wwpcn") pod "141cb709-5da4-4878-b1ce-733402bae3aa" (UID: "141cb709-5da4-4878-b1ce-733402bae3aa"). InnerVolumeSpecName "kube-api-access-wwpcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:59:49.322188 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:49.322101 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/141cb709-5da4-4878-b1ce-733402bae3aa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "141cb709-5da4-4878-b1ce-733402bae3aa" (UID: "141cb709-5da4-4878-b1ce-733402bae3aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 15:59:49.322649 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:49.322625 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/141cb709-5da4-4878-b1ce-733402bae3aa-istio-token" (OuterVolumeSpecName: "istio-token") pod "141cb709-5da4-4878-b1ce-733402bae3aa" (UID: "141cb709-5da4-4878-b1ce-733402bae3aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:59:49.421446 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:49.421351 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/141cb709-5da4-4878-b1ce-733402bae3aa-istio-envoy\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:59:49.421446 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:49.421393 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/141cb709-5da4-4878-b1ce-733402bae3aa-workload-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:59:49.421446 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:49.421404 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/141cb709-5da4-4878-b1ce-733402bae3aa-istiod-ca-cert\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:59:49.421446 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:49.421412 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/141cb709-5da4-4878-b1ce-733402bae3aa-credential-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:59:49.421446 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:49.421421 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwpcn\" (UniqueName: \"kubernetes.io/projected/141cb709-5da4-4878-b1ce-733402bae3aa-kube-api-access-wwpcn\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:59:49.421446 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:49.421430 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/141cb709-5da4-4878-b1ce-733402bae3aa-istio-data\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:59:49.421446 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:49.421438 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/141cb709-5da4-4878-b1ce-733402bae3aa-istio-token\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:59:49.421446 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:49.421445 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/141cb709-5da4-4878-b1ce-733402bae3aa-istio-podinfo\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:59:49.427391 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:49.427354 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:49.531263 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:49.531234 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:49.556337 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:49.556309 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x"] Sep 10 15:59:49.566063 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:49.566036 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-17d9a5-openshift-default-7c486f8f86-cqx6x"] Sep 10 15:59:50.138212 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:50.138181 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9a70cb1a-764b99f6f4-f9scl"] Sep 10 15:59:50.138680 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:50.138480 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a70cb1a-764b99f6f4-f9scl" podUID="6e9e4f69-35d9-40e7-b4a9-65f9fa847512" containerName="maas-api" containerID="cri-o://a0ca44c2f7777f9ec0393528f809fb84625c618248af083c5ed5a06168b02c44" gracePeriod=30 Sep 10 15:59:50.178483 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:50.178459 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:50.279220 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:50.279190 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:50.380508 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:50.380480 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:50.387742 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:50.387711 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a70cb1a-764b99f6f4-f9scl" Sep 10 15:59:50.432489 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:50.432426 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e9e4f69-35d9-40e7-b4a9-65f9fa847512-openshift-service-ca\") pod \"6e9e4f69-35d9-40e7-b4a9-65f9fa847512\" (UID: \"6e9e4f69-35d9-40e7-b4a9-65f9fa847512\") " Sep 10 15:59:50.432489 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:50.432478 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e9e4f69-35d9-40e7-b4a9-65f9fa847512-metrics-certs\") pod \"6e9e4f69-35d9-40e7-b4a9-65f9fa847512\" (UID: \"6e9e4f69-35d9-40e7-b4a9-65f9fa847512\") " Sep 10 15:59:50.432686 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:50.432525 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8ljj\" (UniqueName: \"kubernetes.io/projected/6e9e4f69-35d9-40e7-b4a9-65f9fa847512-kube-api-access-h8ljj\") pod \"6e9e4f69-35d9-40e7-b4a9-65f9fa847512\" (UID: \"6e9e4f69-35d9-40e7-b4a9-65f9fa847512\") " Sep 10 15:59:50.432686 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:50.432564 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e9e4f69-35d9-40e7-b4a9-65f9fa847512-maas-api-tls\") pod \"6e9e4f69-35d9-40e7-b4a9-65f9fa847512\" (UID: \"6e9e4f69-35d9-40e7-b4a9-65f9fa847512\") " Sep 10 15:59:50.432843 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:50.432820 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e9e4f69-35d9-40e7-b4a9-65f9fa847512-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6e9e4f69-35d9-40e7-b4a9-65f9fa847512" (UID: "6e9e4f69-35d9-40e7-b4a9-65f9fa847512"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:59:50.434619 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:50.434580 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e9e4f69-35d9-40e7-b4a9-65f9fa847512-kube-api-access-h8ljj" (OuterVolumeSpecName: "kube-api-access-h8ljj") pod "6e9e4f69-35d9-40e7-b4a9-65f9fa847512" (UID: "6e9e4f69-35d9-40e7-b4a9-65f9fa847512"). InnerVolumeSpecName "kube-api-access-h8ljj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:59:50.434725 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:50.434634 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e9e4f69-35d9-40e7-b4a9-65f9fa847512-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6e9e4f69-35d9-40e7-b4a9-65f9fa847512" (UID: "6e9e4f69-35d9-40e7-b4a9-65f9fa847512"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:59:50.434725 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:50.434672 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e9e4f69-35d9-40e7-b4a9-65f9fa847512-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6e9e4f69-35d9-40e7-b4a9-65f9fa847512" (UID: "6e9e4f69-35d9-40e7-b4a9-65f9fa847512"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:59:50.486160 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:50.486131 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:50.534318 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:50.534282 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e9e4f69-35d9-40e7-b4a9-65f9fa847512-openshift-service-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:59:50.534318 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:50.534321 2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e9e4f69-35d9-40e7-b4a9-65f9fa847512-metrics-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:59:50.534557 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:50.534334 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8ljj\" (UniqueName: \"kubernetes.io/projected/6e9e4f69-35d9-40e7-b4a9-65f9fa847512-kube-api-access-h8ljj\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:59:50.534557 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:50.534346 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e9e4f69-35d9-40e7-b4a9-65f9fa847512-maas-api-tls\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:59:50.831237 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:50.831130 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:51.251674 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:51.251645 2583 generic.go:356] "Generic (PLEG): container finished" podID="6e9e4f69-35d9-40e7-b4a9-65f9fa847512" containerID="a0ca44c2f7777f9ec0393528f809fb84625c618248af083c5ed5a06168b02c44" exitCode=0 Sep 10 15:59:51.252038 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:51.251691 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a70cb1a-764b99f6f4-f9scl" event={"ID":"6e9e4f69-35d9-40e7-b4a9-65f9fa847512","Type":"ContainerDied","Data":"a0ca44c2f7777f9ec0393528f809fb84625c618248af083c5ed5a06168b02c44"} Sep 10 15:59:51.252038 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:51.251729 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a70cb1a-764b99f6f4-f9scl" event={"ID":"6e9e4f69-35d9-40e7-b4a9-65f9fa847512","Type":"ContainerDied","Data":"465cf07c3f5d4c2ad764ca4df30689737ba5d1208e2a501fdf7c4fe6e7f90818"} Sep 10 15:59:51.252038 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:51.251751 2583 scope.go:117] "RemoveContainer" containerID="a0ca44c2f7777f9ec0393528f809fb84625c618248af083c5ed5a06168b02c44" Sep 10 15:59:51.252038 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:51.251694 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9a70cb1a-764b99f6f4-f9scl" Sep 10 15:59:51.261980 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:51.261963 2583 scope.go:117] "RemoveContainer" containerID="a0ca44c2f7777f9ec0393528f809fb84625c618248af083c5ed5a06168b02c44" Sep 10 15:59:51.262272 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:59:51.262254 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0ca44c2f7777f9ec0393528f809fb84625c618248af083c5ed5a06168b02c44\": container with ID starting with a0ca44c2f7777f9ec0393528f809fb84625c618248af083c5ed5a06168b02c44 not found: ID does not exist" containerID="a0ca44c2f7777f9ec0393528f809fb84625c618248af083c5ed5a06168b02c44" Sep 10 15:59:51.262327 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:51.262281 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0ca44c2f7777f9ec0393528f809fb84625c618248af083c5ed5a06168b02c44"} err="failed to get container status \"a0ca44c2f7777f9ec0393528f809fb84625c618248af083c5ed5a06168b02c44\": rpc error: code = NotFound desc = could not find container \"a0ca44c2f7777f9ec0393528f809fb84625c618248af083c5ed5a06168b02c44\": container with ID starting with a0ca44c2f7777f9ec0393528f809fb84625c618248af083c5ed5a06168b02c44 not found: ID does not exist" Sep 10 15:59:51.275195 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:51.275170 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9a70cb1a-764b99f6f4-f9scl"] Sep 10 15:59:51.280149 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:51.280126 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9a70cb1a-764b99f6f4-f9scl"] Sep 10 15:59:51.516786 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:51.516708 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="141cb709-5da4-4878-b1ce-733402bae3aa" path="/var/lib/kubelet/pods/141cb709-5da4-4878-b1ce-733402bae3aa/volumes" Sep 10 15:59:51.517174 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:51.517159 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e9e4f69-35d9-40e7-b4a9-65f9fa847512" path="/var/lib/kubelet/pods/6e9e4f69-35d9-40e7-b4a9-65f9fa847512/volumes" Sep 10 15:59:52.839820 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:52.839790 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:53.556753 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:53.556719 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-83dcaf-9b598689c-gmltw"] Sep 10 15:59:53.556987 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:53.556947 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83dcaf-9b598689c-gmltw" podUID="beee0095-f836-4de6-990e-8e4515e17a74" containerName="maas-api" containerID="cri-o://965a1350d417fd7faeb8b3f907b9c8662d7cec99043f31bda84ab9e719c2cff2" gracePeriod=30 Sep 10 15:59:53.588115 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:53.588081 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:53.630836 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:53.630804 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-83dcaf-698b75cf54-jz542"] Sep 10 15:59:53.631024 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:53.631000 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-83dcaf-698b75cf54-jz542" podUID="2d82f2a2-f582-431a-b837-17ffdf28c815" containerName="payload-pre-processing" containerID="cri-o://b354bb49b803898a0fed9d9bc9f98e2d9de05a84d361bf6a64d776a40ef11e64" gracePeriod=30 Sep 10 15:59:53.688247 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:53.688216 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:53.789490 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:53.789445 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:53.888981 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:53.888955 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:53.894108 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:53.894086 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-83dcaf-698b75cf54-jz542" Sep 10 15:59:53.964927 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:53.964896 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d82f2a2-f582-431a-b837-17ffdf28c815-plugins-config-volume\") pod \"2d82f2a2-f582-431a-b837-17ffdf28c815\" (UID: \"2d82f2a2-f582-431a-b837-17ffdf28c815\") " Sep 10 15:59:53.965070 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:53.964944 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rwrlr\" (UniqueName: \"kubernetes.io/projected/2d82f2a2-f582-431a-b837-17ffdf28c815-kube-api-access-rwrlr\") pod \"2d82f2a2-f582-431a-b837-17ffdf28c815\" (UID: \"2d82f2a2-f582-431a-b837-17ffdf28c815\") " Sep 10 15:59:53.965241 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:53.965219 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d82f2a2-f582-431a-b837-17ffdf28c815-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2d82f2a2-f582-431a-b837-17ffdf28c815" (UID: "2d82f2a2-f582-431a-b837-17ffdf28c815"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:59:53.967059 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:53.967035 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d82f2a2-f582-431a-b837-17ffdf28c815-kube-api-access-rwrlr" (OuterVolumeSpecName: "kube-api-access-rwrlr") pod "2d82f2a2-f582-431a-b837-17ffdf28c815" (UID: "2d82f2a2-f582-431a-b837-17ffdf28c815"). InnerVolumeSpecName "kube-api-access-rwrlr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:59:54.066345 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:54.066321 2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d82f2a2-f582-431a-b837-17ffdf28c815-plugins-config-volume\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:59:54.066345 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:54.066343 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwrlr\" (UniqueName: \"kubernetes.io/projected/2d82f2a2-f582-431a-b837-17ffdf28c815-kube-api-access-rwrlr\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:59:54.193054 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:54.193033 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:54.270238 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:54.270213 2583 generic.go:356] "Generic (PLEG): container finished" podID="2d82f2a2-f582-431a-b837-17ffdf28c815" containerID="b354bb49b803898a0fed9d9bc9f98e2d9de05a84d361bf6a64d776a40ef11e64" exitCode=0 Sep 10 15:59:54.270352 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:54.270260 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-83dcaf-698b75cf54-jz542" Sep 10 15:59:54.270352 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:54.270276 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-83dcaf-698b75cf54-jz542" event={"ID":"2d82f2a2-f582-431a-b837-17ffdf28c815","Type":"ContainerDied","Data":"b354bb49b803898a0fed9d9bc9f98e2d9de05a84d361bf6a64d776a40ef11e64"} Sep 10 15:59:54.270352 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:54.270323 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-83dcaf-698b75cf54-jz542" event={"ID":"2d82f2a2-f582-431a-b837-17ffdf28c815","Type":"ContainerDied","Data":"a5c97455fc9ea01ab6495cecab9fee05d6675f3ecd13524e53364ad8a5d9b855"} Sep 10 15:59:54.270352 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:54.270346 2583 scope.go:117] "RemoveContainer" containerID="b354bb49b803898a0fed9d9bc9f98e2d9de05a84d361bf6a64d776a40ef11e64" Sep 10 15:59:54.279990 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:54.279966 2583 scope.go:117] "RemoveContainer" containerID="b354bb49b803898a0fed9d9bc9f98e2d9de05a84d361bf6a64d776a40ef11e64" Sep 10 15:59:54.280258 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:59:54.280241 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b354bb49b803898a0fed9d9bc9f98e2d9de05a84d361bf6a64d776a40ef11e64\": container with ID starting with b354bb49b803898a0fed9d9bc9f98e2d9de05a84d361bf6a64d776a40ef11e64 not found: ID does not exist" containerID="b354bb49b803898a0fed9d9bc9f98e2d9de05a84d361bf6a64d776a40ef11e64" Sep 10 15:59:54.280312 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:54.280266 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b354bb49b803898a0fed9d9bc9f98e2d9de05a84d361bf6a64d776a40ef11e64"} err="failed to get container status \"b354bb49b803898a0fed9d9bc9f98e2d9de05a84d361bf6a64d776a40ef11e64\": rpc error: code = NotFound desc = could not find container \"b354bb49b803898a0fed9d9bc9f98e2d9de05a84d361bf6a64d776a40ef11e64\": container with ID starting with b354bb49b803898a0fed9d9bc9f98e2d9de05a84d361bf6a64d776a40ef11e64 not found: ID does not exist" Sep 10 15:59:54.295805 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:54.295784 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-83dcaf-698b75cf54-jz542"] Sep 10 15:59:54.300973 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:54.300952 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-83dcaf-698b75cf54-jz542"] Sep 10 15:59:54.798377 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:54.798355 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83dcaf-9b598689c-gmltw" Sep 10 15:59:54.872838 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:54.872804 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69jvv\" (UniqueName: \"kubernetes.io/projected/beee0095-f836-4de6-990e-8e4515e17a74-kube-api-access-69jvv\") pod \"beee0095-f836-4de6-990e-8e4515e17a74\" (UID: \"beee0095-f836-4de6-990e-8e4515e17a74\") " Sep 10 15:59:54.872975 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:54.872928 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/beee0095-f836-4de6-990e-8e4515e17a74-openshift-service-ca\") pod \"beee0095-f836-4de6-990e-8e4515e17a74\" (UID: \"beee0095-f836-4de6-990e-8e4515e17a74\") " Sep 10 15:59:54.872975 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:54.872971 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/beee0095-f836-4de6-990e-8e4515e17a74-metrics-certs\") pod \"beee0095-f836-4de6-990e-8e4515e17a74\" (UID: \"beee0095-f836-4de6-990e-8e4515e17a74\") " Sep 10 15:59:54.873076 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:54.872992 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/beee0095-f836-4de6-990e-8e4515e17a74-maas-api-tls\") pod \"beee0095-f836-4de6-990e-8e4515e17a74\" (UID: \"beee0095-f836-4de6-990e-8e4515e17a74\") " Sep 10 15:59:54.873290 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:54.873257 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beee0095-f836-4de6-990e-8e4515e17a74-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "beee0095-f836-4de6-990e-8e4515e17a74" (UID: "beee0095-f836-4de6-990e-8e4515e17a74"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 15:59:54.875022 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:54.874996 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beee0095-f836-4de6-990e-8e4515e17a74-kube-api-access-69jvv" (OuterVolumeSpecName: "kube-api-access-69jvv") pod "beee0095-f836-4de6-990e-8e4515e17a74" (UID: "beee0095-f836-4de6-990e-8e4515e17a74"). InnerVolumeSpecName "kube-api-access-69jvv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 15:59:54.875138 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:54.875081 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/beee0095-f836-4de6-990e-8e4515e17a74-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "beee0095-f836-4de6-990e-8e4515e17a74" (UID: "beee0095-f836-4de6-990e-8e4515e17a74"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:59:54.875138 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:54.875105 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/beee0095-f836-4de6-990e-8e4515e17a74-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "beee0095-f836-4de6-990e-8e4515e17a74" (UID: "beee0095-f836-4de6-990e-8e4515e17a74"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 15:59:54.974226 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:54.974157 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/beee0095-f836-4de6-990e-8e4515e17a74-openshift-service-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:59:54.974226 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:54.974185 2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/beee0095-f836-4de6-990e-8e4515e17a74-metrics-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:59:54.974226 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:54.974194 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/beee0095-f836-4de6-990e-8e4515e17a74-maas-api-tls\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:59:54.974226 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:54.974202 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69jvv\" (UniqueName: \"kubernetes.io/projected/beee0095-f836-4de6-990e-8e4515e17a74-kube-api-access-69jvv\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 15:59:55.278215 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:55.278123 2583 generic.go:356] "Generic (PLEG): container finished" podID="beee0095-f836-4de6-990e-8e4515e17a74" containerID="965a1350d417fd7faeb8b3f907b9c8662d7cec99043f31bda84ab9e719c2cff2" exitCode=0 Sep 10 15:59:55.278215 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:55.278169 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83dcaf-9b598689c-gmltw" Sep 10 15:59:55.278215 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:55.278205 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83dcaf-9b598689c-gmltw" event={"ID":"beee0095-f836-4de6-990e-8e4515e17a74","Type":"ContainerDied","Data":"965a1350d417fd7faeb8b3f907b9c8662d7cec99043f31bda84ab9e719c2cff2"} Sep 10 15:59:55.278461 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:55.278258 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-83dcaf-9b598689c-gmltw" event={"ID":"beee0095-f836-4de6-990e-8e4515e17a74","Type":"ContainerDied","Data":"07c14220ed09f41286220347baad779bf0e466e7af18b2e34041e05f124ed63c"} Sep 10 15:59:55.278461 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:55.278284 2583 scope.go:117] "RemoveContainer" containerID="965a1350d417fd7faeb8b3f907b9c8662d7cec99043f31bda84ab9e719c2cff2" Sep 10 15:59:55.288217 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:55.288201 2583 scope.go:117] "RemoveContainer" containerID="965a1350d417fd7faeb8b3f907b9c8662d7cec99043f31bda84ab9e719c2cff2" Sep 10 15:59:55.288511 ip-10-0-139-107 kubenswrapper[2583]: E0910 15:59:55.288488 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"965a1350d417fd7faeb8b3f907b9c8662d7cec99043f31bda84ab9e719c2cff2\": container with ID starting with 965a1350d417fd7faeb8b3f907b9c8662d7cec99043f31bda84ab9e719c2cff2 not found: ID does not exist" containerID="965a1350d417fd7faeb8b3f907b9c8662d7cec99043f31bda84ab9e719c2cff2" Sep 10 15:59:55.288573 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:55.288518 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"965a1350d417fd7faeb8b3f907b9c8662d7cec99043f31bda84ab9e719c2cff2"} err="failed to get container status \"965a1350d417fd7faeb8b3f907b9c8662d7cec99043f31bda84ab9e719c2cff2\": rpc error: code = NotFound desc = could not find container \"965a1350d417fd7faeb8b3f907b9c8662d7cec99043f31bda84ab9e719c2cff2\": container with ID starting with 965a1350d417fd7faeb8b3f907b9c8662d7cec99043f31bda84ab9e719c2cff2 not found: ID does not exist" Sep 10 15:59:55.307464 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:55.307443 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-83dcaf-9b598689c-gmltw"] Sep 10 15:59:55.309753 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:55.309735 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-83dcaf-9b598689c-gmltw"] Sep 10 15:59:55.515257 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:55.515224 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d82f2a2-f582-431a-b837-17ffdf28c815" path="/var/lib/kubelet/pods/2d82f2a2-f582-431a-b837-17ffdf28c815/volumes" Sep 10 15:59:55.515648 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:55.515635 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="beee0095-f836-4de6-990e-8e4515e17a74" path="/var/lib/kubelet/pods/beee0095-f836-4de6-990e-8e4515e17a74/volumes" Sep 10 15:59:57.379181 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:57.379050 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:57.483852 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:57.483823 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:57.674183 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:57.674086 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:58.030217 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:58.030115 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:58.132008 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:58.131963 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 15:59:58.261531 ip-10-0-139-107 kubenswrapper[2583]: I0910 15:59:58.261487 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:01.310199 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:01.310166 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:01.410833 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:01.410789 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:01.576980 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:01.576902 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:02.108066 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:02.108015 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:05.405088 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:05.405054 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:05.522790 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:05.522758 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:05.693489 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:05.693412 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:06.438667 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:06.438639 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:12.456284 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:12.456250 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-607b9a-57f97879d-x445g"] Sep 10 16:00:12.456715 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:12.456498 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-607b9a-57f97879d-x445g" podUID="24270400-b9ca-4faf-91b1-698df59b5a0a" containerName="maas-api" containerID="cri-o://34aad7dd2133a4c7fa8fb21942f6e45582e22b2ed2bdea79b870e34e67c030d5" gracePeriod=30 Sep 10 16:00:12.497481 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:12.497451 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:12.599603 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:12.599572 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:12.698638 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:12.698613 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-607b9a-57f97879d-x445g" Sep 10 16:00:12.705561 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:12.705539 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:12.801271 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:12.801203 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:12.832442 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:12.832412 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jgsd\" (UniqueName: \"kubernetes.io/projected/24270400-b9ca-4faf-91b1-698df59b5a0a-kube-api-access-7jgsd\") pod \"24270400-b9ca-4faf-91b1-698df59b5a0a\" (UID: \"24270400-b9ca-4faf-91b1-698df59b5a0a\") " Sep 10 16:00:12.832578 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:12.832459 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24270400-b9ca-4faf-91b1-698df59b5a0a-metrics-certs\") pod \"24270400-b9ca-4faf-91b1-698df59b5a0a\" (UID: \"24270400-b9ca-4faf-91b1-698df59b5a0a\") " Sep 10 16:00:12.832578 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:12.832562 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24270400-b9ca-4faf-91b1-698df59b5a0a-maas-api-tls\") pod \"24270400-b9ca-4faf-91b1-698df59b5a0a\" (UID: \"24270400-b9ca-4faf-91b1-698df59b5a0a\") " Sep 10 16:00:12.832676 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:12.832589 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24270400-b9ca-4faf-91b1-698df59b5a0a-openshift-service-ca\") pod \"24270400-b9ca-4faf-91b1-698df59b5a0a\" (UID: \"24270400-b9ca-4faf-91b1-698df59b5a0a\") " Sep 10 16:00:12.833026 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:12.832994 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24270400-b9ca-4faf-91b1-698df59b5a0a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "24270400-b9ca-4faf-91b1-698df59b5a0a" (UID: "24270400-b9ca-4faf-91b1-698df59b5a0a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:00:12.834671 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:12.834637 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24270400-b9ca-4faf-91b1-698df59b5a0a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "24270400-b9ca-4faf-91b1-698df59b5a0a" (UID: "24270400-b9ca-4faf-91b1-698df59b5a0a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:00:12.834761 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:12.834681 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24270400-b9ca-4faf-91b1-698df59b5a0a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "24270400-b9ca-4faf-91b1-698df59b5a0a" (UID: "24270400-b9ca-4faf-91b1-698df59b5a0a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:00:12.834761 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:12.834743 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24270400-b9ca-4faf-91b1-698df59b5a0a-kube-api-access-7jgsd" (OuterVolumeSpecName: "kube-api-access-7jgsd") pod "24270400-b9ca-4faf-91b1-698df59b5a0a" (UID: "24270400-b9ca-4faf-91b1-698df59b5a0a"). InnerVolumeSpecName "kube-api-access-7jgsd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:00:12.933587 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:12.933557 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24270400-b9ca-4faf-91b1-698df59b5a0a-maas-api-tls\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:00:12.933587 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:12.933586 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24270400-b9ca-4faf-91b1-698df59b5a0a-openshift-service-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:00:12.933786 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:12.933600 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jgsd\" (UniqueName: \"kubernetes.io/projected/24270400-b9ca-4faf-91b1-698df59b5a0a-kube-api-access-7jgsd\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:00:12.933786 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:12.933612 2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24270400-b9ca-4faf-91b1-698df59b5a0a-metrics-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:00:13.142302 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:13.142276 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:13.378967 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:13.378933 2583 generic.go:356] "Generic (PLEG): container finished" podID="24270400-b9ca-4faf-91b1-698df59b5a0a" containerID="34aad7dd2133a4c7fa8fb21942f6e45582e22b2ed2bdea79b870e34e67c030d5" exitCode=0 Sep 10 16:00:13.379139 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:13.378971 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-607b9a-57f97879d-x445g" Sep 10 16:00:13.379139 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:13.379011 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-607b9a-57f97879d-x445g" event={"ID":"24270400-b9ca-4faf-91b1-698df59b5a0a","Type":"ContainerDied","Data":"34aad7dd2133a4c7fa8fb21942f6e45582e22b2ed2bdea79b870e34e67c030d5"} Sep 10 16:00:13.379139 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:13.379063 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-607b9a-57f97879d-x445g" event={"ID":"24270400-b9ca-4faf-91b1-698df59b5a0a","Type":"ContainerDied","Data":"ae67ecddd2dc036443f2ea73d38986f0887d4f0d0ef53d5cb97dc807c3aa93a6"} Sep 10 16:00:13.379139 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:13.379086 2583 scope.go:117] "RemoveContainer" containerID="34aad7dd2133a4c7fa8fb21942f6e45582e22b2ed2bdea79b870e34e67c030d5" Sep 10 16:00:13.388416 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:13.388400 2583 scope.go:117] "RemoveContainer" containerID="34aad7dd2133a4c7fa8fb21942f6e45582e22b2ed2bdea79b870e34e67c030d5" Sep 10 16:00:13.388671 ip-10-0-139-107 kubenswrapper[2583]: E0910 16:00:13.388655 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34aad7dd2133a4c7fa8fb21942f6e45582e22b2ed2bdea79b870e34e67c030d5\": container with ID starting with 34aad7dd2133a4c7fa8fb21942f6e45582e22b2ed2bdea79b870e34e67c030d5 not found: ID does not exist" containerID="34aad7dd2133a4c7fa8fb21942f6e45582e22b2ed2bdea79b870e34e67c030d5" Sep 10 16:00:13.388718 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:13.388678 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34aad7dd2133a4c7fa8fb21942f6e45582e22b2ed2bdea79b870e34e67c030d5"} err="failed to get container status \"34aad7dd2133a4c7fa8fb21942f6e45582e22b2ed2bdea79b870e34e67c030d5\": rpc error: code = NotFound desc = could not find container \"34aad7dd2133a4c7fa8fb21942f6e45582e22b2ed2bdea79b870e34e67c030d5\": container with ID starting with 34aad7dd2133a4c7fa8fb21942f6e45582e22b2ed2bdea79b870e34e67c030d5 not found: ID does not exist" Sep 10 16:00:13.403026 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:13.402976 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-607b9a-57f97879d-x445g"] Sep 10 16:00:13.407297 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:13.407277 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-607b9a-57f97879d-x445g"] Sep 10 16:00:13.516177 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:13.516153 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24270400-b9ca-4faf-91b1-698df59b5a0a" path="/var/lib/kubelet/pods/24270400-b9ca-4faf-91b1-698df59b5a0a/volumes" Sep 10 16:00:13.586781 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:13.586753 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:13.899415 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:13.899372 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:14.919007 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:14.918959 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:16.645278 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:16.645244 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:17.541356 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:17.541322 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:17.637880 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:17.637844 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:17.768283 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:17.768246 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-247f70-5477449978-2rf7b"] Sep 10 16:00:17.768888 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:17.768871 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d82f2a2-f582-431a-b837-17ffdf28c815" containerName="payload-pre-processing" Sep 10 16:00:17.768933 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:17.768892 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d82f2a2-f582-431a-b837-17ffdf28c815" containerName="payload-pre-processing" Sep 10 16:00:17.768933 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:17.768909 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="beee0095-f836-4de6-990e-8e4515e17a74" containerName="maas-api" Sep 10 16:00:17.768933 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:17.768917 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="beee0095-f836-4de6-990e-8e4515e17a74" containerName="maas-api" Sep 10 16:00:17.769031 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:17.768932 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e9e4f69-35d9-40e7-b4a9-65f9fa847512" containerName="maas-api" Sep 10 16:00:17.769031 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:17.768941 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e9e4f69-35d9-40e7-b4a9-65f9fa847512" containerName="maas-api" Sep 10 16:00:17.769031 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:17.768962 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="141cb709-5da4-4878-b1ce-733402bae3aa" containerName="istio-proxy" Sep 10 16:00:17.769031 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:17.768971 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="141cb709-5da4-4878-b1ce-733402bae3aa" containerName="istio-proxy" Sep 10 16:00:17.769031 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:17.768989 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24270400-b9ca-4faf-91b1-698df59b5a0a" containerName="maas-api" Sep 10 16:00:17.769031 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:17.768997 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="24270400-b9ca-4faf-91b1-698df59b5a0a" containerName="maas-api" Sep 10 16:00:17.769217 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:17.769087 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="141cb709-5da4-4878-b1ce-733402bae3aa" containerName="istio-proxy" Sep 10 16:00:17.769217 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:17.769102 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e9e4f69-35d9-40e7-b4a9-65f9fa847512" containerName="maas-api" Sep 10 16:00:17.769217 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:17.769111 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="beee0095-f836-4de6-990e-8e4515e17a74" containerName="maas-api" Sep 10 16:00:17.769217 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:17.769119 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="24270400-b9ca-4faf-91b1-698df59b5a0a" containerName="maas-api" Sep 10 16:00:17.769217 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:17.769129 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d82f2a2-f582-431a-b837-17ffdf28c815" containerName="payload-pre-processing" Sep 10 16:00:17.774136 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:17.774112 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-247f70-5477449978-2rf7b" Sep 10 16:00:17.776241 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:17.776220 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-247f70-serving-cert\"" Sep 10 16:00:17.781184 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:17.781160 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-247f70-5477449978-2rf7b"] Sep 10 16:00:17.842316 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:17.842241 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:17.879926 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:17.879893 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc6c29de-8858-438f-b5b6-d9fe4b5be50d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-247f70-5477449978-2rf7b\" (UID: \"dc6c29de-8858-438f-b5b6-d9fe4b5be50d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-247f70-5477449978-2rf7b" Sep 10 16:00:17.880173 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:17.880155 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6c29de-8858-438f-b5b6-d9fe4b5be50d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-247f70-5477449978-2rf7b\" (UID: \"dc6c29de-8858-438f-b5b6-d9fe4b5be50d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-247f70-5477449978-2rf7b" Sep 10 16:00:17.880426 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:17.880407 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n2wn\" (UniqueName: \"kubernetes.io/projected/dc6c29de-8858-438f-b5b6-d9fe4b5be50d-kube-api-access-6n2wn\") pod \"maas-api-e2e-shared-a-w2-247f70-5477449978-2rf7b\" (UID: \"dc6c29de-8858-438f-b5b6-d9fe4b5be50d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-247f70-5477449978-2rf7b" Sep 10 16:00:17.880616 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:17.880598 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc6c29de-8858-438f-b5b6-d9fe4b5be50d-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-247f70-5477449978-2rf7b\" (UID: \"dc6c29de-8858-438f-b5b6-d9fe4b5be50d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-247f70-5477449978-2rf7b" Sep 10 16:00:17.943426 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:17.943397 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:17.981566 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:17.981536 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n2wn\" (UniqueName: \"kubernetes.io/projected/dc6c29de-8858-438f-b5b6-d9fe4b5be50d-kube-api-access-6n2wn\") pod \"maas-api-e2e-shared-a-w2-247f70-5477449978-2rf7b\" (UID: \"dc6c29de-8858-438f-b5b6-d9fe4b5be50d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-247f70-5477449978-2rf7b" Sep 10 16:00:17.981686 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:17.981586 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc6c29de-8858-438f-b5b6-d9fe4b5be50d-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-247f70-5477449978-2rf7b\" (UID: \"dc6c29de-8858-438f-b5b6-d9fe4b5be50d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-247f70-5477449978-2rf7b" Sep 10 16:00:17.981686 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:17.981626 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc6c29de-8858-438f-b5b6-d9fe4b5be50d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-247f70-5477449978-2rf7b\" (UID: \"dc6c29de-8858-438f-b5b6-d9fe4b5be50d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-247f70-5477449978-2rf7b" Sep 10 16:00:17.981686 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:17.981653 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6c29de-8858-438f-b5b6-d9fe4b5be50d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-247f70-5477449978-2rf7b\" (UID: \"dc6c29de-8858-438f-b5b6-d9fe4b5be50d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-247f70-5477449978-2rf7b" Sep 10 16:00:17.981821 ip-10-0-139-107 kubenswrapper[2583]: E0910 16:00:17.981779 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-247f70-serving-cert: secret "maas-api-e2e-shared-a-w2-247f70-serving-cert" not found Sep 10 16:00:17.981875 ip-10-0-139-107 kubenswrapper[2583]: E0910 16:00:17.981850 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc6c29de-8858-438f-b5b6-d9fe4b5be50d-maas-api-tls podName:dc6c29de-8858-438f-b5b6-d9fe4b5be50d nodeName:}" failed. No retries permitted until 2026-09-10 16:00:18.481830703 +0000 UTC m=+1095.457090647 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dc6c29de-8858-438f-b5b6-d9fe4b5be50d-maas-api-tls") pod "maas-api-e2e-shared-a-w2-247f70-5477449978-2rf7b" (UID: "dc6c29de-8858-438f-b5b6-d9fe4b5be50d") : secret "maas-api-e2e-shared-a-w2-247f70-serving-cert" not found Sep 10 16:00:17.982324 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:17.982307 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6c29de-8858-438f-b5b6-d9fe4b5be50d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-247f70-5477449978-2rf7b\" (UID: \"dc6c29de-8858-438f-b5b6-d9fe4b5be50d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-247f70-5477449978-2rf7b" Sep 10 16:00:17.984100 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:17.984080 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc6c29de-8858-438f-b5b6-d9fe4b5be50d-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-247f70-5477449978-2rf7b\" (UID: \"dc6c29de-8858-438f-b5b6-d9fe4b5be50d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-247f70-5477449978-2rf7b" Sep 10 16:00:17.991317 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:17.991282 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n2wn\" (UniqueName: \"kubernetes.io/projected/dc6c29de-8858-438f-b5b6-d9fe4b5be50d-kube-api-access-6n2wn\") pod \"maas-api-e2e-shared-a-w2-247f70-5477449978-2rf7b\" (UID: \"dc6c29de-8858-438f-b5b6-d9fe4b5be50d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-247f70-5477449978-2rf7b" Sep 10 16:00:18.305756 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:18.305719 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:18.486798 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:18.486768 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc6c29de-8858-438f-b5b6-d9fe4b5be50d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-247f70-5477449978-2rf7b\" (UID: \"dc6c29de-8858-438f-b5b6-d9fe4b5be50d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-247f70-5477449978-2rf7b" Sep 10 16:00:18.489283 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:18.489250 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc6c29de-8858-438f-b5b6-d9fe4b5be50d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-247f70-5477449978-2rf7b\" (UID: \"dc6c29de-8858-438f-b5b6-d9fe4b5be50d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-247f70-5477449978-2rf7b" Sep 10 16:00:18.601090 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:18.601009 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-247f70-59f96f4bd9-crq27"] Sep 10 16:00:18.604848 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:18.604827 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-247f70-59f96f4bd9-crq27" Sep 10 16:00:18.607098 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:18.607077 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-247f70-dockercfg-hr6tq\"" Sep 10 16:00:18.607188 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:18.607081 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-247f70\"" Sep 10 16:00:18.617272 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:18.617247 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-247f70-59f96f4bd9-crq27"] Sep 10 16:00:18.686235 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:18.686196 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-247f70-5477449978-2rf7b" Sep 10 16:00:18.689405 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:18.689119 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnv69\" (UniqueName: \"kubernetes.io/projected/f016ad21-341e-4070-9608-6410eda330e2-kube-api-access-jnv69\") pod \"payload-processing-e2e-shared-a-w2-247f70-59f96f4bd9-crq27\" (UID: \"f016ad21-341e-4070-9608-6410eda330e2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-247f70-59f96f4bd9-crq27" Sep 10 16:00:18.689405 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:18.689210 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f016ad21-341e-4070-9608-6410eda330e2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-247f70-59f96f4bd9-crq27\" (UID: \"f016ad21-341e-4070-9608-6410eda330e2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-247f70-59f96f4bd9-crq27" Sep 10 16:00:18.790657 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:18.790625 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnv69\" (UniqueName: \"kubernetes.io/projected/f016ad21-341e-4070-9608-6410eda330e2-kube-api-access-jnv69\") pod \"payload-processing-e2e-shared-a-w2-247f70-59f96f4bd9-crq27\" (UID: \"f016ad21-341e-4070-9608-6410eda330e2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-247f70-59f96f4bd9-crq27" Sep 10 16:00:18.790989 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:18.790709 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f016ad21-341e-4070-9608-6410eda330e2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-247f70-59f96f4bd9-crq27\" (UID: \"f016ad21-341e-4070-9608-6410eda330e2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-247f70-59f96f4bd9-crq27" Sep 10 16:00:18.791407 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:18.791373 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f016ad21-341e-4070-9608-6410eda330e2-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-247f70-59f96f4bd9-crq27\" (UID: \"f016ad21-341e-4070-9608-6410eda330e2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-247f70-59f96f4bd9-crq27" Sep 10 16:00:18.798925 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:18.798904 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnv69\" (UniqueName: \"kubernetes.io/projected/f016ad21-341e-4070-9608-6410eda330e2-kube-api-access-jnv69\") pod \"payload-processing-e2e-shared-a-w2-247f70-59f96f4bd9-crq27\" (UID: \"f016ad21-341e-4070-9608-6410eda330e2\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-247f70-59f96f4bd9-crq27" Sep 10 16:00:18.916828 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:18.916804 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-247f70-59f96f4bd9-crq27" Sep 10 16:00:18.969867 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:18.969817 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-247f70-5477449978-2rf7b"] Sep 10 16:00:18.971296 ip-10-0-139-107 kubenswrapper[2583]: W0910 16:00:18.971270 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc6c29de_8858_438f_b5b6_d9fe4b5be50d.slice/crio-26ab4458f858fb6a6f4c7bf0cf7adb26433039b85b82707df166ac08092b840b WatchSource:0}: Error finding container 26ab4458f858fb6a6f4c7bf0cf7adb26433039b85b82707df166ac08092b840b: Status 404 returned error can't find the container with id 26ab4458f858fb6a6f4c7bf0cf7adb26433039b85b82707df166ac08092b840b Sep 10 16:00:19.194525 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:19.194497 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-247f70-59f96f4bd9-crq27"] Sep 10 16:00:19.196508 ip-10-0-139-107 kubenswrapper[2583]: W0910 16:00:19.196479 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf016ad21_341e_4070_9608_6410eda330e2.slice/crio-da98455865a71d592d353d1e1606e6dd1d3cc4601328169aaee4671df131ffc5 WatchSource:0}: Error finding container da98455865a71d592d353d1e1606e6dd1d3cc4601328169aaee4671df131ffc5: Status 404 returned error can't find the container with id da98455865a71d592d353d1e1606e6dd1d3cc4601328169aaee4671df131ffc5 Sep 10 16:00:19.222971 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:19.222948 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:19.414953 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:19.414915 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-247f70-59f96f4bd9-crq27" event={"ID":"f016ad21-341e-4070-9608-6410eda330e2","Type":"ContainerStarted","Data":"f6ecc4d1e40f6d6dbfdd6626976b1c0161617d368c7e9d1321766ad7dff17480"} Sep 10 16:00:19.415159 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:19.414978 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-247f70-59f96f4bd9-crq27" event={"ID":"f016ad21-341e-4070-9608-6410eda330e2","Type":"ContainerStarted","Data":"da98455865a71d592d353d1e1606e6dd1d3cc4601328169aaee4671df131ffc5"} Sep 10 16:00:19.415159 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:19.415028 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-247f70-59f96f4bd9-crq27" Sep 10 16:00:19.416267 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:19.416243 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-247f70-5477449978-2rf7b" event={"ID":"dc6c29de-8858-438f-b5b6-d9fe4b5be50d","Type":"ContainerStarted","Data":"26ab4458f858fb6a6f4c7bf0cf7adb26433039b85b82707df166ac08092b840b"} Sep 10 16:00:19.432259 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:19.432195 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-247f70-59f96f4bd9-crq27" podStartSLOduration=1.432181488 podStartE2EDuration="1.432181488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:00:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:00:19.431535028 +0000 UTC m=+1096.406794981" watchObservedRunningTime="2026-09-10 16:00:19.432181488 +0000 UTC m=+1096.407441441" Sep 10 16:00:20.019461 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:20.019429 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:20.424369 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:20.424329 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-247f70-5477449978-2rf7b" event={"ID":"dc6c29de-8858-438f-b5b6-d9fe4b5be50d","Type":"ContainerStarted","Data":"56e6f59e1d5af33b51ef9b00cb09658de72778d098e0a3ba9f14436b6f47b93f"} Sep 10 16:00:20.424564 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:20.424422 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-247f70-5477449978-2rf7b" Sep 10 16:00:20.438740 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:20.438713 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:20.539546 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:20.539516 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:20.658401 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:20.658218 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-247f70-5477449978-2rf7b" podStartSLOduration=3.06172199 podStartE2EDuration="3.658196419s" totalImagesPullingTime="596.474429ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:00:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:00:20.443703276 +0000 UTC m=+1097.418963230" watchObservedRunningTime="2026-09-10 16:00:20.658196419 +0000 UTC m=+1097.633456373" Sep 10 16:00:20.661289 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:20.661259 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv"] Sep 10 16:00:20.661951 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:20.661920 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" podUID="3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc" containerName="istio-proxy" containerID="cri-o://9a44c48efa56498d8b650f0e311000c12e2ddfa8b37aa0a166b40b7089df6f93" gracePeriod=30 Sep 10 16:00:20.689990 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:20.689923 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:21.260359 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:21.260331 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:21.595635 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:21.595552 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:21.779087 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:21.779037 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:25.401730 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:25.401697 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:25.496525 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:25.496498 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:25.908632 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:25.908610 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" Sep 10 16:00:25.985681 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:25.985352 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b52270-6f9778864c-mmxv2"] Sep 10 16:00:25.986532 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:25.986300 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc" containerName="istio-proxy" Sep 10 16:00:25.986532 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:25.986323 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc" containerName="istio-proxy" Sep 10 16:00:25.986682 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:25.986556 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc" containerName="istio-proxy" Sep 10 16:00:25.986682 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:25.986579 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc" containerName="istio-proxy" Sep 10 16:00:25.991272 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:25.991245 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b52270-6f9778864c-mmxv2" Sep 10 16:00:25.993432 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:25.993406 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-b52270-serving-cert\"" Sep 10 16:00:25.999664 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:25.999638 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b52270-6f9778864c-mmxv2"] Sep 10 16:00:26.030731 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.030709 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:26.054447 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.054417 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-istiod-ca-cert\") pod \"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc\" (UID: \"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc\") " Sep 10 16:00:26.054565 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.054500 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-workload-certs\") pod \"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc\" (UID: \"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc\") " Sep 10 16:00:26.054565 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.054543 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-istio-podinfo\") pod \"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc\" (UID: \"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc\") " Sep 10 16:00:26.054672 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.054571 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-istio-token\") pod \"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc\" (UID: \"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc\") " Sep 10 16:00:26.054672 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.054601 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5mk2\" (UniqueName: \"kubernetes.io/projected/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-kube-api-access-t5mk2\") pod \"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc\" (UID: \"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc\") " Sep 10 16:00:26.054672 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.054647 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-istio-envoy\") pod \"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc\" (UID: \"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc\") " Sep 10 16:00:26.054809 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.054678 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-istio-data\") pod \"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc\" (UID: \"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc\") " Sep 10 16:00:26.054809 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.054705 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-workload-socket\") pod \"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc\" (UID: \"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc\") " Sep 10 16:00:26.054809 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.054750 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-credential-socket\") pod \"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc\" (UID: \"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc\") " Sep 10 16:00:26.054954 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.054836 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc" (UID: "3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:00:26.054954 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.054894 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e645ebb-52bc-46c6-9429-b4d7d447649a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-b52270-6f9778864c-mmxv2\" (UID: \"5e645ebb-52bc-46c6-9429-b4d7d447649a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b52270-6f9778864c-mmxv2" Sep 10 16:00:26.055072 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.055042 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e645ebb-52bc-46c6-9429-b4d7d447649a-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-b52270-6f9778864c-mmxv2\" (UID: \"5e645ebb-52bc-46c6-9429-b4d7d447649a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b52270-6f9778864c-mmxv2" Sep 10 16:00:26.055072 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.055022 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-istio-data" (OuterVolumeSpecName: "istio-data") pod "3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc" (UID: "3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:00:26.055173 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.055104 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc" (UID: "3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:00:26.055173 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.055108 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwzr8\" (UniqueName: \"kubernetes.io/projected/5e645ebb-52bc-46c6-9429-b4d7d447649a-kube-api-access-hwzr8\") pod \"maas-api-e2e-shared-a-w3-b52270-6f9778864c-mmxv2\" (UID: \"5e645ebb-52bc-46c6-9429-b4d7d447649a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b52270-6f9778864c-mmxv2" Sep 10 16:00:26.055267 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.055186 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e645ebb-52bc-46c6-9429-b4d7d447649a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-b52270-6f9778864c-mmxv2\" (UID: \"5e645ebb-52bc-46c6-9429-b4d7d447649a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b52270-6f9778864c-mmxv2" Sep 10 16:00:26.055267 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.055236 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-istio-data\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:00:26.055267 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.055251 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-workload-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:00:26.055456 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.055266 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-workload-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:00:26.055456 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.055278 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc" (UID: "3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:00:26.055568 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.055456 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc" (UID: "3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:00:26.057015 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.056983 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc" (UID: "3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:00:26.057108 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.057013 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-kube-api-access-t5mk2" (OuterVolumeSpecName: "kube-api-access-t5mk2") pod "3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc" (UID: "3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc"). InnerVolumeSpecName "kube-api-access-t5mk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:00:26.057243 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.057220 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-istio-token" (OuterVolumeSpecName: "istio-token") pod "3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc" (UID: "3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:00:26.057362 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.057348 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc" (UID: "3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 16:00:26.155847 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.155810 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e645ebb-52bc-46c6-9429-b4d7d447649a-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-b52270-6f9778864c-mmxv2\" (UID: \"5e645ebb-52bc-46c6-9429-b4d7d447649a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b52270-6f9778864c-mmxv2" Sep 10 16:00:26.156004 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.155851 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwzr8\" (UniqueName: \"kubernetes.io/projected/5e645ebb-52bc-46c6-9429-b4d7d447649a-kube-api-access-hwzr8\") pod \"maas-api-e2e-shared-a-w3-b52270-6f9778864c-mmxv2\" (UID: \"5e645ebb-52bc-46c6-9429-b4d7d447649a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b52270-6f9778864c-mmxv2" Sep 10 16:00:26.156092 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.156005 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e645ebb-52bc-46c6-9429-b4d7d447649a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-b52270-6f9778864c-mmxv2\" (UID: \"5e645ebb-52bc-46c6-9429-b4d7d447649a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b52270-6f9778864c-mmxv2" Sep 10 16:00:26.156092 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.156075 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e645ebb-52bc-46c6-9429-b4d7d447649a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-b52270-6f9778864c-mmxv2\" (UID: \"5e645ebb-52bc-46c6-9429-b4d7d447649a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b52270-6f9778864c-mmxv2" Sep 10 16:00:26.156219 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.156174 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5mk2\" (UniqueName: \"kubernetes.io/projected/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-kube-api-access-t5mk2\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:00:26.156219 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.156192 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-istio-envoy\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:00:26.156219 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.156207 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-credential-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:00:26.156394 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.156221 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-istiod-ca-cert\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:00:26.156394 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.156235 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-istio-podinfo\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:00:26.156394 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.156247 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc-istio-token\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:00:26.156779 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.156746 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e645ebb-52bc-46c6-9429-b4d7d447649a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-b52270-6f9778864c-mmxv2\" (UID: \"5e645ebb-52bc-46c6-9429-b4d7d447649a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b52270-6f9778864c-mmxv2" Sep 10 16:00:26.158320 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.158296 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e645ebb-52bc-46c6-9429-b4d7d447649a-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-b52270-6f9778864c-mmxv2\" (UID: \"5e645ebb-52bc-46c6-9429-b4d7d447649a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b52270-6f9778864c-mmxv2" Sep 10 16:00:26.158416 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.158373 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e645ebb-52bc-46c6-9429-b4d7d447649a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-b52270-6f9778864c-mmxv2\" (UID: \"5e645ebb-52bc-46c6-9429-b4d7d447649a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b52270-6f9778864c-mmxv2" Sep 10 16:00:26.166853 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.166826 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwzr8\" (UniqueName: \"kubernetes.io/projected/5e645ebb-52bc-46c6-9429-b4d7d447649a-kube-api-access-hwzr8\") pod \"maas-api-e2e-shared-a-w3-b52270-6f9778864c-mmxv2\" (UID: \"5e645ebb-52bc-46c6-9429-b4d7d447649a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b52270-6f9778864c-mmxv2" Sep 10 16:00:26.273060 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.273005 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:26.305179 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.305154 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b52270-6f9778864c-mmxv2" Sep 10 16:00:26.435914 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.435888 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-247f70-5477449978-2rf7b" Sep 10 16:00:26.460673 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.460649 2583 generic.go:356] "Generic (PLEG): container finished" podID="3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc" containerID="9a44c48efa56498d8b650f0e311000c12e2ddfa8b37aa0a166b40b7089df6f93" exitCode=0 Sep 10 16:00:26.460822 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.460744 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" Sep 10 16:00:26.460822 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.460757 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" event={"ID":"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc","Type":"ContainerDied","Data":"9a44c48efa56498d8b650f0e311000c12e2ddfa8b37aa0a166b40b7089df6f93"} Sep 10 16:00:26.460822 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.460805 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv" event={"ID":"3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc","Type":"ContainerDied","Data":"e926c40732434d1a695bd301d0e748d11e215a40de87b5195ffb23c9daabd8c4"} Sep 10 16:00:26.460949 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.460830 2583 scope.go:117] "RemoveContainer" containerID="9a44c48efa56498d8b650f0e311000c12e2ddfa8b37aa0a166b40b7089df6f93" Sep 10 16:00:26.470489 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.470459 2583 scope.go:117] "RemoveContainer" containerID="22dc9419dcbf54683906cc31e483dcdbc7c24c0969aae44584d9f64b6cc10535" Sep 10 16:00:26.479078 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.479062 2583 scope.go:117] "RemoveContainer" containerID="9a44c48efa56498d8b650f0e311000c12e2ddfa8b37aa0a166b40b7089df6f93" Sep 10 16:00:26.479321 ip-10-0-139-107 kubenswrapper[2583]: E0910 16:00:26.479303 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a44c48efa56498d8b650f0e311000c12e2ddfa8b37aa0a166b40b7089df6f93\": container with ID starting with 9a44c48efa56498d8b650f0e311000c12e2ddfa8b37aa0a166b40b7089df6f93 not found: ID does not exist" containerID="9a44c48efa56498d8b650f0e311000c12e2ddfa8b37aa0a166b40b7089df6f93" Sep 10 16:00:26.479369 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.479329 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a44c48efa56498d8b650f0e311000c12e2ddfa8b37aa0a166b40b7089df6f93"} err="failed to get container status \"9a44c48efa56498d8b650f0e311000c12e2ddfa8b37aa0a166b40b7089df6f93\": rpc error: code = NotFound desc = could not find container \"9a44c48efa56498d8b650f0e311000c12e2ddfa8b37aa0a166b40b7089df6f93\": container with ID starting with 9a44c48efa56498d8b650f0e311000c12e2ddfa8b37aa0a166b40b7089df6f93 not found: ID does not exist" Sep 10 16:00:26.479369 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.479344 2583 scope.go:117] "RemoveContainer" containerID="22dc9419dcbf54683906cc31e483dcdbc7c24c0969aae44584d9f64b6cc10535" Sep 10 16:00:26.479599 ip-10-0-139-107 kubenswrapper[2583]: E0910 16:00:26.479583 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22dc9419dcbf54683906cc31e483dcdbc7c24c0969aae44584d9f64b6cc10535\": container with ID starting with 22dc9419dcbf54683906cc31e483dcdbc7c24c0969aae44584d9f64b6cc10535 not found: ID does not exist" containerID="22dc9419dcbf54683906cc31e483dcdbc7c24c0969aae44584d9f64b6cc10535" Sep 10 16:00:26.479642 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.479604 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22dc9419dcbf54683906cc31e483dcdbc7c24c0969aae44584d9f64b6cc10535"} err="failed to get container status \"22dc9419dcbf54683906cc31e483dcdbc7c24c0969aae44584d9f64b6cc10535\": rpc error: code = NotFound desc = could not find container \"22dc9419dcbf54683906cc31e483dcdbc7c24c0969aae44584d9f64b6cc10535\": container with ID starting with 22dc9419dcbf54683906cc31e483dcdbc7c24c0969aae44584d9f64b6cc10535 not found: ID does not exist" Sep 10 16:00:26.489234 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.489208 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv"] Sep 10 16:00:26.499735 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.499713 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-607b9a-openshift-default-688f5cbf6b-4c5sv"] Sep 10 16:00:26.590925 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.590900 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b52270-6f9778864c-mmxv2"] Sep 10 16:00:26.592411 ip-10-0-139-107 kubenswrapper[2583]: W0910 16:00:26.592365 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e645ebb_52bc_46c6_9429_b4d7d447649a.slice/crio-4e7087c0fac86e3cb82728a978b377773d33a1641ff1d6c1f87402b6a05a0285 WatchSource:0}: Error finding container 4e7087c0fac86e3cb82728a978b377773d33a1641ff1d6c1f87402b6a05a0285: Status 404 returned error can't find the container with id 4e7087c0fac86e3cb82728a978b377773d33a1641ff1d6c1f87402b6a05a0285 Sep 10 16:00:26.785768 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.785737 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:26.882305 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.882280 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:26.883630 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.883606 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b52270-bb595ff74-jg88r"] Sep 10 16:00:26.884057 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.884040 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc" containerName="istio-proxy" Sep 10 16:00:26.884132 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.884060 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc" containerName="istio-proxy" Sep 10 16:00:26.888602 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.888581 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b52270-bb595ff74-jg88r" Sep 10 16:00:26.890618 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.890601 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-b52270\"" Sep 10 16:00:26.898873 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.898844 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b52270-bb595ff74-jg88r"] Sep 10 16:00:26.963266 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.963236 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcwhp\" (UniqueName: \"kubernetes.io/projected/bb00c59d-fe86-4af7-97b2-5d4d81c72330-kube-api-access-kcwhp\") pod \"payload-pre-processing-e2e-shared-a-w3-b52270-bb595ff74-jg88r\" (UID: \"bb00c59d-fe86-4af7-97b2-5d4d81c72330\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b52270-bb595ff74-jg88r" Sep 10 16:00:26.963395 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:26.963315 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb00c59d-fe86-4af7-97b2-5d4d81c72330-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-b52270-bb595ff74-jg88r\" (UID: \"bb00c59d-fe86-4af7-97b2-5d4d81c72330\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b52270-bb595ff74-jg88r" Sep 10 16:00:27.064699 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:27.064666 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb00c59d-fe86-4af7-97b2-5d4d81c72330-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-b52270-bb595ff74-jg88r\" (UID: \"bb00c59d-fe86-4af7-97b2-5d4d81c72330\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b52270-bb595ff74-jg88r" Sep 10 16:00:27.064849 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:27.064758 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcwhp\" (UniqueName: \"kubernetes.io/projected/bb00c59d-fe86-4af7-97b2-5d4d81c72330-kube-api-access-kcwhp\") pod \"payload-pre-processing-e2e-shared-a-w3-b52270-bb595ff74-jg88r\" (UID: \"bb00c59d-fe86-4af7-97b2-5d4d81c72330\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b52270-bb595ff74-jg88r" Sep 10 16:00:27.065408 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:27.065362 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb00c59d-fe86-4af7-97b2-5d4d81c72330-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-b52270-bb595ff74-jg88r\" (UID: \"bb00c59d-fe86-4af7-97b2-5d4d81c72330\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b52270-bb595ff74-jg88r" Sep 10 16:00:27.073003 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:27.072979 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcwhp\" (UniqueName: \"kubernetes.io/projected/bb00c59d-fe86-4af7-97b2-5d4d81c72330-kube-api-access-kcwhp\") pod \"payload-pre-processing-e2e-shared-a-w3-b52270-bb595ff74-jg88r\" (UID: \"bb00c59d-fe86-4af7-97b2-5d4d81c72330\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b52270-bb595ff74-jg88r" Sep 10 16:00:27.201092 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:27.201023 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b52270-bb595ff74-jg88r" Sep 10 16:00:27.469499 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:27.469417 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b52270-6f9778864c-mmxv2" event={"ID":"5e645ebb-52bc-46c6-9429-b4d7d447649a","Type":"ContainerStarted","Data":"f6509b2c49e76ca467f07713336e900eee59a24cbb83709a260b6b21e64f3b3c"} Sep 10 16:00:27.469499 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:27.469475 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b52270-6f9778864c-mmxv2" Sep 10 16:00:27.469499 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:27.469493 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b52270-6f9778864c-mmxv2" event={"ID":"5e645ebb-52bc-46c6-9429-b4d7d447649a","Type":"ContainerStarted","Data":"4e7087c0fac86e3cb82728a978b377773d33a1641ff1d6c1f87402b6a05a0285"} Sep 10 16:00:27.478366 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:27.478344 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b52270-bb595ff74-jg88r"] Sep 10 16:00:27.479159 ip-10-0-139-107 kubenswrapper[2583]: W0910 16:00:27.479132 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb00c59d_fe86_4af7_97b2_5d4d81c72330.slice/crio-53bcb63c234293974bf02ad705f4a0511737232de05ae9fa59400ef7db0da259 WatchSource:0}: Error finding container 53bcb63c234293974bf02ad705f4a0511737232de05ae9fa59400ef7db0da259: Status 404 returned error can't find the container with id 53bcb63c234293974bf02ad705f4a0511737232de05ae9fa59400ef7db0da259 Sep 10 16:00:27.493842 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:27.493802 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b52270-6f9778864c-mmxv2" podStartSLOduration=2.142658587 podStartE2EDuration="2.493788011s" totalImagesPullingTime="351.129424ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:00:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:00:27.491001522 +0000 UTC m=+1104.466261486" watchObservedRunningTime="2026-09-10 16:00:27.493788011 +0000 UTC m=+1104.469047963" Sep 10 16:00:27.515997 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:27.515969 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc" path="/var/lib/kubelet/pods/3bec5d6c-78ee-4ab2-9cb4-1df40dfc13fc/volumes" Sep 10 16:00:27.864266 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:27.864195 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:28.477169 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:28.477133 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b52270-bb595ff74-jg88r" event={"ID":"bb00c59d-fe86-4af7-97b2-5d4d81c72330","Type":"ContainerStarted","Data":"b24260d6ebb5760ed5eb1c0c02589a49303835010022552b104a7438b9f7351e"} Sep 10 16:00:28.477666 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:28.477213 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b52270-bb595ff74-jg88r" Sep 10 16:00:28.477666 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:28.477241 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b52270-bb595ff74-jg88r" event={"ID":"bb00c59d-fe86-4af7-97b2-5d4d81c72330","Type":"ContainerStarted","Data":"53bcb63c234293974bf02ad705f4a0511737232de05ae9fa59400ef7db0da259"} Sep 10 16:00:28.495790 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:28.495743 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b52270-bb595ff74-jg88r" podStartSLOduration=2.495728946 podStartE2EDuration="2.495728946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:00:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:00:28.494098026 +0000 UTC m=+1105.469357980" watchObservedRunningTime="2026-09-10 16:00:28.495728946 +0000 UTC m=+1105.470988946" Sep 10 16:00:30.426728 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:30.426697 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-247f70-59f96f4bd9-crq27" Sep 10 16:00:33.483072 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:33.483032 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b52270-6f9778864c-mmxv2" Sep 10 16:00:39.485567 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:39.485532 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b52270-bb595ff74-jg88r" Sep 10 16:00:40.140666 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:40.140633 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:41.656184 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:41.656142 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:41.998408 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:41.998300 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:42.380743 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:42.380715 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:42.567173 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:42.567139 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:45.935900 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:45.935868 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:46.036669 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:46.036645 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:47.093041 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:47.093006 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-298d9b-6f8fc6656c-qk477"] Sep 10 16:00:47.098074 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:47.098039 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-298d9b-6f8fc6656c-qk477" Sep 10 16:00:47.105660 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:47.105629 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-298d9b-serving-cert\"" Sep 10 16:00:47.118300 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:47.118246 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-298d9b-6f8fc6656c-qk477"] Sep 10 16:00:47.191440 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:47.191393 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:47.245875 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:47.245845 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0976bab9-5429-4077-a7ea-a33edc5175ca-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-298d9b-6f8fc6656c-qk477\" (UID: \"0976bab9-5429-4077-a7ea-a33edc5175ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-298d9b-6f8fc6656c-qk477" Sep 10 16:00:47.246036 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:47.245893 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-859f9\" (UniqueName: \"kubernetes.io/projected/0976bab9-5429-4077-a7ea-a33edc5175ca-kube-api-access-859f9\") pod \"maas-api-e2e-shared-b-w2-298d9b-6f8fc6656c-qk477\" (UID: \"0976bab9-5429-4077-a7ea-a33edc5175ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-298d9b-6f8fc6656c-qk477" Sep 10 16:00:47.246036 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:47.245963 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0976bab9-5429-4077-a7ea-a33edc5175ca-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-298d9b-6f8fc6656c-qk477\" (UID: \"0976bab9-5429-4077-a7ea-a33edc5175ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-298d9b-6f8fc6656c-qk477" Sep 10 16:00:47.246036 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:47.246002 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0976bab9-5429-4077-a7ea-a33edc5175ca-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-298d9b-6f8fc6656c-qk477\" (UID: \"0976bab9-5429-4077-a7ea-a33edc5175ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-298d9b-6f8fc6656c-qk477" Sep 10 16:00:47.347370 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:47.347281 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0976bab9-5429-4077-a7ea-a33edc5175ca-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-298d9b-6f8fc6656c-qk477\" (UID: \"0976bab9-5429-4077-a7ea-a33edc5175ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-298d9b-6f8fc6656c-qk477" Sep 10 16:00:47.347370 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:47.347355 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0976bab9-5429-4077-a7ea-a33edc5175ca-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-298d9b-6f8fc6656c-qk477\" (UID: \"0976bab9-5429-4077-a7ea-a33edc5175ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-298d9b-6f8fc6656c-qk477" Sep 10 16:00:47.347622 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:47.347432 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-859f9\" (UniqueName: \"kubernetes.io/projected/0976bab9-5429-4077-a7ea-a33edc5175ca-kube-api-access-859f9\") pod \"maas-api-e2e-shared-b-w2-298d9b-6f8fc6656c-qk477\" (UID: \"0976bab9-5429-4077-a7ea-a33edc5175ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-298d9b-6f8fc6656c-qk477" Sep 10 16:00:47.347622 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:47.347501 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0976bab9-5429-4077-a7ea-a33edc5175ca-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-298d9b-6f8fc6656c-qk477\" (UID: \"0976bab9-5429-4077-a7ea-a33edc5175ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-298d9b-6f8fc6656c-qk477" Sep 10 16:00:47.347946 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:47.347921 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0976bab9-5429-4077-a7ea-a33edc5175ca-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-298d9b-6f8fc6656c-qk477\" (UID: \"0976bab9-5429-4077-a7ea-a33edc5175ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-298d9b-6f8fc6656c-qk477" Sep 10 16:00:47.349719 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:47.349697 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0976bab9-5429-4077-a7ea-a33edc5175ca-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-298d9b-6f8fc6656c-qk477\" (UID: \"0976bab9-5429-4077-a7ea-a33edc5175ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-298d9b-6f8fc6656c-qk477" Sep 10 16:00:47.349821 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:47.349765 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0976bab9-5429-4077-a7ea-a33edc5175ca-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-298d9b-6f8fc6656c-qk477\" (UID: \"0976bab9-5429-4077-a7ea-a33edc5175ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-298d9b-6f8fc6656c-qk477" Sep 10 16:00:47.356074 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:47.356049 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:47.359008 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:47.358986 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-859f9\" (UniqueName: \"kubernetes.io/projected/0976bab9-5429-4077-a7ea-a33edc5175ca-kube-api-access-859f9\") pod \"maas-api-e2e-shared-b-w2-298d9b-6f8fc6656c-qk477\" (UID: \"0976bab9-5429-4077-a7ea-a33edc5175ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-298d9b-6f8fc6656c-qk477" Sep 10 16:00:47.417415 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:47.417394 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-298d9b-6f8fc6656c-qk477" Sep 10 16:00:47.556253 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:47.556220 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:47.711548 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:47.711524 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-298d9b-6f8fc6656c-qk477"] Sep 10 16:00:47.713314 ip-10-0-139-107 kubenswrapper[2583]: W0910 16:00:47.713289 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0976bab9_5429_4077_a7ea_a33edc5175ca.slice/crio-da26defebce510fca6313c0d4291619bb8dc8ed2e4644366614f1944c43c6ac6 WatchSource:0}: Error finding container da26defebce510fca6313c0d4291619bb8dc8ed2e4644366614f1944c43c6ac6: Status 404 returned error can't find the container with id da26defebce510fca6313c0d4291619bb8dc8ed2e4644366614f1944c43c6ac6 Sep 10 16:00:47.917155 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:47.917124 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:48.033372 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:48.033266 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:48.523868 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:48.523833 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:48.588160 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:48.588127 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-298d9b-6f8fc6656c-qk477" event={"ID":"0976bab9-5429-4077-a7ea-a33edc5175ca","Type":"ContainerStarted","Data":"da26defebce510fca6313c0d4291619bb8dc8ed2e4644366614f1944c43c6ac6"} Sep 10 16:00:49.538071 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:49.538042 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:49.597271 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:49.597236 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-298d9b-6f8fc6656c-qk477" event={"ID":"0976bab9-5429-4077-a7ea-a33edc5175ca","Type":"ContainerStarted","Data":"07611b1834e77f6414bd7fead9b71f7e3141294b6ab6c8bce64e0a33617ae88f"} Sep 10 16:00:49.597443 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:49.597392 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-298d9b-6f8fc6656c-qk477" Sep 10 16:00:49.620061 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:49.620016 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-298d9b-6f8fc6656c-qk477" podStartSLOduration=1.7398296260000001 podStartE2EDuration="2.62000183s" totalImagesPullingTime="880.172204ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:00:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:00:49.618576139 +0000 UTC m=+1126.593836113" watchObservedRunningTime="2026-09-10 16:00:49.62000183 +0000 UTC m=+1126.595261783" Sep 10 16:00:50.300434 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:50.300403 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:54.827209 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:54.827176 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:55.180155 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:55.180128 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:55.608665 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:55.608584 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-298d9b-6f8fc6656c-qk477" Sep 10 16:00:55.854768 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:55.854713 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:56.007597 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:56.007562 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:00:59.019771 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:00:59.019740 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:01:00.388326 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:00.388283 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-108dbb-74d7f9c965-p5hhn"] Sep 10 16:01:00.391984 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:00.391964 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-108dbb-74d7f9c965-p5hhn" Sep 10 16:01:00.394055 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:00.394034 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-108dbb-serving-cert\"" Sep 10 16:01:00.403436 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:00.403363 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-108dbb-74d7f9c965-p5hhn"] Sep 10 16:01:00.451539 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:00.451505 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/807a838c-5645-422a-9ab2-2d1a0b139189-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-108dbb-74d7f9c965-p5hhn\" (UID: \"807a838c-5645-422a-9ab2-2d1a0b139189\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-108dbb-74d7f9c965-p5hhn" Sep 10 16:01:00.451539 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:00.451540 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkf4t\" (UniqueName: \"kubernetes.io/projected/807a838c-5645-422a-9ab2-2d1a0b139189-kube-api-access-pkf4t\") pod \"maas-api-e2e-shared-b-w3-108dbb-74d7f9c965-p5hhn\" (UID: \"807a838c-5645-422a-9ab2-2d1a0b139189\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-108dbb-74d7f9c965-p5hhn" Sep 10 16:01:00.451706 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:00.451558 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/807a838c-5645-422a-9ab2-2d1a0b139189-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-108dbb-74d7f9c965-p5hhn\" (UID: \"807a838c-5645-422a-9ab2-2d1a0b139189\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-108dbb-74d7f9c965-p5hhn" Sep 10 16:01:00.451706 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:00.451687 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/807a838c-5645-422a-9ab2-2d1a0b139189-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-108dbb-74d7f9c965-p5hhn\" (UID: \"807a838c-5645-422a-9ab2-2d1a0b139189\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-108dbb-74d7f9c965-p5hhn" Sep 10 16:01:00.552844 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:00.552802 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/807a838c-5645-422a-9ab2-2d1a0b139189-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-108dbb-74d7f9c965-p5hhn\" (UID: \"807a838c-5645-422a-9ab2-2d1a0b139189\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-108dbb-74d7f9c965-p5hhn" Sep 10 16:01:00.552999 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:00.552853 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkf4t\" (UniqueName: \"kubernetes.io/projected/807a838c-5645-422a-9ab2-2d1a0b139189-kube-api-access-pkf4t\") pod \"maas-api-e2e-shared-b-w3-108dbb-74d7f9c965-p5hhn\" (UID: \"807a838c-5645-422a-9ab2-2d1a0b139189\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-108dbb-74d7f9c965-p5hhn" Sep 10 16:01:00.553052 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:00.553029 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/807a838c-5645-422a-9ab2-2d1a0b139189-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-108dbb-74d7f9c965-p5hhn\" (UID: \"807a838c-5645-422a-9ab2-2d1a0b139189\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-108dbb-74d7f9c965-p5hhn" Sep 10 16:01:00.553139 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:00.553121 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/807a838c-5645-422a-9ab2-2d1a0b139189-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-108dbb-74d7f9c965-p5hhn\" (UID: \"807a838c-5645-422a-9ab2-2d1a0b139189\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-108dbb-74d7f9c965-p5hhn" Sep 10 16:01:00.553281 ip-10-0-139-107 kubenswrapper[2583]: E0910 16:01:00.553258 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-108dbb-serving-cert: secret "maas-api-e2e-shared-b-w3-108dbb-serving-cert" not found Sep 10 16:01:00.553362 ip-10-0-139-107 kubenswrapper[2583]: E0910 16:01:00.553349 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/807a838c-5645-422a-9ab2-2d1a0b139189-maas-api-tls podName:807a838c-5645-422a-9ab2-2d1a0b139189 nodeName:}" failed. No retries permitted until 2026-09-10 16:01:01.053328391 +0000 UTC m=+1138.028588337 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/807a838c-5645-422a-9ab2-2d1a0b139189-maas-api-tls") pod "maas-api-e2e-shared-b-w3-108dbb-74d7f9c965-p5hhn" (UID: "807a838c-5645-422a-9ab2-2d1a0b139189") : secret "maas-api-e2e-shared-b-w3-108dbb-serving-cert" not found Sep 10 16:01:00.553734 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:00.553714 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/807a838c-5645-422a-9ab2-2d1a0b139189-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-108dbb-74d7f9c965-p5hhn\" (UID: \"807a838c-5645-422a-9ab2-2d1a0b139189\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-108dbb-74d7f9c965-p5hhn" Sep 10 16:01:00.555255 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:00.555232 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/807a838c-5645-422a-9ab2-2d1a0b139189-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-108dbb-74d7f9c965-p5hhn\" (UID: \"807a838c-5645-422a-9ab2-2d1a0b139189\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-108dbb-74d7f9c965-p5hhn" Sep 10 16:01:00.565474 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:00.565453 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkf4t\" (UniqueName: \"kubernetes.io/projected/807a838c-5645-422a-9ab2-2d1a0b139189-kube-api-access-pkf4t\") pod \"maas-api-e2e-shared-b-w3-108dbb-74d7f9c965-p5hhn\" (UID: \"807a838c-5645-422a-9ab2-2d1a0b139189\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-108dbb-74d7f9c965-p5hhn" Sep 10 16:01:01.058465 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:01.058423 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/807a838c-5645-422a-9ab2-2d1a0b139189-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-108dbb-74d7f9c965-p5hhn\" (UID: \"807a838c-5645-422a-9ab2-2d1a0b139189\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-108dbb-74d7f9c965-p5hhn" Sep 10 16:01:01.060953 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:01.060927 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/807a838c-5645-422a-9ab2-2d1a0b139189-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-108dbb-74d7f9c965-p5hhn\" (UID: \"807a838c-5645-422a-9ab2-2d1a0b139189\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-108dbb-74d7f9c965-p5hhn" Sep 10 16:01:01.211535 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:01.211498 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-108dbb-c46946898-7sxn5"] Sep 10 16:01:01.216520 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:01.216503 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-108dbb-c46946898-7sxn5" Sep 10 16:01:01.218884 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:01.218860 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-108dbb-dockercfg-m4n96\"" Sep 10 16:01:01.218993 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:01.218862 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-108dbb\"" Sep 10 16:01:01.226981 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:01.226956 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-108dbb-c46946898-7sxn5"] Sep 10 16:01:01.260217 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:01.260179 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8p4f\" (UniqueName: \"kubernetes.io/projected/6c6c9359-d7ae-4c58-9a62-16104d1deb17-kube-api-access-d8p4f\") pod \"payload-pre-processing-e2e-shared-b-w3-108dbb-c46946898-7sxn5\" (UID: \"6c6c9359-d7ae-4c58-9a62-16104d1deb17\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-108dbb-c46946898-7sxn5" Sep 10 16:01:01.260396 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:01.260255 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c6c9359-d7ae-4c58-9a62-16104d1deb17-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-108dbb-c46946898-7sxn5\" (UID: \"6c6c9359-d7ae-4c58-9a62-16104d1deb17\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-108dbb-c46946898-7sxn5" Sep 10 16:01:01.305456 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:01.305427 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-108dbb-74d7f9c965-p5hhn" Sep 10 16:01:01.361016 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:01.360958 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8p4f\" (UniqueName: \"kubernetes.io/projected/6c6c9359-d7ae-4c58-9a62-16104d1deb17-kube-api-access-d8p4f\") pod \"payload-pre-processing-e2e-shared-b-w3-108dbb-c46946898-7sxn5\" (UID: \"6c6c9359-d7ae-4c58-9a62-16104d1deb17\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-108dbb-c46946898-7sxn5" Sep 10 16:01:01.361127 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:01.361044 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c6c9359-d7ae-4c58-9a62-16104d1deb17-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-108dbb-c46946898-7sxn5\" (UID: \"6c6c9359-d7ae-4c58-9a62-16104d1deb17\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-108dbb-c46946898-7sxn5" Sep 10 16:01:01.362109 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:01.362082 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c6c9359-d7ae-4c58-9a62-16104d1deb17-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-108dbb-c46946898-7sxn5\" (UID: \"6c6c9359-d7ae-4c58-9a62-16104d1deb17\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-108dbb-c46946898-7sxn5" Sep 10 16:01:01.370799 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:01.370771 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8p4f\" (UniqueName: \"kubernetes.io/projected/6c6c9359-d7ae-4c58-9a62-16104d1deb17-kube-api-access-d8p4f\") pod \"payload-pre-processing-e2e-shared-b-w3-108dbb-c46946898-7sxn5\" (UID: \"6c6c9359-d7ae-4c58-9a62-16104d1deb17\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-108dbb-c46946898-7sxn5" Sep 10 16:01:01.529877 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:01.529846 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-108dbb-c46946898-7sxn5" Sep 10 16:01:01.584080 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:01.584037 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-108dbb-74d7f9c965-p5hhn"] Sep 10 16:01:01.584786 ip-10-0-139-107 kubenswrapper[2583]: W0910 16:01:01.584499 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod807a838c_5645_422a_9ab2_2d1a0b139189.slice/crio-94de21d441f9c997405d4a2c961ec3d230be9d21cfe4102d8830b0f123183cb6 WatchSource:0}: Error finding container 94de21d441f9c997405d4a2c961ec3d230be9d21cfe4102d8830b0f123183cb6: Status 404 returned error can't find the container with id 94de21d441f9c997405d4a2c961ec3d230be9d21cfe4102d8830b0f123183cb6 Sep 10 16:01:01.668237 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:01.668204 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-108dbb-74d7f9c965-p5hhn" event={"ID":"807a838c-5645-422a-9ab2-2d1a0b139189","Type":"ContainerStarted","Data":"94de21d441f9c997405d4a2c961ec3d230be9d21cfe4102d8830b0f123183cb6"} Sep 10 16:01:01.815620 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:01.815592 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-108dbb-c46946898-7sxn5"] Sep 10 16:01:01.817374 ip-10-0-139-107 kubenswrapper[2583]: W0910 16:01:01.817353 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c6c9359_d7ae_4c58_9a62_16104d1deb17.slice/crio-cf14084d40296ade9e442856c40f60823f89beca08c9b53359fc896f7ce5fa0b WatchSource:0}: Error finding container cf14084d40296ade9e442856c40f60823f89beca08c9b53359fc896f7ce5fa0b: Status 404 returned error can't find the container with id cf14084d40296ade9e442856c40f60823f89beca08c9b53359fc896f7ce5fa0b Sep 10 16:01:02.675255 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:02.675215 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-108dbb-74d7f9c965-p5hhn" event={"ID":"807a838c-5645-422a-9ab2-2d1a0b139189","Type":"ContainerStarted","Data":"ede242e6225ca72075e0f4ef872c0f58dfe877ee53569000a8468ef033ec8585"} Sep 10 16:01:02.675759 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:02.675410 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-108dbb-74d7f9c965-p5hhn" Sep 10 16:01:02.677720 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:02.677688 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-108dbb-c46946898-7sxn5" event={"ID":"6c6c9359-d7ae-4c58-9a62-16104d1deb17","Type":"ContainerStarted","Data":"442b99922db3cdad3ce725935531dfd6fac1f8838a5b0a33331481e57b3a56cf"} Sep 10 16:01:02.677842 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:02.677745 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-108dbb-c46946898-7sxn5" Sep 10 16:01:02.677842 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:02.677772 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-108dbb-c46946898-7sxn5" event={"ID":"6c6c9359-d7ae-4c58-9a62-16104d1deb17","Type":"ContainerStarted","Data":"cf14084d40296ade9e442856c40f60823f89beca08c9b53359fc896f7ce5fa0b"} Sep 10 16:01:02.694498 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:02.694455 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-108dbb-74d7f9c965-p5hhn" podStartSLOduration=1.958828355 podStartE2EDuration="2.694442065s" totalImagesPullingTime="735.61371ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:01:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:01:02.692241964 +0000 UTC m=+1139.667501911" watchObservedRunningTime="2026-09-10 16:01:02.694442065 +0000 UTC m=+1139.669702042" Sep 10 16:01:02.713458 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:02.713411 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-108dbb-c46946898-7sxn5" podStartSLOduration=1.7133665919999999 podStartE2EDuration="1.713366592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:01:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:01:02.711499427 +0000 UTC m=+1139.686759377" watchObservedRunningTime="2026-09-10 16:01:02.713366592 +0000 UTC m=+1139.688626545" Sep 10 16:01:02.713878 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:02.713861 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:01:08.442867 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:08.442832 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:01:08.696174 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:08.696097 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-108dbb-74d7f9c965-p5hhn" Sep 10 16:01:10.913596 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:10.913563 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:01:13.693227 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:13.693196 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-108dbb-c46946898-7sxn5" Sep 10 16:01:19.957584 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:19.957552 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:01:20.271399 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:20.271294 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:01:20.536180 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:20.536101 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:01:21.526712 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:21.526682 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:01:23.231101 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:23.231069 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:01:23.331003 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:23.330970 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:01:23.965882 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:23.965852 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:01:24.373296 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:24.373206 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-69972456-69f59d7dcb-k4dkq"] Sep 10 16:01:24.377132 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:24.377110 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69972456-69f59d7dcb-k4dkq" Sep 10 16:01:24.379974 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:24.379952 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-69972456-serving-cert\"" Sep 10 16:01:24.389813 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:24.389792 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-69972456-69f59d7dcb-k4dkq"] Sep 10 16:01:24.422621 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:24.422598 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:01:24.467869 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:24.467842 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b647f08-2708-43ce-9e90-d5e57cbc34da-openshift-service-ca\") pod \"maas-api-e2e-mt-69972456-69f59d7dcb-k4dkq\" (UID: \"8b647f08-2708-43ce-9e90-d5e57cbc34da\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69972456-69f59d7dcb-k4dkq" Sep 10 16:01:24.467967 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:24.467914 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm7dw\" (UniqueName: \"kubernetes.io/projected/8b647f08-2708-43ce-9e90-d5e57cbc34da-kube-api-access-dm7dw\") pod \"maas-api-e2e-mt-69972456-69f59d7dcb-k4dkq\" (UID: \"8b647f08-2708-43ce-9e90-d5e57cbc34da\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69972456-69f59d7dcb-k4dkq" Sep 10 16:01:24.467967 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:24.467954 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b647f08-2708-43ce-9e90-d5e57cbc34da-metrics-certs\") pod \"maas-api-e2e-mt-69972456-69f59d7dcb-k4dkq\" (UID: \"8b647f08-2708-43ce-9e90-d5e57cbc34da\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69972456-69f59d7dcb-k4dkq" Sep 10 16:01:24.468041 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:24.468025 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b647f08-2708-43ce-9e90-d5e57cbc34da-maas-api-tls\") pod \"maas-api-e2e-mt-69972456-69f59d7dcb-k4dkq\" (UID: \"8b647f08-2708-43ce-9e90-d5e57cbc34da\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69972456-69f59d7dcb-k4dkq" Sep 10 16:01:24.516763 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:24.516731 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:01:24.568974 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:24.568948 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm7dw\" (UniqueName: \"kubernetes.io/projected/8b647f08-2708-43ce-9e90-d5e57cbc34da-kube-api-access-dm7dw\") pod \"maas-api-e2e-mt-69972456-69f59d7dcb-k4dkq\" (UID: \"8b647f08-2708-43ce-9e90-d5e57cbc34da\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69972456-69f59d7dcb-k4dkq" Sep 10 16:01:24.569121 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:24.568988 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b647f08-2708-43ce-9e90-d5e57cbc34da-metrics-certs\") pod \"maas-api-e2e-mt-69972456-69f59d7dcb-k4dkq\" (UID: \"8b647f08-2708-43ce-9e90-d5e57cbc34da\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69972456-69f59d7dcb-k4dkq" Sep 10 16:01:24.569121 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:24.569020 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b647f08-2708-43ce-9e90-d5e57cbc34da-maas-api-tls\") pod \"maas-api-e2e-mt-69972456-69f59d7dcb-k4dkq\" (UID: \"8b647f08-2708-43ce-9e90-d5e57cbc34da\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69972456-69f59d7dcb-k4dkq" Sep 10 16:01:24.569121 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:24.569080 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b647f08-2708-43ce-9e90-d5e57cbc34da-openshift-service-ca\") pod \"maas-api-e2e-mt-69972456-69f59d7dcb-k4dkq\" (UID: \"8b647f08-2708-43ce-9e90-d5e57cbc34da\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69972456-69f59d7dcb-k4dkq" Sep 10 16:01:24.569287 ip-10-0-139-107 kubenswrapper[2583]: E0910 16:01:24.569179 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-69972456-serving-cert: secret "maas-api-e2e-mt-69972456-serving-cert" not found Sep 10 16:01:24.569287 ip-10-0-139-107 kubenswrapper[2583]: E0910 16:01:24.569253 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b647f08-2708-43ce-9e90-d5e57cbc34da-maas-api-tls podName:8b647f08-2708-43ce-9e90-d5e57cbc34da nodeName:}" failed. No retries permitted until 2026-09-10 16:01:25.069237792 +0000 UTC m=+1162.044497722 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8b647f08-2708-43ce-9e90-d5e57cbc34da-maas-api-tls") pod "maas-api-e2e-mt-69972456-69f59d7dcb-k4dkq" (UID: "8b647f08-2708-43ce-9e90-d5e57cbc34da") : secret "maas-api-e2e-mt-69972456-serving-cert" not found Sep 10 16:01:24.569729 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:24.569708 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b647f08-2708-43ce-9e90-d5e57cbc34da-openshift-service-ca\") pod \"maas-api-e2e-mt-69972456-69f59d7dcb-k4dkq\" (UID: \"8b647f08-2708-43ce-9e90-d5e57cbc34da\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69972456-69f59d7dcb-k4dkq" Sep 10 16:01:24.571532 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:24.571511 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b647f08-2708-43ce-9e90-d5e57cbc34da-metrics-certs\") pod \"maas-api-e2e-mt-69972456-69f59d7dcb-k4dkq\" (UID: \"8b647f08-2708-43ce-9e90-d5e57cbc34da\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69972456-69f59d7dcb-k4dkq" Sep 10 16:01:24.577425 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:24.577402 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm7dw\" (UniqueName: \"kubernetes.io/projected/8b647f08-2708-43ce-9e90-d5e57cbc34da-kube-api-access-dm7dw\") pod \"maas-api-e2e-mt-69972456-69f59d7dcb-k4dkq\" (UID: \"8b647f08-2708-43ce-9e90-d5e57cbc34da\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69972456-69f59d7dcb-k4dkq" Sep 10 16:01:24.864812 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:24.864778 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:01:25.073335 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:25.073299 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b647f08-2708-43ce-9e90-d5e57cbc34da-maas-api-tls\") pod \"maas-api-e2e-mt-69972456-69f59d7dcb-k4dkq\" (UID: \"8b647f08-2708-43ce-9e90-d5e57cbc34da\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69972456-69f59d7dcb-k4dkq" Sep 10 16:01:25.076149 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:25.076120 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b647f08-2708-43ce-9e90-d5e57cbc34da-maas-api-tls\") pod \"maas-api-e2e-mt-69972456-69f59d7dcb-k4dkq\" (UID: \"8b647f08-2708-43ce-9e90-d5e57cbc34da\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-69972456-69f59d7dcb-k4dkq" Sep 10 16:01:25.121637 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:25.121549 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:01:25.248299 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:25.248269 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:01:25.289879 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:25.289851 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69972456-69f59d7dcb-k4dkq" Sep 10 16:01:25.570778 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:25.570748 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-69972456-69f59d7dcb-k4dkq"] Sep 10 16:01:25.572482 ip-10-0-139-107 kubenswrapper[2583]: W0910 16:01:25.572454 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b647f08_2708_43ce_9e90_d5e57cbc34da.slice/crio-07e5e728b5048c6d8819aaccd9937d20741d5b2e52394795d8680116aec6773b WatchSource:0}: Error finding container 07e5e728b5048c6d8819aaccd9937d20741d5b2e52394795d8680116aec6773b: Status 404 returned error can't find the container with id 07e5e728b5048c6d8819aaccd9937d20741d5b2e52394795d8680116aec6773b Sep 10 16:01:25.825069 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:25.824986 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69972456-69f59d7dcb-k4dkq" event={"ID":"8b647f08-2708-43ce-9e90-d5e57cbc34da","Type":"ContainerStarted","Data":"07e5e728b5048c6d8819aaccd9937d20741d5b2e52394795d8680116aec6773b"} Sep 10 16:01:26.835701 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:26.835662 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69972456-69f59d7dcb-k4dkq" event={"ID":"8b647f08-2708-43ce-9e90-d5e57cbc34da","Type":"ContainerStarted","Data":"93f0ea4359445a35804e17a7bad8bab97bec23f50b37f2ea1a3408255e23d061"} Sep 10 16:01:26.836061 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:26.835775 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69972456-69f59d7dcb-k4dkq" Sep 10 16:01:26.858530 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:26.858474 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69972456-69f59d7dcb-k4dkq" podStartSLOduration=2.1700110280000002 podStartE2EDuration="2.858457546s" totalImagesPullingTime="688.446518ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:01:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:01:26.854689454 +0000 UTC m=+1163.829949408" watchObservedRunningTime="2026-09-10 16:01:26.858457546 +0000 UTC m=+1163.833717500" Sep 10 16:01:32.847664 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:32.847638 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69972456-69f59d7dcb-k4dkq" Sep 10 16:01:36.211302 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:36.211268 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:01:48.238490 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:01:48.238455 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:02.907529 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:02:02.907494 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:06.580681 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:02:06.580644 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:12.561895 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:02:12.561859 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:15.463746 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:02:15.463715 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:44.222487 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:02:44.222438 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:02:58.946317 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:02:58.946224 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:14.323276 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:03:14.323238 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-69972456-69f59d7dcb-k4dkq"] Sep 10 16:03:14.323791 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:03:14.323467 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69972456-69f59d7dcb-k4dkq" podUID="8b647f08-2708-43ce-9e90-d5e57cbc34da" containerName="maas-api" containerID="cri-o://93f0ea4359445a35804e17a7bad8bab97bec23f50b37f2ea1a3408255e23d061" gracePeriod=30 Sep 10 16:03:14.345238 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:03:14.345214 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:14.446877 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:03:14.446846 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:14.474120 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:03:14.474088 2583 generic.go:356] "Generic (PLEG): container finished" podID="8b647f08-2708-43ce-9e90-d5e57cbc34da" containerID="93f0ea4359445a35804e17a7bad8bab97bec23f50b37f2ea1a3408255e23d061" exitCode=0 Sep 10 16:03:14.474279 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:03:14.474148 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69972456-69f59d7dcb-k4dkq" event={"ID":"8b647f08-2708-43ce-9e90-d5e57cbc34da","Type":"ContainerDied","Data":"93f0ea4359445a35804e17a7bad8bab97bec23f50b37f2ea1a3408255e23d061"} Sep 10 16:03:14.545480 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:03:14.545445 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:14.588281 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:03:14.588228 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69972456-69f59d7dcb-k4dkq" Sep 10 16:03:14.647509 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:03:14.647477 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:14.755922 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:03:14.755887 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b647f08-2708-43ce-9e90-d5e57cbc34da-maas-api-tls\") pod \"8b647f08-2708-43ce-9e90-d5e57cbc34da\" (UID: \"8b647f08-2708-43ce-9e90-d5e57cbc34da\") " Sep 10 16:03:14.756121 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:03:14.755985 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dm7dw\" (UniqueName: \"kubernetes.io/projected/8b647f08-2708-43ce-9e90-d5e57cbc34da-kube-api-access-dm7dw\") pod \"8b647f08-2708-43ce-9e90-d5e57cbc34da\" (UID: \"8b647f08-2708-43ce-9e90-d5e57cbc34da\") " Sep 10 16:03:14.756121 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:03:14.756038 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b647f08-2708-43ce-9e90-d5e57cbc34da-openshift-service-ca\") pod \"8b647f08-2708-43ce-9e90-d5e57cbc34da\" (UID: \"8b647f08-2708-43ce-9e90-d5e57cbc34da\") " Sep 10 16:03:14.756266 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:03:14.756176 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b647f08-2708-43ce-9e90-d5e57cbc34da-metrics-certs\") pod \"8b647f08-2708-43ce-9e90-d5e57cbc34da\" (UID: \"8b647f08-2708-43ce-9e90-d5e57cbc34da\") " Sep 10 16:03:14.756491 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:03:14.756451 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b647f08-2708-43ce-9e90-d5e57cbc34da-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8b647f08-2708-43ce-9e90-d5e57cbc34da" (UID: "8b647f08-2708-43ce-9e90-d5e57cbc34da"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:03:14.756793 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:03:14.756769 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b647f08-2708-43ce-9e90-d5e57cbc34da-openshift-service-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:03:14.758670 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:03:14.758646 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b647f08-2708-43ce-9e90-d5e57cbc34da-kube-api-access-dm7dw" (OuterVolumeSpecName: "kube-api-access-dm7dw") pod "8b647f08-2708-43ce-9e90-d5e57cbc34da" (UID: "8b647f08-2708-43ce-9e90-d5e57cbc34da"). InnerVolumeSpecName "kube-api-access-dm7dw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:03:14.759109 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:03:14.759088 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b647f08-2708-43ce-9e90-d5e57cbc34da-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8b647f08-2708-43ce-9e90-d5e57cbc34da" (UID: "8b647f08-2708-43ce-9e90-d5e57cbc34da"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:03:14.759669 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:03:14.759641 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b647f08-2708-43ce-9e90-d5e57cbc34da-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8b647f08-2708-43ce-9e90-d5e57cbc34da" (UID: "8b647f08-2708-43ce-9e90-d5e57cbc34da"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:03:14.857961 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:03:14.857904 2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8b647f08-2708-43ce-9e90-d5e57cbc34da-metrics-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:03:14.857961 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:03:14.857925 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8b647f08-2708-43ce-9e90-d5e57cbc34da-maas-api-tls\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:03:14.857961 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:03:14.857933 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm7dw\" (UniqueName: \"kubernetes.io/projected/8b647f08-2708-43ce-9e90-d5e57cbc34da-kube-api-access-dm7dw\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:03:15.481295 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:03:15.481259 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69972456-69f59d7dcb-k4dkq" Sep 10 16:03:15.481719 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:03:15.481264 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-69972456-69f59d7dcb-k4dkq" event={"ID":"8b647f08-2708-43ce-9e90-d5e57cbc34da","Type":"ContainerDied","Data":"07e5e728b5048c6d8819aaccd9937d20741d5b2e52394795d8680116aec6773b"} Sep 10 16:03:15.481719 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:03:15.481447 2583 scope.go:117] "RemoveContainer" containerID="93f0ea4359445a35804e17a7bad8bab97bec23f50b37f2ea1a3408255e23d061" Sep 10 16:03:15.504563 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:03:15.504541 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-69972456-69f59d7dcb-k4dkq"] Sep 10 16:03:15.509174 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:03:15.509143 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-69972456-69f59d7dcb-k4dkq"] Sep 10 16:03:15.515543 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:03:15.515520 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b647f08-2708-43ce-9e90-d5e57cbc34da" path="/var/lib/kubelet/pods/8b647f08-2708-43ce-9e90-d5e57cbc34da/volumes" Sep 10 16:03:17.925116 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:03:17.925084 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:22.969633 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:03:22.969599 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:48.137453 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:03:48.137415 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:49.661224 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:03:49.661193 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:55.640665 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:03:55.640601 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:55.741642 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:03:55.741613 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:55.912694 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:03:55.912614 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:57.214232 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:03:57.214195 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:57.568201 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:03:57.568113 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:57.670576 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:03:57.670541 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:58.033657 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:03:58.033619 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:58.126085 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:03:58.126056 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:58.231500 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:03:58.231469 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:03:59.525634 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:03:59.525606 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:04:03.518495 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:03.518465 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:04:03.591800 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:03.591772 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:04:07.019902 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:07.019874 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:04:07.096262 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:07.096220 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:04:07.696463 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:07.696374 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:04:14.398873 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:14.398841 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:04:14.503968 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:14.503937 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:04:14.615931 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:14.615892 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:04:25.082683 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:25.082651 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:04:25.335070 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:25.334988 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:04:40.761433 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:40.761365 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:04:41.687177 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:41.687129 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:04:46.671681 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:46.671642 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:04:53.043605 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:53.043556 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:04:53.129267 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:53.129234 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:04:53.681223 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:53.681185 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:04:53.795836 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:53.795801 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:04:54.063719 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:54.063599 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:04:56.680728 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:56.680696 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:04:56.798180 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:56.798145 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:04:57.830902 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:57.830866 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ea0f68-576fcb85b5-nq2nx"] Sep 10 16:04:57.831370 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:57.831332 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b647f08-2708-43ce-9e90-d5e57cbc34da" containerName="maas-api" Sep 10 16:04:57.831370 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:57.831351 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b647f08-2708-43ce-9e90-d5e57cbc34da" containerName="maas-api" Sep 10 16:04:57.831522 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:57.831492 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b647f08-2708-43ce-9e90-d5e57cbc34da" containerName="maas-api" Sep 10 16:04:57.834822 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:57.834802 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ea0f68-576fcb85b5-nq2nx" Sep 10 16:04:57.837219 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:57.837197 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-ea0f68-serving-cert\"" Sep 10 16:04:57.853978 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:57.853955 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ea0f68-576fcb85b5-nq2nx"] Sep 10 16:04:57.908605 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:57.908583 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:04:57.947009 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:57.946973 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c51cdab-33b8-4291-b766-3af600c4ca57-maas-api-tls\") pod \"maas-api-e2e-api-a-ea0f68-576fcb85b5-nq2nx\" (UID: \"9c51cdab-33b8-4291-b766-3af600c4ca57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ea0f68-576fcb85b5-nq2nx" Sep 10 16:04:57.947140 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:57.947037 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c51cdab-33b8-4291-b766-3af600c4ca57-metrics-certs\") pod \"maas-api-e2e-api-a-ea0f68-576fcb85b5-nq2nx\" (UID: \"9c51cdab-33b8-4291-b766-3af600c4ca57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ea0f68-576fcb85b5-nq2nx" Sep 10 16:04:57.947140 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:57.947069 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c51cdab-33b8-4291-b766-3af600c4ca57-openshift-service-ca\") pod \"maas-api-e2e-api-a-ea0f68-576fcb85b5-nq2nx\" (UID: \"9c51cdab-33b8-4291-b766-3af600c4ca57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ea0f68-576fcb85b5-nq2nx" Sep 10 16:04:57.947228 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:57.947199 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs5gt\" (UniqueName: \"kubernetes.io/projected/9c51cdab-33b8-4291-b766-3af600c4ca57-kube-api-access-hs5gt\") pod \"maas-api-e2e-api-a-ea0f68-576fcb85b5-nq2nx\" (UID: \"9c51cdab-33b8-4291-b766-3af600c4ca57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ea0f68-576fcb85b5-nq2nx" Sep 10 16:04:58.048039 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:58.048011 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c51cdab-33b8-4291-b766-3af600c4ca57-metrics-certs\") pod \"maas-api-e2e-api-a-ea0f68-576fcb85b5-nq2nx\" (UID: \"9c51cdab-33b8-4291-b766-3af600c4ca57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ea0f68-576fcb85b5-nq2nx" Sep 10 16:04:58.048231 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:58.048045 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c51cdab-33b8-4291-b766-3af600c4ca57-openshift-service-ca\") pod \"maas-api-e2e-api-a-ea0f68-576fcb85b5-nq2nx\" (UID: \"9c51cdab-33b8-4291-b766-3af600c4ca57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ea0f68-576fcb85b5-nq2nx" Sep 10 16:04:58.048231 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:58.048114 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs5gt\" (UniqueName: \"kubernetes.io/projected/9c51cdab-33b8-4291-b766-3af600c4ca57-kube-api-access-hs5gt\") pod \"maas-api-e2e-api-a-ea0f68-576fcb85b5-nq2nx\" (UID: \"9c51cdab-33b8-4291-b766-3af600c4ca57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ea0f68-576fcb85b5-nq2nx" Sep 10 16:04:58.048231 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:58.048150 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c51cdab-33b8-4291-b766-3af600c4ca57-maas-api-tls\") pod \"maas-api-e2e-api-a-ea0f68-576fcb85b5-nq2nx\" (UID: \"9c51cdab-33b8-4291-b766-3af600c4ca57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ea0f68-576fcb85b5-nq2nx" Sep 10 16:04:58.048792 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:58.048766 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c51cdab-33b8-4291-b766-3af600c4ca57-openshift-service-ca\") pod \"maas-api-e2e-api-a-ea0f68-576fcb85b5-nq2nx\" (UID: \"9c51cdab-33b8-4291-b766-3af600c4ca57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ea0f68-576fcb85b5-nq2nx" Sep 10 16:04:58.050571 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:58.050540 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c51cdab-33b8-4291-b766-3af600c4ca57-metrics-certs\") pod \"maas-api-e2e-api-a-ea0f68-576fcb85b5-nq2nx\" (UID: \"9c51cdab-33b8-4291-b766-3af600c4ca57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ea0f68-576fcb85b5-nq2nx" Sep 10 16:04:58.050571 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:58.050560 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c51cdab-33b8-4291-b766-3af600c4ca57-maas-api-tls\") pod \"maas-api-e2e-api-a-ea0f68-576fcb85b5-nq2nx\" (UID: \"9c51cdab-33b8-4291-b766-3af600c4ca57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ea0f68-576fcb85b5-nq2nx" Sep 10 16:04:58.058647 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:58.058626 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs5gt\" (UniqueName: \"kubernetes.io/projected/9c51cdab-33b8-4291-b766-3af600c4ca57-kube-api-access-hs5gt\") pod \"maas-api-e2e-api-a-ea0f68-576fcb85b5-nq2nx\" (UID: \"9c51cdab-33b8-4291-b766-3af600c4ca57\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ea0f68-576fcb85b5-nq2nx" Sep 10 16:04:58.145948 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:58.145922 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ea0f68-576fcb85b5-nq2nx" Sep 10 16:04:58.427773 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:58.427738 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ea0f68-576fcb85b5-nq2nx"] Sep 10 16:04:58.428695 ip-10-0-139-107 kubenswrapper[2583]: W0910 16:04:58.428670 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c51cdab_33b8_4291_b766_3af600c4ca57.slice/crio-bce2949599b6913f5b1854a40627bf2692dded955f3dc1c8ead4c3ea358c4d2a WatchSource:0}: Error finding container bce2949599b6913f5b1854a40627bf2692dded955f3dc1c8ead4c3ea358c4d2a: Status 404 returned error can't find the container with id bce2949599b6913f5b1854a40627bf2692dded955f3dc1c8ead4c3ea358c4d2a Sep 10 16:04:58.430036 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:58.430017 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 16:04:58.536226 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:58.536193 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:04:58.663112 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:58.663081 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:04:58.724316 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:58.724245 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ea0f68-7dbd98844b-8nvxp"] Sep 10 16:04:58.729145 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:58.729129 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ea0f68-7dbd98844b-8nvxp" Sep 10 16:04:58.731546 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:58.731521 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-ea0f68\"" Sep 10 16:04:58.731662 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:58.731623 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-ea0f68-dockercfg-b4m76\"" Sep 10 16:04:58.741938 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:58.741918 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ea0f68-7dbd98844b-8nvxp"] Sep 10 16:04:58.856318 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:58.856291 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rscqd\" (UniqueName: \"kubernetes.io/projected/ebc052fa-1663-440d-9157-81b677f2a977-kube-api-access-rscqd\") pod \"payload-pre-processing-e2e-api-a-ea0f68-7dbd98844b-8nvxp\" (UID: \"ebc052fa-1663-440d-9157-81b677f2a977\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ea0f68-7dbd98844b-8nvxp" Sep 10 16:04:58.856609 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:58.856433 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebc052fa-1663-440d-9157-81b677f2a977-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ea0f68-7dbd98844b-8nvxp\" (UID: \"ebc052fa-1663-440d-9157-81b677f2a977\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ea0f68-7dbd98844b-8nvxp" Sep 10 16:04:58.957142 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:58.957110 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebc052fa-1663-440d-9157-81b677f2a977-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ea0f68-7dbd98844b-8nvxp\" (UID: \"ebc052fa-1663-440d-9157-81b677f2a977\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ea0f68-7dbd98844b-8nvxp" Sep 10 16:04:58.957293 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:58.957148 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rscqd\" (UniqueName: \"kubernetes.io/projected/ebc052fa-1663-440d-9157-81b677f2a977-kube-api-access-rscqd\") pod \"payload-pre-processing-e2e-api-a-ea0f68-7dbd98844b-8nvxp\" (UID: \"ebc052fa-1663-440d-9157-81b677f2a977\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ea0f68-7dbd98844b-8nvxp" Sep 10 16:04:58.957780 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:58.957759 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebc052fa-1663-440d-9157-81b677f2a977-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-ea0f68-7dbd98844b-8nvxp\" (UID: \"ebc052fa-1663-440d-9157-81b677f2a977\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ea0f68-7dbd98844b-8nvxp" Sep 10 16:04:58.966322 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:58.966294 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rscqd\" (UniqueName: \"kubernetes.io/projected/ebc052fa-1663-440d-9157-81b677f2a977-kube-api-access-rscqd\") pod \"payload-pre-processing-e2e-api-a-ea0f68-7dbd98844b-8nvxp\" (UID: \"ebc052fa-1663-440d-9157-81b677f2a977\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-ea0f68-7dbd98844b-8nvxp" Sep 10 16:04:59.040312 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:59.040245 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ea0f68-7dbd98844b-8nvxp" Sep 10 16:04:59.082349 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:59.082304 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ea0f68-576fcb85b5-nq2nx" event={"ID":"9c51cdab-33b8-4291-b766-3af600c4ca57","Type":"ContainerStarted","Data":"3b9ed0b2d2823b0c749e3993d0e55ed01ceb473ef401ab070c1f8b5b65643a4c"} Sep 10 16:04:59.082500 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:59.082403 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ea0f68-576fcb85b5-nq2nx" Sep 10 16:04:59.082500 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:59.082433 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ea0f68-576fcb85b5-nq2nx" event={"ID":"9c51cdab-33b8-4291-b766-3af600c4ca57","Type":"ContainerStarted","Data":"bce2949599b6913f5b1854a40627bf2692dded955f3dc1c8ead4c3ea358c4d2a"} Sep 10 16:04:59.103572 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:59.103279 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ea0f68-576fcb85b5-nq2nx" podStartSLOduration=1.7252419749999999 podStartE2EDuration="2.103257188s" totalImagesPullingTime="378.015213ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:04:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:04:59.102466367 +0000 UTC m=+1376.077726340" watchObservedRunningTime="2026-09-10 16:04:59.103257188 +0000 UTC m=+1376.078517141" Sep 10 16:04:59.326975 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:04:59.326474 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ea0f68-7dbd98844b-8nvxp"] Sep 10 16:05:00.090432 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:00.090394 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ea0f68-7dbd98844b-8nvxp" event={"ID":"ebc052fa-1663-440d-9157-81b677f2a977","Type":"ContainerStarted","Data":"675dbd255855dc33cbe09979769eb1259b7470dab3d2ef8e60c5faf1b38f3676"} Sep 10 16:05:00.090902 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:00.090582 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ea0f68-7dbd98844b-8nvxp" Sep 10 16:05:00.090902 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:00.090639 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ea0f68-7dbd98844b-8nvxp" event={"ID":"ebc052fa-1663-440d-9157-81b677f2a977","Type":"ContainerStarted","Data":"1ceb55e6933fed50dcf50a706e8cfe224bd2a4b85b258926ebbda9fb1162b6ae"} Sep 10 16:05:00.111045 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:00.110999 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ea0f68-7dbd98844b-8nvxp" podStartSLOduration=2.11097932 podStartE2EDuration="2.11097932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:04:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:05:00.1082348 +0000 UTC m=+1377.083494781" watchObservedRunningTime="2026-09-10 16:05:00.11097932 +0000 UTC m=+1377.086239273" Sep 10 16:05:05.095615 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:05.095584 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ea0f68-576fcb85b5-nq2nx" Sep 10 16:05:11.099175 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:11.099143 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ea0f68-7dbd98844b-8nvxp" Sep 10 16:05:11.668279 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:11.668246 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:05:11.959469 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:11.959352 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:05:21.558660 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:21.558627 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:05:22.047511 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:22.047469 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:05:22.369057 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:22.368979 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:05:22.587323 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:22.587293 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:05:25.323993 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:25.323959 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:05:25.766364 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:25.766332 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:05:25.867417 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:25.867367 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:05:26.944132 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:26.944100 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2ab381-7448998cfc-4jfcv"] Sep 10 16:05:26.948306 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:26.948283 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ab381-7448998cfc-4jfcv" Sep 10 16:05:26.950488 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:26.950467 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-2ab381-serving-cert\"" Sep 10 16:05:26.964017 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:26.963888 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2ab381-7448998cfc-4jfcv"] Sep 10 16:05:27.015257 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:27.015228 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:05:27.019430 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:27.019401 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fde38b6f-2df3-4638-8779-4a0673ddb504-maas-api-tls\") pod \"maas-api-e2e-api-b-2ab381-7448998cfc-4jfcv\" (UID: \"fde38b6f-2df3-4638-8779-4a0673ddb504\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ab381-7448998cfc-4jfcv" Sep 10 16:05:27.019549 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:27.019448 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgvlp\" (UniqueName: \"kubernetes.io/projected/fde38b6f-2df3-4638-8779-4a0673ddb504-kube-api-access-xgvlp\") pod \"maas-api-e2e-api-b-2ab381-7448998cfc-4jfcv\" (UID: \"fde38b6f-2df3-4638-8779-4a0673ddb504\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ab381-7448998cfc-4jfcv" Sep 10 16:05:27.019613 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:27.019593 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fde38b6f-2df3-4638-8779-4a0673ddb504-openshift-service-ca\") pod \"maas-api-e2e-api-b-2ab381-7448998cfc-4jfcv\" (UID: \"fde38b6f-2df3-4638-8779-4a0673ddb504\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ab381-7448998cfc-4jfcv" Sep 10 16:05:27.019653 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:27.019634 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fde38b6f-2df3-4638-8779-4a0673ddb504-metrics-certs\") pod \"maas-api-e2e-api-b-2ab381-7448998cfc-4jfcv\" (UID: \"fde38b6f-2df3-4638-8779-4a0673ddb504\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ab381-7448998cfc-4jfcv" Sep 10 16:05:27.108105 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:27.108067 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:05:27.120328 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:27.120299 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fde38b6f-2df3-4638-8779-4a0673ddb504-openshift-service-ca\") pod \"maas-api-e2e-api-b-2ab381-7448998cfc-4jfcv\" (UID: \"fde38b6f-2df3-4638-8779-4a0673ddb504\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ab381-7448998cfc-4jfcv" Sep 10 16:05:27.120518 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:27.120335 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fde38b6f-2df3-4638-8779-4a0673ddb504-metrics-certs\") pod \"maas-api-e2e-api-b-2ab381-7448998cfc-4jfcv\" (UID: \"fde38b6f-2df3-4638-8779-4a0673ddb504\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ab381-7448998cfc-4jfcv" Sep 10 16:05:27.120518 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:27.120423 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fde38b6f-2df3-4638-8779-4a0673ddb504-maas-api-tls\") pod \"maas-api-e2e-api-b-2ab381-7448998cfc-4jfcv\" (UID: \"fde38b6f-2df3-4638-8779-4a0673ddb504\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ab381-7448998cfc-4jfcv" Sep 10 16:05:27.120518 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:27.120479 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgvlp\" (UniqueName: \"kubernetes.io/projected/fde38b6f-2df3-4638-8779-4a0673ddb504-kube-api-access-xgvlp\") pod \"maas-api-e2e-api-b-2ab381-7448998cfc-4jfcv\" (UID: \"fde38b6f-2df3-4638-8779-4a0673ddb504\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ab381-7448998cfc-4jfcv" Sep 10 16:05:27.120739 ip-10-0-139-107 kubenswrapper[2583]: E0910 16:05:27.120572 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-2ab381-serving-cert: secret "maas-api-e2e-api-b-2ab381-serving-cert" not found Sep 10 16:05:27.120739 ip-10-0-139-107 kubenswrapper[2583]: E0910 16:05:27.120656 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fde38b6f-2df3-4638-8779-4a0673ddb504-maas-api-tls podName:fde38b6f-2df3-4638-8779-4a0673ddb504 nodeName:}" failed. No retries permitted until 2026-09-10 16:05:27.620637516 +0000 UTC m=+1404.595897466 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fde38b6f-2df3-4638-8779-4a0673ddb504-maas-api-tls") pod "maas-api-e2e-api-b-2ab381-7448998cfc-4jfcv" (UID: "fde38b6f-2df3-4638-8779-4a0673ddb504") : secret "maas-api-e2e-api-b-2ab381-serving-cert" not found Sep 10 16:05:27.121158 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:27.121137 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fde38b6f-2df3-4638-8779-4a0673ddb504-openshift-service-ca\") pod \"maas-api-e2e-api-b-2ab381-7448998cfc-4jfcv\" (UID: \"fde38b6f-2df3-4638-8779-4a0673ddb504\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ab381-7448998cfc-4jfcv" Sep 10 16:05:27.122872 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:27.122849 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fde38b6f-2df3-4638-8779-4a0673ddb504-metrics-certs\") pod \"maas-api-e2e-api-b-2ab381-7448998cfc-4jfcv\" (UID: \"fde38b6f-2df3-4638-8779-4a0673ddb504\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ab381-7448998cfc-4jfcv" Sep 10 16:05:27.130574 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:27.130539 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgvlp\" (UniqueName: \"kubernetes.io/projected/fde38b6f-2df3-4638-8779-4a0673ddb504-kube-api-access-xgvlp\") pod \"maas-api-e2e-api-b-2ab381-7448998cfc-4jfcv\" (UID: \"fde38b6f-2df3-4638-8779-4a0673ddb504\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ab381-7448998cfc-4jfcv" Sep 10 16:05:27.266471 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:27.266395 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:05:27.624792 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:27.624686 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fde38b6f-2df3-4638-8779-4a0673ddb504-maas-api-tls\") pod \"maas-api-e2e-api-b-2ab381-7448998cfc-4jfcv\" (UID: \"fde38b6f-2df3-4638-8779-4a0673ddb504\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ab381-7448998cfc-4jfcv" Sep 10 16:05:27.626966 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:27.626936 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fde38b6f-2df3-4638-8779-4a0673ddb504-maas-api-tls\") pod \"maas-api-e2e-api-b-2ab381-7448998cfc-4jfcv\" (UID: \"fde38b6f-2df3-4638-8779-4a0673ddb504\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ab381-7448998cfc-4jfcv" Sep 10 16:05:27.679334 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:27.679307 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:05:27.803321 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:27.803291 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:05:27.861771 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:27.861743 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ab381-7448998cfc-4jfcv" Sep 10 16:05:28.161761 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:28.161733 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2ab381-7448998cfc-4jfcv"] Sep 10 16:05:28.162584 ip-10-0-139-107 kubenswrapper[2583]: W0910 16:05:28.162557 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfde38b6f_2df3_4638_8779_4a0673ddb504.slice/crio-943fa0350938406c630f97ba6e26ba9b9850f786fee415ba51dc2f56777c51df WatchSource:0}: Error finding container 943fa0350938406c630f97ba6e26ba9b9850f786fee415ba51dc2f56777c51df: Status 404 returned error can't find the container with id 943fa0350938406c630f97ba6e26ba9b9850f786fee415ba51dc2f56777c51df Sep 10 16:05:28.264197 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:28.264166 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ab381-7448998cfc-4jfcv" event={"ID":"fde38b6f-2df3-4638-8779-4a0673ddb504","Type":"ContainerStarted","Data":"943fa0350938406c630f97ba6e26ba9b9850f786fee415ba51dc2f56777c51df"} Sep 10 16:05:29.271306 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:29.271223 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ab381-7448998cfc-4jfcv" event={"ID":"fde38b6f-2df3-4638-8779-4a0673ddb504","Type":"ContainerStarted","Data":"18f7c9b9663a5cfcb7bb5447def7910b3a47151d14d1da2a53beb7b053c6275e"} Sep 10 16:05:29.271306 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:29.271287 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ab381-7448998cfc-4jfcv" Sep 10 16:05:29.295527 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:29.295478 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ab381-7448998cfc-4jfcv" podStartSLOduration=2.476357496 podStartE2EDuration="3.29546257s" totalImagesPullingTime="819.105074ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:05:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:05:29.293034898 +0000 UTC m=+1406.268294865" watchObservedRunningTime="2026-09-10 16:05:29.29546257 +0000 UTC m=+1406.270722551" Sep 10 16:05:35.195073 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:35.195038 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:05:35.283444 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:35.283413 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ab381-7448998cfc-4jfcv" Sep 10 16:05:35.928323 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:35.928291 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:05:41.003522 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:41.003489 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:05:48.318301 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:48.318263 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:05:50.635340 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:50.635269 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:05:50.730973 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:50.730940 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:05:50.859394 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:50.859357 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:05:53.872220 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:05:53.872187 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:06:10.100373 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:10.100293 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ea0f68-576fcb85b5-nq2nx"] Sep 10 16:06:10.100866 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:10.100548 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ea0f68-576fcb85b5-nq2nx" podUID="9c51cdab-33b8-4291-b766-3af600c4ca57" containerName="maas-api" containerID="cri-o://3b9ed0b2d2823b0c749e3993d0e55ed01ceb473ef401ab070c1f8b5b65643a4c" gracePeriod=30 Sep 10 16:06:10.182923 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:10.182887 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ea0f68-7dbd98844b-8nvxp"] Sep 10 16:06:10.183159 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:10.183136 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ea0f68-7dbd98844b-8nvxp" podUID="ebc052fa-1663-440d-9157-81b677f2a977" containerName="payload-pre-processing" containerID="cri-o://675dbd255855dc33cbe09979769eb1259b7470dab3d2ef8e60c5faf1b38f3676" gracePeriod=30 Sep 10 16:06:10.321461 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:10.321425 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:06:10.434746 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:10.434720 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ea0f68-576fcb85b5-nq2nx" Sep 10 16:06:10.467626 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:10.467362 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ea0f68-7dbd98844b-8nvxp" Sep 10 16:06:10.512621 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:10.512596 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hs5gt\" (UniqueName: \"kubernetes.io/projected/9c51cdab-33b8-4291-b766-3af600c4ca57-kube-api-access-hs5gt\") pod \"9c51cdab-33b8-4291-b766-3af600c4ca57\" (UID: \"9c51cdab-33b8-4291-b766-3af600c4ca57\") " Sep 10 16:06:10.512786 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:10.512645 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c51cdab-33b8-4291-b766-3af600c4ca57-maas-api-tls\") pod \"9c51cdab-33b8-4291-b766-3af600c4ca57\" (UID: \"9c51cdab-33b8-4291-b766-3af600c4ca57\") " Sep 10 16:06:10.512786 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:10.512713 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c51cdab-33b8-4291-b766-3af600c4ca57-openshift-service-ca\") pod \"9c51cdab-33b8-4291-b766-3af600c4ca57\" (UID: \"9c51cdab-33b8-4291-b766-3af600c4ca57\") " Sep 10 16:06:10.512910 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:10.512797 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c51cdab-33b8-4291-b766-3af600c4ca57-metrics-certs\") pod \"9c51cdab-33b8-4291-b766-3af600c4ca57\" (UID: \"9c51cdab-33b8-4291-b766-3af600c4ca57\") " Sep 10 16:06:10.513164 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:10.513133 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c51cdab-33b8-4291-b766-3af600c4ca57-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9c51cdab-33b8-4291-b766-3af600c4ca57" (UID: "9c51cdab-33b8-4291-b766-3af600c4ca57"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:06:10.513164 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:10.513142 2583 generic.go:356] "Generic (PLEG): container finished" podID="9c51cdab-33b8-4291-b766-3af600c4ca57" containerID="3b9ed0b2d2823b0c749e3993d0e55ed01ceb473ef401ab070c1f8b5b65643a4c" exitCode=0 Sep 10 16:06:10.513343 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:10.513204 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ea0f68-576fcb85b5-nq2nx" Sep 10 16:06:10.513343 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:10.513246 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ea0f68-576fcb85b5-nq2nx" event={"ID":"9c51cdab-33b8-4291-b766-3af600c4ca57","Type":"ContainerDied","Data":"3b9ed0b2d2823b0c749e3993d0e55ed01ceb473ef401ab070c1f8b5b65643a4c"} Sep 10 16:06:10.513343 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:10.513287 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-ea0f68-576fcb85b5-nq2nx" event={"ID":"9c51cdab-33b8-4291-b766-3af600c4ca57","Type":"ContainerDied","Data":"bce2949599b6913f5b1854a40627bf2692dded955f3dc1c8ead4c3ea358c4d2a"} Sep 10 16:06:10.513343 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:10.513316 2583 scope.go:117] "RemoveContainer" containerID="3b9ed0b2d2823b0c749e3993d0e55ed01ceb473ef401ab070c1f8b5b65643a4c" Sep 10 16:06:10.514991 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:10.514966 2583 generic.go:356] "Generic (PLEG): container finished" podID="ebc052fa-1663-440d-9157-81b677f2a977" containerID="675dbd255855dc33cbe09979769eb1259b7470dab3d2ef8e60c5faf1b38f3676" exitCode=0 Sep 10 16:06:10.515106 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:10.515026 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ea0f68-7dbd98844b-8nvxp" Sep 10 16:06:10.515106 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:10.515053 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ea0f68-7dbd98844b-8nvxp" event={"ID":"ebc052fa-1663-440d-9157-81b677f2a977","Type":"ContainerDied","Data":"675dbd255855dc33cbe09979769eb1259b7470dab3d2ef8e60c5faf1b38f3676"} Sep 10 16:06:10.515237 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:10.515105 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-ea0f68-7dbd98844b-8nvxp" event={"ID":"ebc052fa-1663-440d-9157-81b677f2a977","Type":"ContainerDied","Data":"1ceb55e6933fed50dcf50a706e8cfe224bd2a4b85b258926ebbda9fb1162b6ae"} Sep 10 16:06:10.515577 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:10.515537 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c51cdab-33b8-4291-b766-3af600c4ca57-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9c51cdab-33b8-4291-b766-3af600c4ca57" (UID: "9c51cdab-33b8-4291-b766-3af600c4ca57"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:06:10.515690 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:10.515577 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c51cdab-33b8-4291-b766-3af600c4ca57-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9c51cdab-33b8-4291-b766-3af600c4ca57" (UID: "9c51cdab-33b8-4291-b766-3af600c4ca57"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:06:10.515690 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:10.515624 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c51cdab-33b8-4291-b766-3af600c4ca57-kube-api-access-hs5gt" (OuterVolumeSpecName: "kube-api-access-hs5gt") pod "9c51cdab-33b8-4291-b766-3af600c4ca57" (UID: "9c51cdab-33b8-4291-b766-3af600c4ca57"). InnerVolumeSpecName "kube-api-access-hs5gt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:06:10.522847 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:10.522828 2583 scope.go:117] "RemoveContainer" containerID="3b9ed0b2d2823b0c749e3993d0e55ed01ceb473ef401ab070c1f8b5b65643a4c" Sep 10 16:06:10.523121 ip-10-0-139-107 kubenswrapper[2583]: E0910 16:06:10.523100 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b9ed0b2d2823b0c749e3993d0e55ed01ceb473ef401ab070c1f8b5b65643a4c\": container with ID starting with 3b9ed0b2d2823b0c749e3993d0e55ed01ceb473ef401ab070c1f8b5b65643a4c not found: ID does not exist" containerID="3b9ed0b2d2823b0c749e3993d0e55ed01ceb473ef401ab070c1f8b5b65643a4c" Sep 10 16:06:10.523165 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:10.523128 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b9ed0b2d2823b0c749e3993d0e55ed01ceb473ef401ab070c1f8b5b65643a4c"} err="failed to get container status \"3b9ed0b2d2823b0c749e3993d0e55ed01ceb473ef401ab070c1f8b5b65643a4c\": rpc error: code = NotFound desc = could not find container \"3b9ed0b2d2823b0c749e3993d0e55ed01ceb473ef401ab070c1f8b5b65643a4c\": container with ID starting with 3b9ed0b2d2823b0c749e3993d0e55ed01ceb473ef401ab070c1f8b5b65643a4c not found: ID does not exist" Sep 10 16:06:10.523165 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:10.523145 2583 scope.go:117] "RemoveContainer" containerID="675dbd255855dc33cbe09979769eb1259b7470dab3d2ef8e60c5faf1b38f3676" Sep 10 16:06:10.533341 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:10.533319 2583 scope.go:117] "RemoveContainer" containerID="675dbd255855dc33cbe09979769eb1259b7470dab3d2ef8e60c5faf1b38f3676" Sep 10 16:06:10.533815 ip-10-0-139-107 kubenswrapper[2583]: E0910 16:06:10.533792 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"675dbd255855dc33cbe09979769eb1259b7470dab3d2ef8e60c5faf1b38f3676\": container with ID starting with 675dbd255855dc33cbe09979769eb1259b7470dab3d2ef8e60c5faf1b38f3676 not found: ID does not exist" containerID="675dbd255855dc33cbe09979769eb1259b7470dab3d2ef8e60c5faf1b38f3676" Sep 10 16:06:10.533884 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:10.533823 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"675dbd255855dc33cbe09979769eb1259b7470dab3d2ef8e60c5faf1b38f3676"} err="failed to get container status \"675dbd255855dc33cbe09979769eb1259b7470dab3d2ef8e60c5faf1b38f3676\": rpc error: code = NotFound desc = could not find container \"675dbd255855dc33cbe09979769eb1259b7470dab3d2ef8e60c5faf1b38f3676\": container with ID starting with 675dbd255855dc33cbe09979769eb1259b7470dab3d2ef8e60c5faf1b38f3676 not found: ID does not exist" Sep 10 16:06:10.613994 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:10.613967 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebc052fa-1663-440d-9157-81b677f2a977-plugins-config-volume\") pod \"ebc052fa-1663-440d-9157-81b677f2a977\" (UID: \"ebc052fa-1663-440d-9157-81b677f2a977\") " Sep 10 16:06:10.614126 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:10.614014 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rscqd\" (UniqueName: \"kubernetes.io/projected/ebc052fa-1663-440d-9157-81b677f2a977-kube-api-access-rscqd\") pod \"ebc052fa-1663-440d-9157-81b677f2a977\" (UID: \"ebc052fa-1663-440d-9157-81b677f2a977\") " Sep 10 16:06:10.614355 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:10.614323 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebc052fa-1663-440d-9157-81b677f2a977-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ebc052fa-1663-440d-9157-81b677f2a977" (UID: "ebc052fa-1663-440d-9157-81b677f2a977"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:06:10.614472 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:10.614435 2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebc052fa-1663-440d-9157-81b677f2a977-plugins-config-volume\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:06:10.614472 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:10.614455 2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c51cdab-33b8-4291-b766-3af600c4ca57-metrics-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:06:10.614472 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:10.614466 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs5gt\" (UniqueName: \"kubernetes.io/projected/9c51cdab-33b8-4291-b766-3af600c4ca57-kube-api-access-hs5gt\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:06:10.614654 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:10.614476 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c51cdab-33b8-4291-b766-3af600c4ca57-maas-api-tls\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:06:10.614654 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:10.614490 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c51cdab-33b8-4291-b766-3af600c4ca57-openshift-service-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:06:10.616078 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:10.616059 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebc052fa-1663-440d-9157-81b677f2a977-kube-api-access-rscqd" (OuterVolumeSpecName: "kube-api-access-rscqd") pod "ebc052fa-1663-440d-9157-81b677f2a977" (UID: "ebc052fa-1663-440d-9157-81b677f2a977"). InnerVolumeSpecName "kube-api-access-rscqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:06:10.715572 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:10.715512 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rscqd\" (UniqueName: \"kubernetes.io/projected/ebc052fa-1663-440d-9157-81b677f2a977-kube-api-access-rscqd\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:06:10.852137 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:10.852113 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ea0f68-576fcb85b5-nq2nx"] Sep 10 16:06:10.860687 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:10.860666 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-ea0f68-576fcb85b5-nq2nx"] Sep 10 16:06:10.876740 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:10.876716 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ea0f68-7dbd98844b-8nvxp"] Sep 10 16:06:10.882793 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:10.882773 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-ea0f68-7dbd98844b-8nvxp"] Sep 10 16:06:11.515260 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:11.515231 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c51cdab-33b8-4291-b766-3af600c4ca57" path="/var/lib/kubelet/pods/9c51cdab-33b8-4291-b766-3af600c4ca57/volumes" Sep 10 16:06:11.515658 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:11.515641 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebc052fa-1663-440d-9157-81b677f2a977" path="/var/lib/kubelet/pods/ebc052fa-1663-440d-9157-81b677f2a977/volumes" Sep 10 16:06:13.355895 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:13.355860 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:06:16.853622 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:16.853587 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:06:18.770545 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:18.770509 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:06:18.879116 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:18.879088 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:06:19.052669 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:19.052576 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:06:19.636328 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:19.636291 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:06:22.451961 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:22.451929 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:06:29.609466 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:29.609430 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:06:29.708801 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:29.708753 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:06:35.257982 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:35.257942 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2ab381-7448998cfc-4jfcv"] Sep 10 16:06:35.258575 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:35.258224 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ab381-7448998cfc-4jfcv" podUID="fde38b6f-2df3-4638-8779-4a0673ddb504" containerName="maas-api" containerID="cri-o://18f7c9b9663a5cfcb7bb5447def7910b3a47151d14d1da2a53beb7b053c6275e" gracePeriod=30 Sep 10 16:06:35.276808 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:35.276775 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:06:35.278824 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:35.278797 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ab381-7448998cfc-4jfcv" podUID="fde38b6f-2df3-4638-8779-4a0673ddb504" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.67:8443/health\": dial tcp 10.134.0.67:8443: connect: connection refused" Sep 10 16:06:35.378274 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:35.378162 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:06:35.478854 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:35.478823 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:06:35.517160 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:35.517100 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ab381-7448998cfc-4jfcv" Sep 10 16:06:35.546740 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:35.546716 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fde38b6f-2df3-4638-8779-4a0673ddb504-openshift-service-ca\") pod \"fde38b6f-2df3-4638-8779-4a0673ddb504\" (UID: \"fde38b6f-2df3-4638-8779-4a0673ddb504\") " Sep 10 16:06:35.546889 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:35.546849 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xgvlp\" (UniqueName: \"kubernetes.io/projected/fde38b6f-2df3-4638-8779-4a0673ddb504-kube-api-access-xgvlp\") pod \"fde38b6f-2df3-4638-8779-4a0673ddb504\" (UID: \"fde38b6f-2df3-4638-8779-4a0673ddb504\") " Sep 10 16:06:35.546962 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:35.546942 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fde38b6f-2df3-4638-8779-4a0673ddb504-maas-api-tls\") pod \"fde38b6f-2df3-4638-8779-4a0673ddb504\" (UID: \"fde38b6f-2df3-4638-8779-4a0673ddb504\") " Sep 10 16:06:35.547034 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:35.546994 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fde38b6f-2df3-4638-8779-4a0673ddb504-metrics-certs\") pod \"fde38b6f-2df3-4638-8779-4a0673ddb504\" (UID: \"fde38b6f-2df3-4638-8779-4a0673ddb504\") " Sep 10 16:06:35.547653 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:35.547618 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fde38b6f-2df3-4638-8779-4a0673ddb504-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fde38b6f-2df3-4638-8779-4a0673ddb504" (UID: "fde38b6f-2df3-4638-8779-4a0673ddb504"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:06:35.550073 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:35.550045 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fde38b6f-2df3-4638-8779-4a0673ddb504-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fde38b6f-2df3-4638-8779-4a0673ddb504" (UID: "fde38b6f-2df3-4638-8779-4a0673ddb504"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:06:35.550246 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:35.550220 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fde38b6f-2df3-4638-8779-4a0673ddb504-kube-api-access-xgvlp" (OuterVolumeSpecName: "kube-api-access-xgvlp") pod "fde38b6f-2df3-4638-8779-4a0673ddb504" (UID: "fde38b6f-2df3-4638-8779-4a0673ddb504"). InnerVolumeSpecName "kube-api-access-xgvlp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:06:35.550514 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:35.550489 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fde38b6f-2df3-4638-8779-4a0673ddb504-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fde38b6f-2df3-4638-8779-4a0673ddb504" (UID: "fde38b6f-2df3-4638-8779-4a0673ddb504"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:06:35.583146 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:35.583125 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:06:35.648242 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:35.648215 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgvlp\" (UniqueName: \"kubernetes.io/projected/fde38b6f-2df3-4638-8779-4a0673ddb504-kube-api-access-xgvlp\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:06:35.648242 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:35.648242 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fde38b6f-2df3-4638-8779-4a0673ddb504-maas-api-tls\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:06:35.648403 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:35.648256 2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fde38b6f-2df3-4638-8779-4a0673ddb504-metrics-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:06:35.648403 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:35.648267 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fde38b6f-2df3-4638-8779-4a0673ddb504-openshift-service-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:06:35.663902 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:35.663877 2583 generic.go:356] "Generic (PLEG): container finished" podID="fde38b6f-2df3-4638-8779-4a0673ddb504" containerID="18f7c9b9663a5cfcb7bb5447def7910b3a47151d14d1da2a53beb7b053c6275e" exitCode=0 Sep 10 16:06:35.664021 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:35.663922 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ab381-7448998cfc-4jfcv" Sep 10 16:06:35.664021 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:35.663962 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ab381-7448998cfc-4jfcv" event={"ID":"fde38b6f-2df3-4638-8779-4a0673ddb504","Type":"ContainerDied","Data":"18f7c9b9663a5cfcb7bb5447def7910b3a47151d14d1da2a53beb7b053c6275e"} Sep 10 16:06:35.664111 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:35.664019 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ab381-7448998cfc-4jfcv" event={"ID":"fde38b6f-2df3-4638-8779-4a0673ddb504","Type":"ContainerDied","Data":"943fa0350938406c630f97ba6e26ba9b9850f786fee415ba51dc2f56777c51df"} Sep 10 16:06:35.664111 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:35.664050 2583 scope.go:117] "RemoveContainer" containerID="18f7c9b9663a5cfcb7bb5447def7910b3a47151d14d1da2a53beb7b053c6275e" Sep 10 16:06:35.683522 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:35.683497 2583 scope.go:117] "RemoveContainer" containerID="18f7c9b9663a5cfcb7bb5447def7910b3a47151d14d1da2a53beb7b053c6275e" Sep 10 16:06:35.684663 ip-10-0-139-107 kubenswrapper[2583]: E0910 16:06:35.683897 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18f7c9b9663a5cfcb7bb5447def7910b3a47151d14d1da2a53beb7b053c6275e\": container with ID starting with 18f7c9b9663a5cfcb7bb5447def7910b3a47151d14d1da2a53beb7b053c6275e not found: ID does not exist" containerID="18f7c9b9663a5cfcb7bb5447def7910b3a47151d14d1da2a53beb7b053c6275e" Sep 10 16:06:35.684663 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:35.683938 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18f7c9b9663a5cfcb7bb5447def7910b3a47151d14d1da2a53beb7b053c6275e"} err="failed to get container status \"18f7c9b9663a5cfcb7bb5447def7910b3a47151d14d1da2a53beb7b053c6275e\": rpc error: code = NotFound desc = could not find container \"18f7c9b9663a5cfcb7bb5447def7910b3a47151d14d1da2a53beb7b053c6275e\": container with ID starting with 18f7c9b9663a5cfcb7bb5447def7910b3a47151d14d1da2a53beb7b053c6275e not found: ID does not exist" Sep 10 16:06:35.708650 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:35.708624 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2ab381-7448998cfc-4jfcv"] Sep 10 16:06:35.713510 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:35.713486 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2ab381-7448998cfc-4jfcv"] Sep 10 16:06:35.953155 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:35.953122 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:06:37.516061 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:37.516029 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fde38b6f-2df3-4638-8779-4a0673ddb504" path="/var/lib/kubelet/pods/fde38b6f-2df3-4638-8779-4a0673ddb504/volumes" Sep 10 16:06:42.732346 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:42.732314 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:06:42.829602 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:42.829561 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:06:43.363134 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:43.363107 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:06:43.583749 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:43.583722 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:06:43.940203 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:43.940174 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:06:44.969090 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:44.969060 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:06:53.969172 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:53.969133 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-247f70-5477449978-2rf7b"] Sep 10 16:06:53.969698 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:53.969492 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-247f70-5477449978-2rf7b" podUID="dc6c29de-8858-438f-b5b6-d9fe4b5be50d" containerName="maas-api" containerID="cri-o://56e6f59e1d5af33b51ef9b00cb09658de72778d098e0a3ba9f14436b6f47b93f" gracePeriod=30 Sep 10 16:06:54.001485 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:54.001458 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:06:54.038693 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:54.038656 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-247f70-59f96f4bd9-crq27"] Sep 10 16:06:54.038975 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:54.038915 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-247f70-59f96f4bd9-crq27" podUID="f016ad21-341e-4070-9608-6410eda330e2" containerName="payload-processing" containerID="cri-o://f6ecc4d1e40f6d6dbfdd6626976b1c0161617d368c7e9d1321766ad7dff17480" gracePeriod=30 Sep 10 16:06:54.293400 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:54.293358 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-247f70-5477449978-2rf7b" Sep 10 16:06:54.320936 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:54.320883 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-247f70-59f96f4bd9-crq27" Sep 10 16:06:54.394398 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:54.394338 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:06:54.414224 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:54.414204 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6n2wn\" (UniqueName: \"kubernetes.io/projected/dc6c29de-8858-438f-b5b6-d9fe4b5be50d-kube-api-access-6n2wn\") pod \"dc6c29de-8858-438f-b5b6-d9fe4b5be50d\" (UID: \"dc6c29de-8858-438f-b5b6-d9fe4b5be50d\") " Sep 10 16:06:54.414297 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:54.414238 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc6c29de-8858-438f-b5b6-d9fe4b5be50d-metrics-certs\") pod \"dc6c29de-8858-438f-b5b6-d9fe4b5be50d\" (UID: \"dc6c29de-8858-438f-b5b6-d9fe4b5be50d\") " Sep 10 16:06:54.414297 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:54.414276 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f016ad21-341e-4070-9608-6410eda330e2-plugins-config-volume\") pod \"f016ad21-341e-4070-9608-6410eda330e2\" (UID: \"f016ad21-341e-4070-9608-6410eda330e2\") " Sep 10 16:06:54.414367 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:54.414332 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc6c29de-8858-438f-b5b6-d9fe4b5be50d-maas-api-tls\") pod \"dc6c29de-8858-438f-b5b6-d9fe4b5be50d\" (UID: \"dc6c29de-8858-438f-b5b6-d9fe4b5be50d\") " Sep 10 16:06:54.414435 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:54.414408 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6c29de-8858-438f-b5b6-d9fe4b5be50d-openshift-service-ca\") pod \"dc6c29de-8858-438f-b5b6-d9fe4b5be50d\" (UID: \"dc6c29de-8858-438f-b5b6-d9fe4b5be50d\") " Sep 10 16:06:54.414564 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:54.414542 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnv69\" (UniqueName: \"kubernetes.io/projected/f016ad21-341e-4070-9608-6410eda330e2-kube-api-access-jnv69\") pod \"f016ad21-341e-4070-9608-6410eda330e2\" (UID: \"f016ad21-341e-4070-9608-6410eda330e2\") " Sep 10 16:06:54.414741 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:54.414707 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f016ad21-341e-4070-9608-6410eda330e2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f016ad21-341e-4070-9608-6410eda330e2" (UID: "f016ad21-341e-4070-9608-6410eda330e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:06:54.414809 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:54.414759 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc6c29de-8858-438f-b5b6-d9fe4b5be50d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "dc6c29de-8858-438f-b5b6-d9fe4b5be50d" (UID: "dc6c29de-8858-438f-b5b6-d9fe4b5be50d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:06:54.415010 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:54.414982 2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f016ad21-341e-4070-9608-6410eda330e2-plugins-config-volume\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:06:54.415089 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:54.415016 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc6c29de-8858-438f-b5b6-d9fe4b5be50d-openshift-service-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:06:54.416457 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:54.416430 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc6c29de-8858-438f-b5b6-d9fe4b5be50d-kube-api-access-6n2wn" (OuterVolumeSpecName: "kube-api-access-6n2wn") pod "dc6c29de-8858-438f-b5b6-d9fe4b5be50d" (UID: "dc6c29de-8858-438f-b5b6-d9fe4b5be50d"). InnerVolumeSpecName "kube-api-access-6n2wn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:06:54.416560 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:54.416488 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc6c29de-8858-438f-b5b6-d9fe4b5be50d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "dc6c29de-8858-438f-b5b6-d9fe4b5be50d" (UID: "dc6c29de-8858-438f-b5b6-d9fe4b5be50d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:06:54.416606 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:54.416587 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc6c29de-8858-438f-b5b6-d9fe4b5be50d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "dc6c29de-8858-438f-b5b6-d9fe4b5be50d" (UID: "dc6c29de-8858-438f-b5b6-d9fe4b5be50d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:06:54.416657 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:54.416635 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f016ad21-341e-4070-9608-6410eda330e2-kube-api-access-jnv69" (OuterVolumeSpecName: "kube-api-access-jnv69") pod "f016ad21-341e-4070-9608-6410eda330e2" (UID: "f016ad21-341e-4070-9608-6410eda330e2"). InnerVolumeSpecName "kube-api-access-jnv69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:06:54.497130 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:54.497059 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:06:54.516215 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:54.516188 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n2wn\" (UniqueName: \"kubernetes.io/projected/dc6c29de-8858-438f-b5b6-d9fe4b5be50d-kube-api-access-6n2wn\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:06:54.516215 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:54.516212 2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc6c29de-8858-438f-b5b6-d9fe4b5be50d-metrics-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:06:54.516335 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:54.516222 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc6c29de-8858-438f-b5b6-d9fe4b5be50d-maas-api-tls\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:06:54.516335 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:54.516230 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnv69\" (UniqueName: \"kubernetes.io/projected/f016ad21-341e-4070-9608-6410eda330e2-kube-api-access-jnv69\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:06:54.634265 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:54.634243 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:06:54.773987 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:54.773920 2583 generic.go:356] "Generic (PLEG): container finished" podID="f016ad21-341e-4070-9608-6410eda330e2" containerID="f6ecc4d1e40f6d6dbfdd6626976b1c0161617d368c7e9d1321766ad7dff17480" exitCode=0 Sep 10 16:06:54.773987 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:54.773959 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-247f70-59f96f4bd9-crq27" Sep 10 16:06:54.773987 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:54.773974 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-247f70-59f96f4bd9-crq27" event={"ID":"f016ad21-341e-4070-9608-6410eda330e2","Type":"ContainerDied","Data":"f6ecc4d1e40f6d6dbfdd6626976b1c0161617d368c7e9d1321766ad7dff17480"} Sep 10 16:06:54.774208 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:54.774027 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-247f70-59f96f4bd9-crq27" event={"ID":"f016ad21-341e-4070-9608-6410eda330e2","Type":"ContainerDied","Data":"da98455865a71d592d353d1e1606e6dd1d3cc4601328169aaee4671df131ffc5"} Sep 10 16:06:54.774208 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:54.774057 2583 scope.go:117] "RemoveContainer" containerID="f6ecc4d1e40f6d6dbfdd6626976b1c0161617d368c7e9d1321766ad7dff17480" Sep 10 16:06:54.775398 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:54.775364 2583 generic.go:356] "Generic (PLEG): container finished" podID="dc6c29de-8858-438f-b5b6-d9fe4b5be50d" containerID="56e6f59e1d5af33b51ef9b00cb09658de72778d098e0a3ba9f14436b6f47b93f" exitCode=0 Sep 10 16:06:54.775502 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:54.775404 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-247f70-5477449978-2rf7b" event={"ID":"dc6c29de-8858-438f-b5b6-d9fe4b5be50d","Type":"ContainerDied","Data":"56e6f59e1d5af33b51ef9b00cb09658de72778d098e0a3ba9f14436b6f47b93f"} Sep 10 16:06:54.775502 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:54.775432 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-247f70-5477449978-2rf7b" Sep 10 16:06:54.775502 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:54.775452 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-247f70-5477449978-2rf7b" event={"ID":"dc6c29de-8858-438f-b5b6-d9fe4b5be50d","Type":"ContainerDied","Data":"26ab4458f858fb6a6f4c7bf0cf7adb26433039b85b82707df166ac08092b840b"} Sep 10 16:06:54.784543 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:54.784527 2583 scope.go:117] "RemoveContainer" containerID="f6ecc4d1e40f6d6dbfdd6626976b1c0161617d368c7e9d1321766ad7dff17480" Sep 10 16:06:54.784834 ip-10-0-139-107 kubenswrapper[2583]: E0910 16:06:54.784815 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6ecc4d1e40f6d6dbfdd6626976b1c0161617d368c7e9d1321766ad7dff17480\": container with ID starting with f6ecc4d1e40f6d6dbfdd6626976b1c0161617d368c7e9d1321766ad7dff17480 not found: ID does not exist" containerID="f6ecc4d1e40f6d6dbfdd6626976b1c0161617d368c7e9d1321766ad7dff17480" Sep 10 16:06:54.784887 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:54.784840 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6ecc4d1e40f6d6dbfdd6626976b1c0161617d368c7e9d1321766ad7dff17480"} err="failed to get container status \"f6ecc4d1e40f6d6dbfdd6626976b1c0161617d368c7e9d1321766ad7dff17480\": rpc error: code = NotFound desc = could not find container \"f6ecc4d1e40f6d6dbfdd6626976b1c0161617d368c7e9d1321766ad7dff17480\": container with ID starting with f6ecc4d1e40f6d6dbfdd6626976b1c0161617d368c7e9d1321766ad7dff17480 not found: ID does not exist" Sep 10 16:06:54.784887 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:54.784857 2583 scope.go:117] "RemoveContainer" containerID="56e6f59e1d5af33b51ef9b00cb09658de72778d098e0a3ba9f14436b6f47b93f" Sep 10 16:06:54.793162 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:54.793146 2583 scope.go:117] "RemoveContainer" containerID="56e6f59e1d5af33b51ef9b00cb09658de72778d098e0a3ba9f14436b6f47b93f" Sep 10 16:06:54.793416 ip-10-0-139-107 kubenswrapper[2583]: E0910 16:06:54.793397 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56e6f59e1d5af33b51ef9b00cb09658de72778d098e0a3ba9f14436b6f47b93f\": container with ID starting with 56e6f59e1d5af33b51ef9b00cb09658de72778d098e0a3ba9f14436b6f47b93f not found: ID does not exist" containerID="56e6f59e1d5af33b51ef9b00cb09658de72778d098e0a3ba9f14436b6f47b93f" Sep 10 16:06:54.793489 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:54.793426 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56e6f59e1d5af33b51ef9b00cb09658de72778d098e0a3ba9f14436b6f47b93f"} err="failed to get container status \"56e6f59e1d5af33b51ef9b00cb09658de72778d098e0a3ba9f14436b6f47b93f\": rpc error: code = NotFound desc = could not find container \"56e6f59e1d5af33b51ef9b00cb09658de72778d098e0a3ba9f14436b6f47b93f\": container with ID starting with 56e6f59e1d5af33b51ef9b00cb09658de72778d098e0a3ba9f14436b6f47b93f not found: ID does not exist" Sep 10 16:06:54.802959 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:54.802936 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-247f70-5477449978-2rf7b"] Sep 10 16:06:54.808357 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:54.808334 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-247f70-5477449978-2rf7b"] Sep 10 16:06:54.819959 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:54.819940 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-247f70-59f96f4bd9-crq27"] Sep 10 16:06:54.822114 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:54.822095 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-247f70-59f96f4bd9-crq27"] Sep 10 16:06:55.515277 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:55.515242 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc6c29de-8858-438f-b5b6-d9fe4b5be50d" path="/var/lib/kubelet/pods/dc6c29de-8858-438f-b5b6-d9fe4b5be50d/volumes" Sep 10 16:06:55.515670 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:06:55.515620 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f016ad21-341e-4070-9608-6410eda330e2" path="/var/lib/kubelet/pods/f016ad21-341e-4070-9608-6410eda330e2/volumes" Sep 10 16:07:01.356706 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:07:01.356674 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:01.439767 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:07:01.439711 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:01.778671 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:07:01.778632 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:01.879474 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:07:01.879445 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:03.685948 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:07:03.685896 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:11.905145 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:07:11.905106 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:12.603640 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:07:12.603605 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-298d9b-6f8fc6656c-qk477"] Sep 10 16:07:12.603847 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:07:12.603811 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-298d9b-6f8fc6656c-qk477" podUID="0976bab9-5429-4077-a7ea-a33edc5175ca" containerName="maas-api" containerID="cri-o://07611b1834e77f6414bd7fead9b71f7e3141294b6ab6c8bce64e0a33617ae88f" gracePeriod=30 Sep 10 16:07:12.674880 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:07:12.674850 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:12.770805 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:07:12.770760 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:12.850165 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:07:12.850142 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-298d9b-6f8fc6656c-qk477" Sep 10 16:07:12.874325 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:07:12.874259 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:12.885053 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:07:12.885023 2583 generic.go:356] "Generic (PLEG): container finished" podID="0976bab9-5429-4077-a7ea-a33edc5175ca" containerID="07611b1834e77f6414bd7fead9b71f7e3141294b6ab6c8bce64e0a33617ae88f" exitCode=0 Sep 10 16:07:12.885172 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:07:12.885070 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-298d9b-6f8fc6656c-qk477" Sep 10 16:07:12.885172 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:07:12.885111 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-298d9b-6f8fc6656c-qk477" event={"ID":"0976bab9-5429-4077-a7ea-a33edc5175ca","Type":"ContainerDied","Data":"07611b1834e77f6414bd7fead9b71f7e3141294b6ab6c8bce64e0a33617ae88f"} Sep 10 16:07:12.885251 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:07:12.885184 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-298d9b-6f8fc6656c-qk477" event={"ID":"0976bab9-5429-4077-a7ea-a33edc5175ca","Type":"ContainerDied","Data":"da26defebce510fca6313c0d4291619bb8dc8ed2e4644366614f1944c43c6ac6"} Sep 10 16:07:12.885251 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:07:12.885216 2583 scope.go:117] "RemoveContainer" containerID="07611b1834e77f6414bd7fead9b71f7e3141294b6ab6c8bce64e0a33617ae88f" Sep 10 16:07:12.897123 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:07:12.897103 2583 scope.go:117] "RemoveContainer" containerID="07611b1834e77f6414bd7fead9b71f7e3141294b6ab6c8bce64e0a33617ae88f" Sep 10 16:07:12.897463 ip-10-0-139-107 kubenswrapper[2583]: E0910 16:07:12.897365 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07611b1834e77f6414bd7fead9b71f7e3141294b6ab6c8bce64e0a33617ae88f\": container with ID starting with 07611b1834e77f6414bd7fead9b71f7e3141294b6ab6c8bce64e0a33617ae88f not found: ID does not exist" containerID="07611b1834e77f6414bd7fead9b71f7e3141294b6ab6c8bce64e0a33617ae88f" Sep 10 16:07:12.897463 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:07:12.897405 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07611b1834e77f6414bd7fead9b71f7e3141294b6ab6c8bce64e0a33617ae88f"} err="failed to get container status \"07611b1834e77f6414bd7fead9b71f7e3141294b6ab6c8bce64e0a33617ae88f\": rpc error: code = NotFound desc = could not find container \"07611b1834e77f6414bd7fead9b71f7e3141294b6ab6c8bce64e0a33617ae88f\": container with ID starting with 07611b1834e77f6414bd7fead9b71f7e3141294b6ab6c8bce64e0a33617ae88f not found: ID does not exist" Sep 10 16:07:12.969068 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:07:12.969043 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0976bab9-5429-4077-a7ea-a33edc5175ca-metrics-certs\") pod \"0976bab9-5429-4077-a7ea-a33edc5175ca\" (UID: \"0976bab9-5429-4077-a7ea-a33edc5175ca\") " Sep 10 16:07:12.969523 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:07:12.969084 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0976bab9-5429-4077-a7ea-a33edc5175ca-maas-api-tls\") pod \"0976bab9-5429-4077-a7ea-a33edc5175ca\" (UID: \"0976bab9-5429-4077-a7ea-a33edc5175ca\") " Sep 10 16:07:12.969523 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:07:12.969147 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0976bab9-5429-4077-a7ea-a33edc5175ca-openshift-service-ca\") pod \"0976bab9-5429-4077-a7ea-a33edc5175ca\" (UID: \"0976bab9-5429-4077-a7ea-a33edc5175ca\") " Sep 10 16:07:12.969523 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:07:12.969240 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-859f9\" (UniqueName: \"kubernetes.io/projected/0976bab9-5429-4077-a7ea-a33edc5175ca-kube-api-access-859f9\") pod \"0976bab9-5429-4077-a7ea-a33edc5175ca\" (UID: \"0976bab9-5429-4077-a7ea-a33edc5175ca\") " Sep 10 16:07:12.969523 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:07:12.969448 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:12.969754 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:07:12.969583 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0976bab9-5429-4077-a7ea-a33edc5175ca-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0976bab9-5429-4077-a7ea-a33edc5175ca" (UID: "0976bab9-5429-4077-a7ea-a33edc5175ca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:07:12.971205 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:07:12.971181 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0976bab9-5429-4077-a7ea-a33edc5175ca-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0976bab9-5429-4077-a7ea-a33edc5175ca" (UID: "0976bab9-5429-4077-a7ea-a33edc5175ca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:07:12.971706 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:07:12.971684 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0976bab9-5429-4077-a7ea-a33edc5175ca-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0976bab9-5429-4077-a7ea-a33edc5175ca" (UID: "0976bab9-5429-4077-a7ea-a33edc5175ca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:07:12.971706 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:07:12.971699 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0976bab9-5429-4077-a7ea-a33edc5175ca-kube-api-access-859f9" (OuterVolumeSpecName: "kube-api-access-859f9") pod "0976bab9-5429-4077-a7ea-a33edc5175ca" (UID: "0976bab9-5429-4077-a7ea-a33edc5175ca"). InnerVolumeSpecName "kube-api-access-859f9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:07:13.070592 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:07:13.070560 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-859f9\" (UniqueName: \"kubernetes.io/projected/0976bab9-5429-4077-a7ea-a33edc5175ca-kube-api-access-859f9\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:07:13.070592 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:07:13.070587 2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0976bab9-5429-4077-a7ea-a33edc5175ca-metrics-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:07:13.070592 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:07:13.070597 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0976bab9-5429-4077-a7ea-a33edc5175ca-maas-api-tls\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:07:13.070813 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:07:13.070605 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0976bab9-5429-4077-a7ea-a33edc5175ca-openshift-service-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:07:13.215944 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:07:13.215914 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-298d9b-6f8fc6656c-qk477"] Sep 10 16:07:13.222033 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:07:13.221995 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-298d9b-6f8fc6656c-qk477"] Sep 10 16:07:13.271329 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:07:13.271301 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:13.516354 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:07:13.516274 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0976bab9-5429-4077-a7ea-a33edc5175ca" path="/var/lib/kubelet/pods/0976bab9-5429-4077-a7ea-a33edc5175ca/volumes" Sep 10 16:07:20.174602 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:07:20.174567 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:20.275998 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:07:20.275954 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:07:40.395320 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:07:40.395283 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:08:27.028927 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:27.028857 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:08:27.973095 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:27.973060 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:08:28.403890 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:28.403860 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:08:28.721814 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:28.721732 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:08:28.899452 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:28.899421 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:08:29.832713 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:29.832679 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:08:31.989469 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:31.989437 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:08:32.071689 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:32.071660 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:08:33.306846 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:33.306817 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc8c1e-5d988dffbd-w96tp"] Sep 10 16:08:33.307310 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:33.307234 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc6c29de-8858-438f-b5b6-d9fe4b5be50d" containerName="maas-api" Sep 10 16:08:33.307310 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:33.307246 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc6c29de-8858-438f-b5b6-d9fe4b5be50d" containerName="maas-api" Sep 10 16:08:33.307310 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:33.307254 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0976bab9-5429-4077-a7ea-a33edc5175ca" containerName="maas-api" Sep 10 16:08:33.307310 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:33.307260 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="0976bab9-5429-4077-a7ea-a33edc5175ca" containerName="maas-api" Sep 10 16:08:33.307310 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:33.307270 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c51cdab-33b8-4291-b766-3af600c4ca57" containerName="maas-api" Sep 10 16:08:33.307310 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:33.307276 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c51cdab-33b8-4291-b766-3af600c4ca57" containerName="maas-api" Sep 10 16:08:33.307310 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:33.307290 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebc052fa-1663-440d-9157-81b677f2a977" containerName="payload-pre-processing" Sep 10 16:08:33.307310 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:33.307295 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebc052fa-1663-440d-9157-81b677f2a977" containerName="payload-pre-processing" Sep 10 16:08:33.307310 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:33.307304 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fde38b6f-2df3-4638-8779-4a0673ddb504" containerName="maas-api" Sep 10 16:08:33.307310 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:33.307308 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fde38b6f-2df3-4638-8779-4a0673ddb504" containerName="maas-api" Sep 10 16:08:33.307310 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:33.307316 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f016ad21-341e-4070-9608-6410eda330e2" containerName="payload-processing" Sep 10 16:08:33.307714 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:33.307321 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="f016ad21-341e-4070-9608-6410eda330e2" containerName="payload-processing" Sep 10 16:08:33.307714 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:33.307411 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebc052fa-1663-440d-9157-81b677f2a977" containerName="payload-pre-processing" Sep 10 16:08:33.307714 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:33.307419 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fde38b6f-2df3-4638-8779-4a0673ddb504" containerName="maas-api" Sep 10 16:08:33.307714 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:33.307427 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c51cdab-33b8-4291-b766-3af600c4ca57" containerName="maas-api" Sep 10 16:08:33.307714 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:33.307435 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="f016ad21-341e-4070-9608-6410eda330e2" containerName="payload-processing" Sep 10 16:08:33.307714 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:33.307441 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="0976bab9-5429-4077-a7ea-a33edc5175ca" containerName="maas-api" Sep 10 16:08:33.307714 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:33.307448 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc6c29de-8858-438f-b5b6-d9fe4b5be50d" containerName="maas-api" Sep 10 16:08:33.310654 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:33.310638 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc8c1e-5d988dffbd-w96tp" Sep 10 16:08:33.313722 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:33.313702 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-fc8c1e-serving-cert\"" Sep 10 16:08:33.323754 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:33.323730 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc8c1e-5d988dffbd-w96tp"] Sep 10 16:08:33.348076 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:33.348051 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:08:33.372944 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:33.372916 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caeb4219-f316-40dc-8ca9-5542b58a99d2-metrics-certs\") pod \"maas-api-e2e-ipp-a-fc8c1e-5d988dffbd-w96tp\" (UID: \"caeb4219-f316-40dc-8ca9-5542b58a99d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc8c1e-5d988dffbd-w96tp" Sep 10 16:08:33.373063 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:33.372958 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caeb4219-f316-40dc-8ca9-5542b58a99d2-maas-api-tls\") pod \"maas-api-e2e-ipp-a-fc8c1e-5d988dffbd-w96tp\" (UID: \"caeb4219-f316-40dc-8ca9-5542b58a99d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc8c1e-5d988dffbd-w96tp" Sep 10 16:08:33.373063 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:33.373044 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7rkx\" (UniqueName: \"kubernetes.io/projected/caeb4219-f316-40dc-8ca9-5542b58a99d2-kube-api-access-b7rkx\") pod \"maas-api-e2e-ipp-a-fc8c1e-5d988dffbd-w96tp\" (UID: \"caeb4219-f316-40dc-8ca9-5542b58a99d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc8c1e-5d988dffbd-w96tp" Sep 10 16:08:33.373143 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:33.373096 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/caeb4219-f316-40dc-8ca9-5542b58a99d2-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-fc8c1e-5d988dffbd-w96tp\" (UID: \"caeb4219-f316-40dc-8ca9-5542b58a99d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc8c1e-5d988dffbd-w96tp" Sep 10 16:08:33.474377 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:33.474340 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7rkx\" (UniqueName: \"kubernetes.io/projected/caeb4219-f316-40dc-8ca9-5542b58a99d2-kube-api-access-b7rkx\") pod \"maas-api-e2e-ipp-a-fc8c1e-5d988dffbd-w96tp\" (UID: \"caeb4219-f316-40dc-8ca9-5542b58a99d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc8c1e-5d988dffbd-w96tp" Sep 10 16:08:33.474560 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:33.474435 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/caeb4219-f316-40dc-8ca9-5542b58a99d2-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-fc8c1e-5d988dffbd-w96tp\" (UID: \"caeb4219-f316-40dc-8ca9-5542b58a99d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc8c1e-5d988dffbd-w96tp" Sep 10 16:08:33.474560 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:33.474534 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caeb4219-f316-40dc-8ca9-5542b58a99d2-metrics-certs\") pod \"maas-api-e2e-ipp-a-fc8c1e-5d988dffbd-w96tp\" (UID: \"caeb4219-f316-40dc-8ca9-5542b58a99d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc8c1e-5d988dffbd-w96tp" Sep 10 16:08:33.474687 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:33.474564 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caeb4219-f316-40dc-8ca9-5542b58a99d2-maas-api-tls\") pod \"maas-api-e2e-ipp-a-fc8c1e-5d988dffbd-w96tp\" (UID: \"caeb4219-f316-40dc-8ca9-5542b58a99d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc8c1e-5d988dffbd-w96tp" Sep 10 16:08:33.474745 ip-10-0-139-107 kubenswrapper[2583]: E0910 16:08:33.474699 2583 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc8c1e-serving-cert: secret "maas-api-e2e-ipp-a-fc8c1e-serving-cert" not found Sep 10 16:08:33.474798 ip-10-0-139-107 kubenswrapper[2583]: E0910 16:08:33.474774 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/caeb4219-f316-40dc-8ca9-5542b58a99d2-maas-api-tls podName:caeb4219-f316-40dc-8ca9-5542b58a99d2 nodeName:}" failed. No retries permitted until 2026-09-10 16:08:33.974750251 +0000 UTC m=+1590.950010185 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/caeb4219-f316-40dc-8ca9-5542b58a99d2-maas-api-tls") pod "maas-api-e2e-ipp-a-fc8c1e-5d988dffbd-w96tp" (UID: "caeb4219-f316-40dc-8ca9-5542b58a99d2") : secret "maas-api-e2e-ipp-a-fc8c1e-serving-cert" not found Sep 10 16:08:33.475135 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:33.475111 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/caeb4219-f316-40dc-8ca9-5542b58a99d2-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-fc8c1e-5d988dffbd-w96tp\" (UID: \"caeb4219-f316-40dc-8ca9-5542b58a99d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc8c1e-5d988dffbd-w96tp" Sep 10 16:08:33.476899 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:33.476880 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caeb4219-f316-40dc-8ca9-5542b58a99d2-metrics-certs\") pod \"maas-api-e2e-ipp-a-fc8c1e-5d988dffbd-w96tp\" (UID: \"caeb4219-f316-40dc-8ca9-5542b58a99d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc8c1e-5d988dffbd-w96tp" Sep 10 16:08:33.491195 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:33.491165 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7rkx\" (UniqueName: \"kubernetes.io/projected/caeb4219-f316-40dc-8ca9-5542b58a99d2-kube-api-access-b7rkx\") pod \"maas-api-e2e-ipp-a-fc8c1e-5d988dffbd-w96tp\" (UID: \"caeb4219-f316-40dc-8ca9-5542b58a99d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc8c1e-5d988dffbd-w96tp" Sep 10 16:08:33.492451 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:33.492427 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:08:33.927973 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:33.927942 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:08:33.979653 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:33.979627 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caeb4219-f316-40dc-8ca9-5542b58a99d2-maas-api-tls\") pod \"maas-api-e2e-ipp-a-fc8c1e-5d988dffbd-w96tp\" (UID: \"caeb4219-f316-40dc-8ca9-5542b58a99d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc8c1e-5d988dffbd-w96tp" Sep 10 16:08:33.982226 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:33.982196 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caeb4219-f316-40dc-8ca9-5542b58a99d2-maas-api-tls\") pod \"maas-api-e2e-ipp-a-fc8c1e-5d988dffbd-w96tp\" (UID: \"caeb4219-f316-40dc-8ca9-5542b58a99d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc8c1e-5d988dffbd-w96tp" Sep 10 16:08:34.031712 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:34.031686 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:08:34.137800 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:34.137749 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fc8c1e-59bdb57d4b-8w54b"] Sep 10 16:08:34.138661 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:34.138636 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:08:34.141777 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:34.141762 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc8c1e-59bdb57d4b-8w54b" Sep 10 16:08:34.143892 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:34.143875 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-fc8c1e-dockercfg-bbb2s\"" Sep 10 16:08:34.144155 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:34.144142 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-fc8c1e\"" Sep 10 16:08:34.155770 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:34.155748 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fc8c1e-59bdb57d4b-8w54b"] Sep 10 16:08:34.207318 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:34.203727 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fc8c1e-6f8488ff89-6cp8m"] Sep 10 16:08:34.208716 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:34.208667 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc8c1e-6f8488ff89-6cp8m" Sep 10 16:08:34.219511 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:34.219480 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fc8c1e-6f8488ff89-6cp8m"] Sep 10 16:08:34.220762 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:34.220741 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc8c1e-5d988dffbd-w96tp" Sep 10 16:08:34.283369 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:34.283155 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/801d4560-01db-4a3b-8794-41b3779141b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fc8c1e-59bdb57d4b-8w54b\" (UID: \"801d4560-01db-4a3b-8794-41b3779141b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc8c1e-59bdb57d4b-8w54b" Sep 10 16:08:34.283369 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:34.283301 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0c29a25-b303-4838-8a4e-0b1564b14e06-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fc8c1e-6f8488ff89-6cp8m\" (UID: \"e0c29a25-b303-4838-8a4e-0b1564b14e06\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fc8c1e-6f8488ff89-6cp8m" Sep 10 16:08:34.283369 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:34.283342 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p4cr\" (UniqueName: \"kubernetes.io/projected/801d4560-01db-4a3b-8794-41b3779141b6-kube-api-access-6p4cr\") pod \"payload-pre-processing-e2e-ipp-a-fc8c1e-59bdb57d4b-8w54b\" (UID: \"801d4560-01db-4a3b-8794-41b3779141b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc8c1e-59bdb57d4b-8w54b" Sep 10 16:08:34.283627 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:34.283430 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9vqm\" (UniqueName: \"kubernetes.io/projected/e0c29a25-b303-4838-8a4e-0b1564b14e06-kube-api-access-q9vqm\") pod \"payload-processing-e2e-ipp-a-fc8c1e-6f8488ff89-6cp8m\" (UID: \"e0c29a25-b303-4838-8a4e-0b1564b14e06\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fc8c1e-6f8488ff89-6cp8m" Sep 10 16:08:34.384466 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:34.384436 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0c29a25-b303-4838-8a4e-0b1564b14e06-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fc8c1e-6f8488ff89-6cp8m\" (UID: \"e0c29a25-b303-4838-8a4e-0b1564b14e06\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fc8c1e-6f8488ff89-6cp8m" Sep 10 16:08:34.384809 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:34.384477 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p4cr\" (UniqueName: \"kubernetes.io/projected/801d4560-01db-4a3b-8794-41b3779141b6-kube-api-access-6p4cr\") pod \"payload-pre-processing-e2e-ipp-a-fc8c1e-59bdb57d4b-8w54b\" (UID: \"801d4560-01db-4a3b-8794-41b3779141b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc8c1e-59bdb57d4b-8w54b" Sep 10 16:08:34.384809 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:34.384522 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9vqm\" (UniqueName: \"kubernetes.io/projected/e0c29a25-b303-4838-8a4e-0b1564b14e06-kube-api-access-q9vqm\") pod \"payload-processing-e2e-ipp-a-fc8c1e-6f8488ff89-6cp8m\" (UID: \"e0c29a25-b303-4838-8a4e-0b1564b14e06\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fc8c1e-6f8488ff89-6cp8m" Sep 10 16:08:34.384809 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:34.384580 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/801d4560-01db-4a3b-8794-41b3779141b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fc8c1e-59bdb57d4b-8w54b\" (UID: \"801d4560-01db-4a3b-8794-41b3779141b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc8c1e-59bdb57d4b-8w54b" Sep 10 16:08:34.385174 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:34.385154 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0c29a25-b303-4838-8a4e-0b1564b14e06-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fc8c1e-6f8488ff89-6cp8m\" (UID: \"e0c29a25-b303-4838-8a4e-0b1564b14e06\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fc8c1e-6f8488ff89-6cp8m" Sep 10 16:08:34.385276 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:34.385258 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/801d4560-01db-4a3b-8794-41b3779141b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-fc8c1e-59bdb57d4b-8w54b\" (UID: \"801d4560-01db-4a3b-8794-41b3779141b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc8c1e-59bdb57d4b-8w54b" Sep 10 16:08:34.399675 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:34.399644 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9vqm\" (UniqueName: \"kubernetes.io/projected/e0c29a25-b303-4838-8a4e-0b1564b14e06-kube-api-access-q9vqm\") pod \"payload-processing-e2e-ipp-a-fc8c1e-6f8488ff89-6cp8m\" (UID: \"e0c29a25-b303-4838-8a4e-0b1564b14e06\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fc8c1e-6f8488ff89-6cp8m" Sep 10 16:08:34.399791 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:34.399648 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p4cr\" (UniqueName: \"kubernetes.io/projected/801d4560-01db-4a3b-8794-41b3779141b6-kube-api-access-6p4cr\") pod \"payload-pre-processing-e2e-ipp-a-fc8c1e-59bdb57d4b-8w54b\" (UID: \"801d4560-01db-4a3b-8794-41b3779141b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc8c1e-59bdb57d4b-8w54b" Sep 10 16:08:34.451815 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:34.451790 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc8c1e-59bdb57d4b-8w54b" Sep 10 16:08:34.508515 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:34.508485 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc8c1e-5d988dffbd-w96tp"] Sep 10 16:08:34.509772 ip-10-0-139-107 kubenswrapper[2583]: W0910 16:08:34.509736 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaeb4219_f316_40dc_8ca9_5542b58a99d2.slice/crio-4b774bb4193713c0c5bf04eb363fb08a85d6364cc97abd5221d416c1fd706684 WatchSource:0}: Error finding container 4b774bb4193713c0c5bf04eb363fb08a85d6364cc97abd5221d416c1fd706684: Status 404 returned error can't find the container with id 4b774bb4193713c0c5bf04eb363fb08a85d6364cc97abd5221d416c1fd706684 Sep 10 16:08:34.521721 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:34.521696 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc8c1e-6f8488ff89-6cp8m" Sep 10 16:08:34.736322 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:34.736298 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fc8c1e-59bdb57d4b-8w54b"] Sep 10 16:08:34.737317 ip-10-0-139-107 kubenswrapper[2583]: W0910 16:08:34.737296 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod801d4560_01db_4a3b_8794_41b3779141b6.slice/crio-f45e94ea0d7dfbeb4f72dd1bb232802c77962871f1e76b7a3f8b90d0999a10cb WatchSource:0}: Error finding container f45e94ea0d7dfbeb4f72dd1bb232802c77962871f1e76b7a3f8b90d0999a10cb: Status 404 returned error can't find the container with id f45e94ea0d7dfbeb4f72dd1bb232802c77962871f1e76b7a3f8b90d0999a10cb Sep 10 16:08:34.802182 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:34.802158 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fc8c1e-6f8488ff89-6cp8m"] Sep 10 16:08:34.803503 ip-10-0-139-107 kubenswrapper[2583]: W0910 16:08:34.803481 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0c29a25_b303_4838_8a4e_0b1564b14e06.slice/crio-40e1ea7d6ba99960222195590d34ddb705f81aac5a49f1a3b222609e0cbdc293 WatchSource:0}: Error finding container 40e1ea7d6ba99960222195590d34ddb705f81aac5a49f1a3b222609e0cbdc293: Status 404 returned error can't find the container with id 40e1ea7d6ba99960222195590d34ddb705f81aac5a49f1a3b222609e0cbdc293 Sep 10 16:08:35.350417 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:35.350300 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc8c1e-6f8488ff89-6cp8m" event={"ID":"e0c29a25-b303-4838-8a4e-0b1564b14e06","Type":"ContainerStarted","Data":"5facf13e7fb9e21d4c7cc3f2fc3e4e0455f54b49d198fe28a888fbc9779104af"} Sep 10 16:08:35.350417 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:35.350368 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc8c1e-6f8488ff89-6cp8m" Sep 10 16:08:35.350417 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:35.350414 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc8c1e-6f8488ff89-6cp8m" event={"ID":"e0c29a25-b303-4838-8a4e-0b1564b14e06","Type":"ContainerStarted","Data":"40e1ea7d6ba99960222195590d34ddb705f81aac5a49f1a3b222609e0cbdc293"} Sep 10 16:08:35.351960 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:35.351935 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc8c1e-59bdb57d4b-8w54b" event={"ID":"801d4560-01db-4a3b-8794-41b3779141b6","Type":"ContainerStarted","Data":"ffa23524ad2676970f4970f526dbd4b628237d09e2697ce785b2a0e9169d07be"} Sep 10 16:08:35.352089 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:35.351981 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc8c1e-59bdb57d4b-8w54b" Sep 10 16:08:35.352089 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:35.352002 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc8c1e-59bdb57d4b-8w54b" event={"ID":"801d4560-01db-4a3b-8794-41b3779141b6","Type":"ContainerStarted","Data":"f45e94ea0d7dfbeb4f72dd1bb232802c77962871f1e76b7a3f8b90d0999a10cb"} Sep 10 16:08:35.353507 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:35.353485 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc8c1e-5d988dffbd-w96tp" event={"ID":"caeb4219-f316-40dc-8ca9-5542b58a99d2","Type":"ContainerStarted","Data":"8fadaa333cac9d78934118adbfa59ee0b070c56f91eeeb16c3a07dee8ea641a5"} Sep 10 16:08:35.353603 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:35.353525 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc8c1e-5d988dffbd-w96tp" Sep 10 16:08:35.353603 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:35.353540 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc8c1e-5d988dffbd-w96tp" event={"ID":"caeb4219-f316-40dc-8ca9-5542b58a99d2","Type":"ContainerStarted","Data":"4b774bb4193713c0c5bf04eb363fb08a85d6364cc97abd5221d416c1fd706684"} Sep 10 16:08:35.383677 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:35.383632 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc8c1e-6f8488ff89-6cp8m" podStartSLOduration=1.3836197700000001 podStartE2EDuration="1.38361977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:08:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:08:35.381567681 +0000 UTC m=+1592.356827638" watchObservedRunningTime="2026-09-10 16:08:35.38361977 +0000 UTC m=+1592.358879722" Sep 10 16:08:35.410082 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:35.410040 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc8c1e-5d988dffbd-w96tp" podStartSLOduration=1.9318224320000001 podStartE2EDuration="2.410028382s" totalImagesPullingTime="478.20595ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:08:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:08:35.407045311 +0000 UTC m=+1592.382305263" watchObservedRunningTime="2026-09-10 16:08:35.410028382 +0000 UTC m=+1592.385288380" Sep 10 16:08:35.432770 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:35.432733 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc8c1e-59bdb57d4b-8w54b" podStartSLOduration=1.432721322 podStartE2EDuration="1.432721322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:08:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:08:35.430225219 +0000 UTC m=+1592.405485174" watchObservedRunningTime="2026-09-10 16:08:35.432721322 +0000 UTC m=+1592.407981275" Sep 10 16:08:41.368989 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:41.368959 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc8c1e-5d988dffbd-w96tp" Sep 10 16:08:46.366008 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:46.365979 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc8c1e-59bdb57d4b-8w54b" Sep 10 16:08:46.366424 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:46.366047 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc8c1e-6f8488ff89-6cp8m" Sep 10 16:08:56.959976 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:56.959885 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:08:57.633868 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:57.633832 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:08:57.977824 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:57.977725 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:08:58.189095 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:08:58.189063 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:09:01.181877 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:09:01.181834 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:09:01.278478 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:09:01.278449 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:09:02.516701 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:09:02.516631 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:09:02.595366 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:09:02.595334 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:09:02.697748 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:09:02.697722 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:09:03.078278 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:09:03.078246 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:09:03.171330 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:09:03.171302 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:09:26.161271 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:09:26.161233 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:09:45.909304 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:09:45.909269 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:09:55.217314 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:09:55.217281 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:10:22.241985 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:22.241956 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-fc8c1e-6f8488ff89-6cp8m_e0c29a25-b303-4838-8a4e-0b1564b14e06/payload-processing/0.log" Sep 10 16:10:23.021936 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:23.021901 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc"] Sep 10 16:10:23.025338 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:23.025315 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" Sep 10 16:10:23.027722 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:23.027699 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-65ee1d-openshift-default-dockercfg-bm7cs\"" Sep 10 16:10:23.037875 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:23.037852 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc"] Sep 10 16:10:23.135795 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:23.135756 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-istio-podinfo\") pod \"e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc\" (UID: \"fcddf00e-6d3f-4c5c-99a9-2f50745879c2\") " pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" Sep 10 16:10:23.136127 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:23.135828 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-workload-certs\") pod \"e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc\" (UID: \"fcddf00e-6d3f-4c5c-99a9-2f50745879c2\") " pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" Sep 10 16:10:23.136127 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:23.135943 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-workload-socket\") pod \"e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc\" (UID: \"fcddf00e-6d3f-4c5c-99a9-2f50745879c2\") " pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" Sep 10 16:10:23.136127 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:23.135994 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-istiod-ca-cert\") pod \"e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc\" (UID: \"fcddf00e-6d3f-4c5c-99a9-2f50745879c2\") " pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" Sep 10 16:10:23.136127 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:23.136104 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-istio-envoy\") pod \"e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc\" (UID: \"fcddf00e-6d3f-4c5c-99a9-2f50745879c2\") " pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" Sep 10 16:10:23.136352 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:23.136213 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-istio-token\") pod \"e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc\" (UID: \"fcddf00e-6d3f-4c5c-99a9-2f50745879c2\") " pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" Sep 10 16:10:23.136352 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:23.136252 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-credential-socket\") pod \"e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc\" (UID: \"fcddf00e-6d3f-4c5c-99a9-2f50745879c2\") " pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" Sep 10 16:10:23.136352 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:23.136276 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-istio-data\") pod \"e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc\" (UID: \"fcddf00e-6d3f-4c5c-99a9-2f50745879c2\") " pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" Sep 10 16:10:23.136352 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:23.136347 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slrcs\" (UniqueName: \"kubernetes.io/projected/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-kube-api-access-slrcs\") pod \"e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc\" (UID: \"fcddf00e-6d3f-4c5c-99a9-2f50745879c2\") " pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" Sep 10 16:10:23.237443 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:23.237409 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-workload-socket\") pod \"e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc\" (UID: \"fcddf00e-6d3f-4c5c-99a9-2f50745879c2\") " pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" Sep 10 16:10:23.237631 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:23.237466 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-istiod-ca-cert\") pod \"e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc\" (UID: \"fcddf00e-6d3f-4c5c-99a9-2f50745879c2\") " pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" Sep 10 16:10:23.237631 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:23.237494 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-istio-envoy\") pod \"e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc\" (UID: \"fcddf00e-6d3f-4c5c-99a9-2f50745879c2\") " pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" Sep 10 16:10:23.237631 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:23.237521 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-istio-token\") pod \"e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc\" (UID: \"fcddf00e-6d3f-4c5c-99a9-2f50745879c2\") " pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" Sep 10 16:10:23.237631 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:23.237552 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-credential-socket\") pod \"e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc\" (UID: \"fcddf00e-6d3f-4c5c-99a9-2f50745879c2\") " pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" Sep 10 16:10:23.237631 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:23.237583 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-istio-data\") pod \"e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc\" (UID: \"fcddf00e-6d3f-4c5c-99a9-2f50745879c2\") " pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" Sep 10 16:10:23.237631 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:23.237619 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slrcs\" (UniqueName: \"kubernetes.io/projected/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-kube-api-access-slrcs\") pod \"e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc\" (UID: \"fcddf00e-6d3f-4c5c-99a9-2f50745879c2\") " pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" Sep 10 16:10:23.237969 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:23.237698 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-istio-podinfo\") pod \"e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc\" (UID: \"fcddf00e-6d3f-4c5c-99a9-2f50745879c2\") " pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" Sep 10 16:10:23.237969 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:23.237742 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-workload-certs\") pod \"e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc\" (UID: \"fcddf00e-6d3f-4c5c-99a9-2f50745879c2\") " pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" Sep 10 16:10:23.237969 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:23.237795 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-workload-socket\") pod \"e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc\" (UID: \"fcddf00e-6d3f-4c5c-99a9-2f50745879c2\") " pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" Sep 10 16:10:23.237969 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:23.237932 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-credential-socket\") pod \"e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc\" (UID: \"fcddf00e-6d3f-4c5c-99a9-2f50745879c2\") " pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" Sep 10 16:10:23.238193 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:23.238054 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-workload-certs\") pod \"e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc\" (UID: \"fcddf00e-6d3f-4c5c-99a9-2f50745879c2\") " pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" Sep 10 16:10:23.238193 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:23.238146 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-istio-data\") pod \"e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc\" (UID: \"fcddf00e-6d3f-4c5c-99a9-2f50745879c2\") " pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" Sep 10 16:10:23.238304 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:23.238272 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-istiod-ca-cert\") pod \"e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc\" (UID: \"fcddf00e-6d3f-4c5c-99a9-2f50745879c2\") " pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" Sep 10 16:10:23.240058 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:23.240036 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-istio-envoy\") pod \"e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc\" (UID: \"fcddf00e-6d3f-4c5c-99a9-2f50745879c2\") " pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" Sep 10 16:10:23.240178 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:23.240159 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-istio-podinfo\") pod \"e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc\" (UID: \"fcddf00e-6d3f-4c5c-99a9-2f50745879c2\") " pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" Sep 10 16:10:23.246341 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:23.246319 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slrcs\" (UniqueName: \"kubernetes.io/projected/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-kube-api-access-slrcs\") pod \"e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc\" (UID: \"fcddf00e-6d3f-4c5c-99a9-2f50745879c2\") " pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" Sep 10 16:10:23.246732 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:23.246453 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-istio-token\") pod \"e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc\" (UID: \"fcddf00e-6d3f-4c5c-99a9-2f50745879c2\") " pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" Sep 10 16:10:23.337085 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:23.337029 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" Sep 10 16:10:23.623504 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:23.623476 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc"] Sep 10 16:10:23.624983 ip-10-0-139-107 kubenswrapper[2583]: W0910 16:10:23.624957 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcddf00e_6d3f_4c5c_99a9_2f50745879c2.slice/crio-285167ede105402b4ec003c5bbb724958f47b54be2eb3fdfec4bbef1c3729ff6 WatchSource:0}: Error finding container 285167ede105402b4ec003c5bbb724958f47b54be2eb3fdfec4bbef1c3729ff6: Status 404 returned error can't find the container with id 285167ede105402b4ec003c5bbb724958f47b54be2eb3fdfec4bbef1c3729ff6 Sep 10 16:10:23.627187 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:23.627154 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 16:10:23.627286 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:23.627227 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 16:10:23.627286 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:23.627275 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 16:10:23.972962 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:23.972926 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" event={"ID":"fcddf00e-6d3f-4c5c-99a9-2f50745879c2","Type":"ContainerStarted","Data":"5ebbe9a6a1ec366e10b1f76f0c2dc60c5767775da8de466c184d34cfbdadacc0"} Sep 10 16:10:23.973170 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:23.972982 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" event={"ID":"fcddf00e-6d3f-4c5c-99a9-2f50745879c2","Type":"ContainerStarted","Data":"285167ede105402b4ec003c5bbb724958f47b54be2eb3fdfec4bbef1c3729ff6"} Sep 10 16:10:23.995962 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:23.995901 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" podStartSLOduration=1.995883096 podStartE2EDuration="1.995883096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:10:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:10:23.994570326 +0000 UTC m=+1700.969830280" watchObservedRunningTime="2026-09-10 16:10:23.995883096 +0000 UTC m=+1700.971143061" Sep 10 16:10:24.068371 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:24.068341 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:10:24.168921 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:24.168884 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:10:24.337615 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:24.337548 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" Sep 10 16:10:24.343146 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:24.343123 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" Sep 10 16:10:24.979335 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:24.979302 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" Sep 10 16:10:24.980102 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:24.980085 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" Sep 10 16:10:26.023631 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:26.023556 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:10:26.108826 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:26.108791 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:10:27.183711 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:27.183678 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:10:27.278332 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:27.278304 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:10:27.413015 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:27.412986 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:10:27.757992 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:27.757960 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:10:27.855179 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:27.855151 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:10:27.880777 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:27.880752 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-65ee1d-6d88fd7c4c-qc4b4"] Sep 10 16:10:27.884625 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:27.884606 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-65ee1d-6d88fd7c4c-qc4b4" Sep 10 16:10:27.886757 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:27.886733 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-65ee1d-dockercfg-nsxc9\"" Sep 10 16:10:27.886876 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:27.886791 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-65ee1d\"" Sep 10 16:10:27.904998 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:27.904973 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-65ee1d-6d88fd7c4c-qc4b4"] Sep 10 16:10:27.983497 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:27.983459 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acbbca2a-492b-416a-b1ba-73c757caef1c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-65ee1d-6d88fd7c4c-qc4b4\" (UID: \"acbbca2a-492b-416a-b1ba-73c757caef1c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-65ee1d-6d88fd7c4c-qc4b4" Sep 10 16:10:27.983680 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:27.983514 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh4fr\" (UniqueName: \"kubernetes.io/projected/acbbca2a-492b-416a-b1ba-73c757caef1c-kube-api-access-lh4fr\") pod \"payload-processing-e2e-ipp-cleanup-65ee1d-6d88fd7c4c-qc4b4\" (UID: \"acbbca2a-492b-416a-b1ba-73c757caef1c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-65ee1d-6d88fd7c4c-qc4b4" Sep 10 16:10:28.084825 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:28.084749 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acbbca2a-492b-416a-b1ba-73c757caef1c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-65ee1d-6d88fd7c4c-qc4b4\" (UID: \"acbbca2a-492b-416a-b1ba-73c757caef1c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-65ee1d-6d88fd7c4c-qc4b4" Sep 10 16:10:28.084825 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:28.084803 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh4fr\" (UniqueName: \"kubernetes.io/projected/acbbca2a-492b-416a-b1ba-73c757caef1c-kube-api-access-lh4fr\") pod \"payload-processing-e2e-ipp-cleanup-65ee1d-6d88fd7c4c-qc4b4\" (UID: \"acbbca2a-492b-416a-b1ba-73c757caef1c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-65ee1d-6d88fd7c4c-qc4b4" Sep 10 16:10:28.085451 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:28.085431 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acbbca2a-492b-416a-b1ba-73c757caef1c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-65ee1d-6d88fd7c4c-qc4b4\" (UID: \"acbbca2a-492b-416a-b1ba-73c757caef1c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-65ee1d-6d88fd7c4c-qc4b4" Sep 10 16:10:28.094143 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:28.094114 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh4fr\" (UniqueName: \"kubernetes.io/projected/acbbca2a-492b-416a-b1ba-73c757caef1c-kube-api-access-lh4fr\") pod \"payload-processing-e2e-ipp-cleanup-65ee1d-6d88fd7c4c-qc4b4\" (UID: \"acbbca2a-492b-416a-b1ba-73c757caef1c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-65ee1d-6d88fd7c4c-qc4b4" Sep 10 16:10:28.195931 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:28.195902 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-65ee1d-6d88fd7c4c-qc4b4" Sep 10 16:10:28.471724 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:28.471693 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-65ee1d-6d88fd7c4c-qc4b4"] Sep 10 16:10:28.472646 ip-10-0-139-107 kubenswrapper[2583]: W0910 16:10:28.472619 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacbbca2a_492b_416a_b1ba_73c757caef1c.slice/crio-7e148846f2680eb18877ed43c641fc394baa68bcfd33f0b229b1ce2e7d149b18 WatchSource:0}: Error finding container 7e148846f2680eb18877ed43c641fc394baa68bcfd33f0b229b1ce2e7d149b18: Status 404 returned error can't find the container with id 7e148846f2680eb18877ed43c641fc394baa68bcfd33f0b229b1ce2e7d149b18 Sep 10 16:10:29.008085 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:29.008046 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-65ee1d-6d88fd7c4c-qc4b4" event={"ID":"acbbca2a-492b-416a-b1ba-73c757caef1c","Type":"ContainerStarted","Data":"359c1b24e4384de2d5abb2fe9237d358d480e3597d94998aee32f2f121b40f6e"} Sep 10 16:10:29.008263 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:29.008131 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-65ee1d-6d88fd7c4c-qc4b4" event={"ID":"acbbca2a-492b-416a-b1ba-73c757caef1c","Type":"ContainerStarted","Data":"7e148846f2680eb18877ed43c641fc394baa68bcfd33f0b229b1ce2e7d149b18"} Sep 10 16:10:29.008263 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:29.008160 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-65ee1d-6d88fd7c4c-qc4b4" Sep 10 16:10:29.029448 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:29.029394 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-65ee1d-6d88fd7c4c-qc4b4" podStartSLOduration=2.029367051 podStartE2EDuration="2.029367051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:10:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:10:29.026872585 +0000 UTC m=+1706.002132557" watchObservedRunningTime="2026-09-10 16:10:29.029367051 +0000 UTC m=+1706.004627037" Sep 10 16:10:40.018247 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:40.018216 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-65ee1d-6d88fd7c4c-qc4b4" Sep 10 16:10:46.128953 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:46.128918 2583 generic.go:356] "Generic (PLEG): container finished" podID="fcddf00e-6d3f-4c5c-99a9-2f50745879c2" containerID="5ebbe9a6a1ec366e10b1f76f0c2dc60c5767775da8de466c184d34cfbdadacc0" exitCode=0 Sep 10 16:10:46.129404 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:46.128977 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" event={"ID":"fcddf00e-6d3f-4c5c-99a9-2f50745879c2","Type":"ContainerDied","Data":"5ebbe9a6a1ec366e10b1f76f0c2dc60c5767775da8de466c184d34cfbdadacc0"} Sep 10 16:10:46.129404 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:46.129361 2583 scope.go:117] "RemoveContainer" containerID="5ebbe9a6a1ec366e10b1f76f0c2dc60c5767775da8de466c184d34cfbdadacc0" Sep 10 16:10:46.130443 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:46.130417 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 16:10:46.130511 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:46.130481 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 16:10:46.130549 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:46.130512 2583 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 10 16:10:47.136838 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:47.136803 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" event={"ID":"fcddf00e-6d3f-4c5c-99a9-2f50745879c2","Type":"ContainerStarted","Data":"6492b9d05573e00e4ad262d599b4296daa2a570510a415fbb985e29ea257797a"} Sep 10 16:10:47.337544 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:47.337508 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" Sep 10 16:10:47.337743 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:47.337720 2583 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Sep 10 16:10:47.337827 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:47.337788 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" podUID="fcddf00e-6d3f-4c5c-99a9-2f50745879c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Sep 10 16:10:48.338104 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:48.338073 2583 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Sep 10 16:10:48.338497 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:48.338130 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" podUID="fcddf00e-6d3f-4c5c-99a9-2f50745879c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Sep 10 16:10:49.337963 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:49.337928 2583 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Sep 10 16:10:49.338138 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:49.337992 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" podUID="fcddf00e-6d3f-4c5c-99a9-2f50745879c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Sep 10 16:10:50.337971 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:50.337930 2583 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Sep 10 16:10:50.338145 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:50.337998 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" podUID="fcddf00e-6d3f-4c5c-99a9-2f50745879c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Sep 10 16:10:50.957313 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:50.957279 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:10:51.287573 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:51.287455 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:10:51.338278 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:51.338248 2583 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Sep 10 16:10:51.338649 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:51.338307 2583 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" podUID="fcddf00e-6d3f-4c5c-99a9-2f50745879c2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Sep 10 16:10:52.061756 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:52.061719 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:10:52.372354 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:52.372262 2583 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" Sep 10 16:10:52.372876 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:52.372853 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" Sep 10 16:10:52.374197 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:52.374174 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" Sep 10 16:10:57.807000 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:57.806970 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:10:57.838447 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:57.838413 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-65ee1d-6d88fd7c4c-qc4b4"] Sep 10 16:10:57.838718 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:57.838663 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-65ee1d-6d88fd7c4c-qc4b4" podUID="acbbca2a-492b-416a-b1ba-73c757caef1c" containerName="payload-processing" containerID="cri-o://359c1b24e4384de2d5abb2fe9237d358d480e3597d94998aee32f2f121b40f6e" gracePeriod=30 Sep 10 16:10:57.900481 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:57.900452 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:10:57.999267 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:57.999232 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:10:58.072794 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:58.072752 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:10:58.094207 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:58.094180 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-65ee1d-6d88fd7c4c-qc4b4" Sep 10 16:10:58.182789 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:58.182740 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acbbca2a-492b-416a-b1ba-73c757caef1c-plugins-config-volume\") pod \"acbbca2a-492b-416a-b1ba-73c757caef1c\" (UID: \"acbbca2a-492b-416a-b1ba-73c757caef1c\") " Sep 10 16:10:58.182976 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:58.182956 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lh4fr\" (UniqueName: \"kubernetes.io/projected/acbbca2a-492b-416a-b1ba-73c757caef1c-kube-api-access-lh4fr\") pod \"acbbca2a-492b-416a-b1ba-73c757caef1c\" (UID: \"acbbca2a-492b-416a-b1ba-73c757caef1c\") " Sep 10 16:10:58.183227 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:58.183203 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acbbca2a-492b-416a-b1ba-73c757caef1c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "acbbca2a-492b-416a-b1ba-73c757caef1c" (UID: "acbbca2a-492b-416a-b1ba-73c757caef1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:10:58.183334 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:58.183322 2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acbbca2a-492b-416a-b1ba-73c757caef1c-plugins-config-volume\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:10:58.185185 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:58.185159 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acbbca2a-492b-416a-b1ba-73c757caef1c-kube-api-access-lh4fr" (OuterVolumeSpecName: "kube-api-access-lh4fr") pod "acbbca2a-492b-416a-b1ba-73c757caef1c" (UID: "acbbca2a-492b-416a-b1ba-73c757caef1c"). InnerVolumeSpecName "kube-api-access-lh4fr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:10:58.203927 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:58.203895 2583 generic.go:356] "Generic (PLEG): container finished" podID="acbbca2a-492b-416a-b1ba-73c757caef1c" containerID="359c1b24e4384de2d5abb2fe9237d358d480e3597d94998aee32f2f121b40f6e" exitCode=0 Sep 10 16:10:58.204067 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:58.203959 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-65ee1d-6d88fd7c4c-qc4b4" Sep 10 16:10:58.204067 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:58.203978 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-65ee1d-6d88fd7c4c-qc4b4" event={"ID":"acbbca2a-492b-416a-b1ba-73c757caef1c","Type":"ContainerDied","Data":"359c1b24e4384de2d5abb2fe9237d358d480e3597d94998aee32f2f121b40f6e"} Sep 10 16:10:58.204067 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:58.204031 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-65ee1d-6d88fd7c4c-qc4b4" event={"ID":"acbbca2a-492b-416a-b1ba-73c757caef1c","Type":"ContainerDied","Data":"7e148846f2680eb18877ed43c641fc394baa68bcfd33f0b229b1ce2e7d149b18"} Sep 10 16:10:58.204067 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:58.204053 2583 scope.go:117] "RemoveContainer" containerID="359c1b24e4384de2d5abb2fe9237d358d480e3597d94998aee32f2f121b40f6e" Sep 10 16:10:58.213873 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:58.213848 2583 scope.go:117] "RemoveContainer" containerID="359c1b24e4384de2d5abb2fe9237d358d480e3597d94998aee32f2f121b40f6e" Sep 10 16:10:58.214145 ip-10-0-139-107 kubenswrapper[2583]: E0910 16:10:58.214126 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"359c1b24e4384de2d5abb2fe9237d358d480e3597d94998aee32f2f121b40f6e\": container with ID starting with 359c1b24e4384de2d5abb2fe9237d358d480e3597d94998aee32f2f121b40f6e not found: ID does not exist" containerID="359c1b24e4384de2d5abb2fe9237d358d480e3597d94998aee32f2f121b40f6e" Sep 10 16:10:58.214189 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:58.214153 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"359c1b24e4384de2d5abb2fe9237d358d480e3597d94998aee32f2f121b40f6e"} err="failed to get container status \"359c1b24e4384de2d5abb2fe9237d358d480e3597d94998aee32f2f121b40f6e\": rpc error: code = NotFound desc = could not find container \"359c1b24e4384de2d5abb2fe9237d358d480e3597d94998aee32f2f121b40f6e\": container with ID starting with 359c1b24e4384de2d5abb2fe9237d358d480e3597d94998aee32f2f121b40f6e not found: ID does not exist" Sep 10 16:10:58.229356 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:58.229329 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-65ee1d-6d88fd7c4c-qc4b4"] Sep 10 16:10:58.236745 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:58.236722 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-65ee1d-6d88fd7c4c-qc4b4"] Sep 10 16:10:58.283892 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:58.283869 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh4fr\" (UniqueName: \"kubernetes.io/projected/acbbca2a-492b-416a-b1ba-73c757caef1c-kube-api-access-lh4fr\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:10:58.375722 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:58.375666 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:10:59.515914 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:10:59.515869 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acbbca2a-492b-416a-b1ba-73c757caef1c" path="/var/lib/kubelet/pods/acbbca2a-492b-416a-b1ba-73c757caef1c/volumes" Sep 10 16:11:04.633332 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:04.633300 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:04.735312 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:04.735273 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:04.873240 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:04.873206 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc"] Sep 10 16:11:04.873473 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:04.873448 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" podUID="fcddf00e-6d3f-4c5c-99a9-2f50745879c2" containerName="istio-proxy" containerID="cri-o://6492b9d05573e00e4ad262d599b4296daa2a570510a415fbb985e29ea257797a" gracePeriod=30 Sep 10 16:11:04.896180 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:04.896093 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:07.375168 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:07.375133 2583 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Sep 10 16:11:07.375621 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:07.375191 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" podUID="fcddf00e-6d3f-4c5c-99a9-2f50745879c2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 10 16:11:08.612580 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:08.612547 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:10.132461 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:10.132438 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" Sep 10 16:11:10.274593 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:10.274518 2583 generic.go:356] "Generic (PLEG): container finished" podID="fcddf00e-6d3f-4c5c-99a9-2f50745879c2" containerID="6492b9d05573e00e4ad262d599b4296daa2a570510a415fbb985e29ea257797a" exitCode=0 Sep 10 16:11:10.274593 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:10.274563 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" Sep 10 16:11:10.274755 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:10.274593 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" event={"ID":"fcddf00e-6d3f-4c5c-99a9-2f50745879c2","Type":"ContainerDied","Data":"6492b9d05573e00e4ad262d599b4296daa2a570510a415fbb985e29ea257797a"} Sep 10 16:11:10.274755 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:10.274644 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc" event={"ID":"fcddf00e-6d3f-4c5c-99a9-2f50745879c2","Type":"ContainerDied","Data":"285167ede105402b4ec003c5bbb724958f47b54be2eb3fdfec4bbef1c3729ff6"} Sep 10 16:11:10.274755 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:10.274671 2583 scope.go:117] "RemoveContainer" containerID="6492b9d05573e00e4ad262d599b4296daa2a570510a415fbb985e29ea257797a" Sep 10 16:11:10.283907 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:10.283890 2583 scope.go:117] "RemoveContainer" containerID="5ebbe9a6a1ec366e10b1f76f0c2dc60c5767775da8de466c184d34cfbdadacc0" Sep 10 16:11:10.292319 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:10.292301 2583 scope.go:117] "RemoveContainer" containerID="6492b9d05573e00e4ad262d599b4296daa2a570510a415fbb985e29ea257797a" Sep 10 16:11:10.292602 ip-10-0-139-107 kubenswrapper[2583]: E0910 16:11:10.292575 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6492b9d05573e00e4ad262d599b4296daa2a570510a415fbb985e29ea257797a\": container with ID starting with 6492b9d05573e00e4ad262d599b4296daa2a570510a415fbb985e29ea257797a not found: ID does not exist" containerID="6492b9d05573e00e4ad262d599b4296daa2a570510a415fbb985e29ea257797a" Sep 10 16:11:10.292655 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:10.292611 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6492b9d05573e00e4ad262d599b4296daa2a570510a415fbb985e29ea257797a"} err="failed to get container status \"6492b9d05573e00e4ad262d599b4296daa2a570510a415fbb985e29ea257797a\": rpc error: code = NotFound desc = could not find container \"6492b9d05573e00e4ad262d599b4296daa2a570510a415fbb985e29ea257797a\": container with ID starting with 6492b9d05573e00e4ad262d599b4296daa2a570510a415fbb985e29ea257797a not found: ID does not exist" Sep 10 16:11:10.292655 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:10.292627 2583 scope.go:117] "RemoveContainer" containerID="5ebbe9a6a1ec366e10b1f76f0c2dc60c5767775da8de466c184d34cfbdadacc0" Sep 10 16:11:10.292866 ip-10-0-139-107 kubenswrapper[2583]: E0910 16:11:10.292851 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ebbe9a6a1ec366e10b1f76f0c2dc60c5767775da8de466c184d34cfbdadacc0\": container with ID starting with 5ebbe9a6a1ec366e10b1f76f0c2dc60c5767775da8de466c184d34cfbdadacc0 not found: ID does not exist" containerID="5ebbe9a6a1ec366e10b1f76f0c2dc60c5767775da8de466c184d34cfbdadacc0" Sep 10 16:11:10.292910 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:10.292870 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ebbe9a6a1ec366e10b1f76f0c2dc60c5767775da8de466c184d34cfbdadacc0"} err="failed to get container status \"5ebbe9a6a1ec366e10b1f76f0c2dc60c5767775da8de466c184d34cfbdadacc0\": rpc error: code = NotFound desc = could not find container \"5ebbe9a6a1ec366e10b1f76f0c2dc60c5767775da8de466c184d34cfbdadacc0\": container with ID starting with 5ebbe9a6a1ec366e10b1f76f0c2dc60c5767775da8de466c184d34cfbdadacc0 not found: ID does not exist" Sep 10 16:11:10.302176 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:10.302158 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-workload-socket\") pod \"fcddf00e-6d3f-4c5c-99a9-2f50745879c2\" (UID: \"fcddf00e-6d3f-4c5c-99a9-2f50745879c2\") " Sep 10 16:11:10.302252 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:10.302203 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-istio-data\") pod \"fcddf00e-6d3f-4c5c-99a9-2f50745879c2\" (UID: \"fcddf00e-6d3f-4c5c-99a9-2f50745879c2\") " Sep 10 16:11:10.302252 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:10.302238 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-istiod-ca-cert\") pod \"fcddf00e-6d3f-4c5c-99a9-2f50745879c2\" (UID: \"fcddf00e-6d3f-4c5c-99a9-2f50745879c2\") " Sep 10 16:11:10.302333 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:10.302274 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-credential-socket\") pod \"fcddf00e-6d3f-4c5c-99a9-2f50745879c2\" (UID: \"fcddf00e-6d3f-4c5c-99a9-2f50745879c2\") " Sep 10 16:11:10.302333 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:10.302297 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-istio-envoy\") pod \"fcddf00e-6d3f-4c5c-99a9-2f50745879c2\" (UID: \"fcddf00e-6d3f-4c5c-99a9-2f50745879c2\") " Sep 10 16:11:10.302439 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:10.302336 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-workload-certs\") pod \"fcddf00e-6d3f-4c5c-99a9-2f50745879c2\" (UID: \"fcddf00e-6d3f-4c5c-99a9-2f50745879c2\") " Sep 10 16:11:10.302439 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:10.302365 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-istio-token\") pod \"fcddf00e-6d3f-4c5c-99a9-2f50745879c2\" (UID: \"fcddf00e-6d3f-4c5c-99a9-2f50745879c2\") " Sep 10 16:11:10.302439 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:10.302433 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slrcs\" (UniqueName: \"kubernetes.io/projected/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-kube-api-access-slrcs\") pod \"fcddf00e-6d3f-4c5c-99a9-2f50745879c2\" (UID: \"fcddf00e-6d3f-4c5c-99a9-2f50745879c2\") " Sep 10 16:11:10.302574 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:10.302446 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fcddf00e-6d3f-4c5c-99a9-2f50745879c2" (UID: "fcddf00e-6d3f-4c5c-99a9-2f50745879c2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:11:10.302574 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:10.302477 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-istio-podinfo\") pod \"fcddf00e-6d3f-4c5c-99a9-2f50745879c2\" (UID: \"fcddf00e-6d3f-4c5c-99a9-2f50745879c2\") " Sep 10 16:11:10.302574 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:10.302517 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-istio-data" (OuterVolumeSpecName: "istio-data") pod "fcddf00e-6d3f-4c5c-99a9-2f50745879c2" (UID: "fcddf00e-6d3f-4c5c-99a9-2f50745879c2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:11:10.302736 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:10.302589 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fcddf00e-6d3f-4c5c-99a9-2f50745879c2" (UID: "fcddf00e-6d3f-4c5c-99a9-2f50745879c2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:11:10.302794 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:10.302771 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fcddf00e-6d3f-4c5c-99a9-2f50745879c2" (UID: "fcddf00e-6d3f-4c5c-99a9-2f50745879c2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:11:10.302794 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:10.302782 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fcddf00e-6d3f-4c5c-99a9-2f50745879c2" (UID: "fcddf00e-6d3f-4c5c-99a9-2f50745879c2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:11:10.302910 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:10.302882 2583 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-credential-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:11:10.302910 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:10.302897 2583 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-workload-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:11:10.303019 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:10.302910 2583 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-workload-socket\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:11:10.303019 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:10.302925 2583 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-istio-data\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:11:10.303019 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:10.302939 2583 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-istiod-ca-cert\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:11:10.304733 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:10.304702 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-istio-token" (OuterVolumeSpecName: "istio-token") pod "fcddf00e-6d3f-4c5c-99a9-2f50745879c2" (UID: "fcddf00e-6d3f-4c5c-99a9-2f50745879c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:11:10.304733 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:10.304712 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fcddf00e-6d3f-4c5c-99a9-2f50745879c2" (UID: "fcddf00e-6d3f-4c5c-99a9-2f50745879c2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 10 16:11:10.304865 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:10.304732 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-kube-api-access-slrcs" (OuterVolumeSpecName: "kube-api-access-slrcs") pod "fcddf00e-6d3f-4c5c-99a9-2f50745879c2" (UID: "fcddf00e-6d3f-4c5c-99a9-2f50745879c2"). InnerVolumeSpecName "kube-api-access-slrcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:11:10.304865 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:10.304788 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fcddf00e-6d3f-4c5c-99a9-2f50745879c2" (UID: "fcddf00e-6d3f-4c5c-99a9-2f50745879c2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 10 16:11:10.403608 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:10.403585 2583 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-istio-podinfo\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:11:10.403608 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:10.403605 2583 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-istio-envoy\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:11:10.403740 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:10.403613 2583 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-istio-token\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:11:10.403740 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:10.403621 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slrcs\" (UniqueName: \"kubernetes.io/projected/fcddf00e-6d3f-4c5c-99a9-2f50745879c2-kube-api-access-slrcs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:11:10.599743 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:10.599711 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc"] Sep 10 16:11:10.611653 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:10.611621 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-65ee1d-openshift-default-645c49bf9b-s45fc"] Sep 10 16:11:11.515789 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:11.515754 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcddf00e-6d3f-4c5c-99a9-2f50745879c2" path="/var/lib/kubelet/pods/fcddf00e-6d3f-4c5c-99a9-2f50745879c2/volumes" Sep 10 16:11:23.395560 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:23.395501 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:24.003748 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:24.003713 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc8c1e-5d988dffbd-w96tp"] Sep 10 16:11:24.003960 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:24.003936 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc8c1e-5d988dffbd-w96tp" podUID="caeb4219-f316-40dc-8ca9-5542b58a99d2" containerName="maas-api" containerID="cri-o://8fadaa333cac9d78934118adbfa59ee0b070c56f91eeeb16c3a07dee8ea641a5" gracePeriod=30 Sep 10 16:11:24.064924 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:24.064885 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fc8c1e-6f8488ff89-6cp8m"] Sep 10 16:11:24.065176 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:24.065129 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc8c1e-6f8488ff89-6cp8m" podUID="e0c29a25-b303-4838-8a4e-0b1564b14e06" containerName="payload-processing" containerID="cri-o://5facf13e7fb9e21d4c7cc3f2fc3e4e0455f54b49d198fe28a888fbc9779104af" gracePeriod=30 Sep 10 16:11:24.074184 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:24.074158 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fc8c1e-59bdb57d4b-8w54b"] Sep 10 16:11:24.074432 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:24.074352 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc8c1e-59bdb57d4b-8w54b" podUID="801d4560-01db-4a3b-8794-41b3779141b6" containerName="payload-pre-processing" containerID="cri-o://ffa23524ad2676970f4970f526dbd4b628237d09e2697ce785b2a0e9169d07be" gracePeriod=30 Sep 10 16:11:24.359869 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:24.359838 2583 generic.go:356] "Generic (PLEG): container finished" podID="801d4560-01db-4a3b-8794-41b3779141b6" containerID="ffa23524ad2676970f4970f526dbd4b628237d09e2697ce785b2a0e9169d07be" exitCode=0 Sep 10 16:11:24.359990 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:24.359910 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc8c1e-59bdb57d4b-8w54b" event={"ID":"801d4560-01db-4a3b-8794-41b3779141b6","Type":"ContainerDied","Data":"ffa23524ad2676970f4970f526dbd4b628237d09e2697ce785b2a0e9169d07be"} Sep 10 16:11:24.361418 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:24.361367 2583 generic.go:356] "Generic (PLEG): container finished" podID="caeb4219-f316-40dc-8ca9-5542b58a99d2" containerID="8fadaa333cac9d78934118adbfa59ee0b070c56f91eeeb16c3a07dee8ea641a5" exitCode=0 Sep 10 16:11:24.361511 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:24.361422 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc8c1e-5d988dffbd-w96tp" event={"ID":"caeb4219-f316-40dc-8ca9-5542b58a99d2","Type":"ContainerDied","Data":"8fadaa333cac9d78934118adbfa59ee0b070c56f91eeeb16c3a07dee8ea641a5"} Sep 10 16:11:24.361511 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:24.361465 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc8c1e-5d988dffbd-w96tp" event={"ID":"caeb4219-f316-40dc-8ca9-5542b58a99d2","Type":"ContainerDied","Data":"4b774bb4193713c0c5bf04eb363fb08a85d6364cc97abd5221d416c1fd706684"} Sep 10 16:11:24.361511 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:24.361487 2583 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b774bb4193713c0c5bf04eb363fb08a85d6364cc97abd5221d416c1fd706684" Sep 10 16:11:24.362696 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:24.362675 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc8c1e-5d988dffbd-w96tp" Sep 10 16:11:24.363295 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:24.363274 2583 generic.go:356] "Generic (PLEG): container finished" podID="e0c29a25-b303-4838-8a4e-0b1564b14e06" containerID="5facf13e7fb9e21d4c7cc3f2fc3e4e0455f54b49d198fe28a888fbc9779104af" exitCode=0 Sep 10 16:11:24.363389 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:24.363317 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc8c1e-6f8488ff89-6cp8m" event={"ID":"e0c29a25-b303-4838-8a4e-0b1564b14e06","Type":"ContainerDied","Data":"5facf13e7fb9e21d4c7cc3f2fc3e4e0455f54b49d198fe28a888fbc9779104af"} Sep 10 16:11:24.379331 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:24.379301 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc8c1e-59bdb57d4b-8w54b" Sep 10 16:11:24.390626 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:24.390597 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc8c1e-6f8488ff89-6cp8m" Sep 10 16:11:24.426938 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:24.426855 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7rkx\" (UniqueName: \"kubernetes.io/projected/caeb4219-f316-40dc-8ca9-5542b58a99d2-kube-api-access-b7rkx\") pod \"caeb4219-f316-40dc-8ca9-5542b58a99d2\" (UID: \"caeb4219-f316-40dc-8ca9-5542b58a99d2\") " Sep 10 16:11:24.428681 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:24.427292 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/caeb4219-f316-40dc-8ca9-5542b58a99d2-openshift-service-ca\") pod \"caeb4219-f316-40dc-8ca9-5542b58a99d2\" (UID: \"caeb4219-f316-40dc-8ca9-5542b58a99d2\") " Sep 10 16:11:24.428681 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:24.427594 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/801d4560-01db-4a3b-8794-41b3779141b6-plugins-config-volume\") pod \"801d4560-01db-4a3b-8794-41b3779141b6\" (UID: \"801d4560-01db-4a3b-8794-41b3779141b6\") " Sep 10 16:11:24.428681 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:24.427633 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0c29a25-b303-4838-8a4e-0b1564b14e06-plugins-config-volume\") pod \"e0c29a25-b303-4838-8a4e-0b1564b14e06\" (UID: \"e0c29a25-b303-4838-8a4e-0b1564b14e06\") " Sep 10 16:11:24.428681 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:24.427665 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caeb4219-f316-40dc-8ca9-5542b58a99d2-metrics-certs\") pod \"caeb4219-f316-40dc-8ca9-5542b58a99d2\" (UID: \"caeb4219-f316-40dc-8ca9-5542b58a99d2\") " Sep 10 16:11:24.428681 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:24.427774 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9vqm\" (UniqueName: \"kubernetes.io/projected/e0c29a25-b303-4838-8a4e-0b1564b14e06-kube-api-access-q9vqm\") pod \"e0c29a25-b303-4838-8a4e-0b1564b14e06\" (UID: \"e0c29a25-b303-4838-8a4e-0b1564b14e06\") " Sep 10 16:11:24.428681 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:24.427821 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caeb4219-f316-40dc-8ca9-5542b58a99d2-maas-api-tls\") pod \"caeb4219-f316-40dc-8ca9-5542b58a99d2\" (UID: \"caeb4219-f316-40dc-8ca9-5542b58a99d2\") " Sep 10 16:11:24.428681 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:24.427853 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6p4cr\" (UniqueName: \"kubernetes.io/projected/801d4560-01db-4a3b-8794-41b3779141b6-kube-api-access-6p4cr\") pod \"801d4560-01db-4a3b-8794-41b3779141b6\" (UID: \"801d4560-01db-4a3b-8794-41b3779141b6\") " Sep 10 16:11:24.428681 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:24.428349 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0c29a25-b303-4838-8a4e-0b1564b14e06-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e0c29a25-b303-4838-8a4e-0b1564b14e06" (UID: "e0c29a25-b303-4838-8a4e-0b1564b14e06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:11:24.429143 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:24.428688 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/801d4560-01db-4a3b-8794-41b3779141b6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "801d4560-01db-4a3b-8794-41b3779141b6" (UID: "801d4560-01db-4a3b-8794-41b3779141b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:11:24.429143 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:24.428857 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caeb4219-f316-40dc-8ca9-5542b58a99d2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "caeb4219-f316-40dc-8ca9-5542b58a99d2" (UID: "caeb4219-f316-40dc-8ca9-5542b58a99d2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:11:24.429841 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:24.429782 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caeb4219-f316-40dc-8ca9-5542b58a99d2-kube-api-access-b7rkx" (OuterVolumeSpecName: "kube-api-access-b7rkx") pod "caeb4219-f316-40dc-8ca9-5542b58a99d2" (UID: "caeb4219-f316-40dc-8ca9-5542b58a99d2"). InnerVolumeSpecName "kube-api-access-b7rkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:11:24.431127 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:24.431096 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0c29a25-b303-4838-8a4e-0b1564b14e06-kube-api-access-q9vqm" (OuterVolumeSpecName: "kube-api-access-q9vqm") pod "e0c29a25-b303-4838-8a4e-0b1564b14e06" (UID: "e0c29a25-b303-4838-8a4e-0b1564b14e06"). InnerVolumeSpecName "kube-api-access-q9vqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:11:24.431430 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:24.431369 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caeb4219-f316-40dc-8ca9-5542b58a99d2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "caeb4219-f316-40dc-8ca9-5542b58a99d2" (UID: "caeb4219-f316-40dc-8ca9-5542b58a99d2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:11:24.431528 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:24.431441 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caeb4219-f316-40dc-8ca9-5542b58a99d2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "caeb4219-f316-40dc-8ca9-5542b58a99d2" (UID: "caeb4219-f316-40dc-8ca9-5542b58a99d2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:11:24.431717 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:24.431699 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/801d4560-01db-4a3b-8794-41b3779141b6-kube-api-access-6p4cr" (OuterVolumeSpecName: "kube-api-access-6p4cr") pod "801d4560-01db-4a3b-8794-41b3779141b6" (UID: "801d4560-01db-4a3b-8794-41b3779141b6"). InnerVolumeSpecName "kube-api-access-6p4cr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:11:24.529355 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:24.529321 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9vqm\" (UniqueName: \"kubernetes.io/projected/e0c29a25-b303-4838-8a4e-0b1564b14e06-kube-api-access-q9vqm\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:11:24.529355 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:24.529349 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/caeb4219-f316-40dc-8ca9-5542b58a99d2-maas-api-tls\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:11:24.529355 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:24.529359 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p4cr\" (UniqueName: \"kubernetes.io/projected/801d4560-01db-4a3b-8794-41b3779141b6-kube-api-access-6p4cr\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:11:24.529604 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:24.529367 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7rkx\" (UniqueName: \"kubernetes.io/projected/caeb4219-f316-40dc-8ca9-5542b58a99d2-kube-api-access-b7rkx\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:11:24.529604 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:24.529376 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/caeb4219-f316-40dc-8ca9-5542b58a99d2-openshift-service-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:11:24.529604 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:24.529411 2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/801d4560-01db-4a3b-8794-41b3779141b6-plugins-config-volume\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:11:24.529604 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:24.529419 2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e0c29a25-b303-4838-8a4e-0b1564b14e06-plugins-config-volume\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:11:24.529604 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:24.529427 2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/caeb4219-f316-40dc-8ca9-5542b58a99d2-metrics-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:11:25.370221 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:25.370147 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc8c1e-59bdb57d4b-8w54b" Sep 10 16:11:25.370221 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:25.370175 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-fc8c1e-59bdb57d4b-8w54b" event={"ID":"801d4560-01db-4a3b-8794-41b3779141b6","Type":"ContainerDied","Data":"f45e94ea0d7dfbeb4f72dd1bb232802c77962871f1e76b7a3f8b90d0999a10cb"} Sep 10 16:11:25.370457 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:25.370239 2583 scope.go:117] "RemoveContainer" containerID="ffa23524ad2676970f4970f526dbd4b628237d09e2697ce785b2a0e9169d07be" Sep 10 16:11:25.371762 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:25.371740 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc8c1e-6f8488ff89-6cp8m" event={"ID":"e0c29a25-b303-4838-8a4e-0b1564b14e06","Type":"ContainerDied","Data":"40e1ea7d6ba99960222195590d34ddb705f81aac5a49f1a3b222609e0cbdc293"} Sep 10 16:11:25.371762 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:25.371757 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fc8c1e-6f8488ff89-6cp8m" Sep 10 16:11:25.371883 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:25.371770 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc8c1e-5d988dffbd-w96tp" Sep 10 16:11:25.380779 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:25.380762 2583 scope.go:117] "RemoveContainer" containerID="5facf13e7fb9e21d4c7cc3f2fc3e4e0455f54b49d198fe28a888fbc9779104af" Sep 10 16:11:25.396832 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:25.396812 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fc8c1e-59bdb57d4b-8w54b"] Sep 10 16:11:25.401694 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:25.401671 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-fc8c1e-59bdb57d4b-8w54b"] Sep 10 16:11:25.418342 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:25.418320 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc8c1e-5d988dffbd-w96tp"] Sep 10 16:11:25.420596 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:25.420573 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fc8c1e-5d988dffbd-w96tp"] Sep 10 16:11:25.427747 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:25.427729 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fc8c1e-6f8488ff89-6cp8m"] Sep 10 16:11:25.432345 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:25.432326 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fc8c1e-6f8488ff89-6cp8m"] Sep 10 16:11:25.482884 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:25.482855 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:11:25.515075 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:25.515048 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="801d4560-01db-4a3b-8794-41b3779141b6" path="/var/lib/kubelet/pods/801d4560-01db-4a3b-8794-41b3779141b6/volumes" Sep 10 16:11:25.515497 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:25.515484 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="caeb4219-f316-40dc-8ca9-5542b58a99d2" path="/var/lib/kubelet/pods/caeb4219-f316-40dc-8ca9-5542b58a99d2/volumes" Sep 10 16:11:25.515808 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:25.515797 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0c29a25-b303-4838-8a4e-0b1564b14e06" path="/var/lib/kubelet/pods/e0c29a25-b303-4838-8a4e-0b1564b14e06/volumes" Sep 10 16:11:29.842627 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:11:29.842593 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:04.505540 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:04.505506 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:06.595102 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:06.595070 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:07.815885 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:07.815841 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:14.552366 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:14.552329 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:14.653394 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:14.653347 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:14.868441 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:14.868337 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:15.361901 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:15.361870 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:25.876534 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:25.876496 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:25.954222 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:25.954188 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:26.096666 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:26.096631 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:26.222244 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:26.222214 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:26.344635 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:26.344604 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:26.582976 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:26.582886 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:32.022516 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:32.022486 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:32.956317 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:32.956280 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:33.116076 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:33.116036 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:33.224541 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:33.224454 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:33.780706 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:33.780669 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:44.206080 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:44.206041 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b52270-6f9778864c-mmxv2"] Sep 10 16:12:44.206560 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:44.206301 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b52270-6f9778864c-mmxv2" podUID="5e645ebb-52bc-46c6-9429-b4d7d447649a" containerName="maas-api" containerID="cri-o://f6509b2c49e76ca467f07713336e900eee59a24cbb83709a260b6b21e64f3b3c" gracePeriod=30 Sep 10 16:12:44.296626 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:44.296595 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:44.314815 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:44.314789 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b52270-bb595ff74-jg88r"] Sep 10 16:12:44.315087 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:44.315050 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b52270-bb595ff74-jg88r" podUID="bb00c59d-fe86-4af7-97b2-5d4d81c72330" containerName="payload-pre-processing" containerID="cri-o://b24260d6ebb5760ed5eb1c0c02589a49303835010022552b104a7438b9f7351e" gracePeriod=30 Sep 10 16:12:44.401921 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:44.401888 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:44.457728 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:44.457672 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b52270-6f9778864c-mmxv2" Sep 10 16:12:44.540865 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:44.540826 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:44.566369 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:44.566345 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b52270-bb595ff74-jg88r" Sep 10 16:12:44.623501 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:44.623470 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e645ebb-52bc-46c6-9429-b4d7d447649a-openshift-service-ca\") pod \"5e645ebb-52bc-46c6-9429-b4d7d447649a\" (UID: \"5e645ebb-52bc-46c6-9429-b4d7d447649a\") " Sep 10 16:12:44.623649 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:44.623568 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e645ebb-52bc-46c6-9429-b4d7d447649a-metrics-certs\") pod \"5e645ebb-52bc-46c6-9429-b4d7d447649a\" (UID: \"5e645ebb-52bc-46c6-9429-b4d7d447649a\") " Sep 10 16:12:44.623649 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:44.623630 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e645ebb-52bc-46c6-9429-b4d7d447649a-maas-api-tls\") pod \"5e645ebb-52bc-46c6-9429-b4d7d447649a\" (UID: \"5e645ebb-52bc-46c6-9429-b4d7d447649a\") " Sep 10 16:12:44.623766 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:44.623660 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwzr8\" (UniqueName: \"kubernetes.io/projected/5e645ebb-52bc-46c6-9429-b4d7d447649a-kube-api-access-hwzr8\") pod \"5e645ebb-52bc-46c6-9429-b4d7d447649a\" (UID: \"5e645ebb-52bc-46c6-9429-b4d7d447649a\") " Sep 10 16:12:44.623862 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:44.623835 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e645ebb-52bc-46c6-9429-b4d7d447649a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5e645ebb-52bc-46c6-9429-b4d7d447649a" (UID: "5e645ebb-52bc-46c6-9429-b4d7d447649a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:12:44.624265 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:44.624241 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e645ebb-52bc-46c6-9429-b4d7d447649a-openshift-service-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:12:44.625662 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:44.625639 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e645ebb-52bc-46c6-9429-b4d7d447649a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5e645ebb-52bc-46c6-9429-b4d7d447649a" (UID: "5e645ebb-52bc-46c6-9429-b4d7d447649a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:12:44.625771 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:44.625727 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e645ebb-52bc-46c6-9429-b4d7d447649a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5e645ebb-52bc-46c6-9429-b4d7d447649a" (UID: "5e645ebb-52bc-46c6-9429-b4d7d447649a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:12:44.625818 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:44.625782 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e645ebb-52bc-46c6-9429-b4d7d447649a-kube-api-access-hwzr8" (OuterVolumeSpecName: "kube-api-access-hwzr8") pod "5e645ebb-52bc-46c6-9429-b4d7d447649a" (UID: "5e645ebb-52bc-46c6-9429-b4d7d447649a"). InnerVolumeSpecName "kube-api-access-hwzr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:12:44.699725 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:44.699692 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:44.725125 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:44.725057 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb00c59d-fe86-4af7-97b2-5d4d81c72330-plugins-config-volume\") pod \"bb00c59d-fe86-4af7-97b2-5d4d81c72330\" (UID: \"bb00c59d-fe86-4af7-97b2-5d4d81c72330\") " Sep 10 16:12:44.725125 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:44.725102 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcwhp\" (UniqueName: \"kubernetes.io/projected/bb00c59d-fe86-4af7-97b2-5d4d81c72330-kube-api-access-kcwhp\") pod \"bb00c59d-fe86-4af7-97b2-5d4d81c72330\" (UID: \"bb00c59d-fe86-4af7-97b2-5d4d81c72330\") " Sep 10 16:12:44.725325 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:44.725313 2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e645ebb-52bc-46c6-9429-b4d7d447649a-metrics-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:12:44.725365 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:44.725331 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e645ebb-52bc-46c6-9429-b4d7d447649a-maas-api-tls\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:12:44.725365 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:44.725344 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwzr8\" (UniqueName: \"kubernetes.io/projected/5e645ebb-52bc-46c6-9429-b4d7d447649a-kube-api-access-hwzr8\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:12:44.725538 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:44.725512 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb00c59d-fe86-4af7-97b2-5d4d81c72330-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bb00c59d-fe86-4af7-97b2-5d4d81c72330" (UID: "bb00c59d-fe86-4af7-97b2-5d4d81c72330"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:12:44.727165 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:44.727137 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb00c59d-fe86-4af7-97b2-5d4d81c72330-kube-api-access-kcwhp" (OuterVolumeSpecName: "kube-api-access-kcwhp") pod "bb00c59d-fe86-4af7-97b2-5d4d81c72330" (UID: "bb00c59d-fe86-4af7-97b2-5d4d81c72330"). InnerVolumeSpecName "kube-api-access-kcwhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:12:44.813092 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:44.813061 2583 generic.go:356] "Generic (PLEG): container finished" podID="bb00c59d-fe86-4af7-97b2-5d4d81c72330" containerID="b24260d6ebb5760ed5eb1c0c02589a49303835010022552b104a7438b9f7351e" exitCode=0 Sep 10 16:12:44.813201 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:44.813111 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b52270-bb595ff74-jg88r" Sep 10 16:12:44.813201 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:44.813145 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b52270-bb595ff74-jg88r" event={"ID":"bb00c59d-fe86-4af7-97b2-5d4d81c72330","Type":"ContainerDied","Data":"b24260d6ebb5760ed5eb1c0c02589a49303835010022552b104a7438b9f7351e"} Sep 10 16:12:44.813201 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:44.813196 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b52270-bb595ff74-jg88r" event={"ID":"bb00c59d-fe86-4af7-97b2-5d4d81c72330","Type":"ContainerDied","Data":"53bcb63c234293974bf02ad705f4a0511737232de05ae9fa59400ef7db0da259"} Sep 10 16:12:44.813342 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:44.813222 2583 scope.go:117] "RemoveContainer" containerID="b24260d6ebb5760ed5eb1c0c02589a49303835010022552b104a7438b9f7351e" Sep 10 16:12:44.814577 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:44.814551 2583 generic.go:356] "Generic (PLEG): container finished" podID="5e645ebb-52bc-46c6-9429-b4d7d447649a" containerID="f6509b2c49e76ca467f07713336e900eee59a24cbb83709a260b6b21e64f3b3c" exitCode=0 Sep 10 16:12:44.814702 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:44.814584 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b52270-6f9778864c-mmxv2" event={"ID":"5e645ebb-52bc-46c6-9429-b4d7d447649a","Type":"ContainerDied","Data":"f6509b2c49e76ca467f07713336e900eee59a24cbb83709a260b6b21e64f3b3c"} Sep 10 16:12:44.814702 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:44.814597 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b52270-6f9778864c-mmxv2" Sep 10 16:12:44.814702 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:44.814633 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b52270-6f9778864c-mmxv2" event={"ID":"5e645ebb-52bc-46c6-9429-b4d7d447649a","Type":"ContainerDied","Data":"4e7087c0fac86e3cb82728a978b377773d33a1641ff1d6c1f87402b6a05a0285"} Sep 10 16:12:44.823993 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:44.823976 2583 scope.go:117] "RemoveContainer" containerID="b24260d6ebb5760ed5eb1c0c02589a49303835010022552b104a7438b9f7351e" Sep 10 16:12:44.824305 ip-10-0-139-107 kubenswrapper[2583]: E0910 16:12:44.824286 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b24260d6ebb5760ed5eb1c0c02589a49303835010022552b104a7438b9f7351e\": container with ID starting with b24260d6ebb5760ed5eb1c0c02589a49303835010022552b104a7438b9f7351e not found: ID does not exist" containerID="b24260d6ebb5760ed5eb1c0c02589a49303835010022552b104a7438b9f7351e" Sep 10 16:12:44.824395 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:44.824317 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b24260d6ebb5760ed5eb1c0c02589a49303835010022552b104a7438b9f7351e"} err="failed to get container status \"b24260d6ebb5760ed5eb1c0c02589a49303835010022552b104a7438b9f7351e\": rpc error: code = NotFound desc = could not find container \"b24260d6ebb5760ed5eb1c0c02589a49303835010022552b104a7438b9f7351e\": container with ID starting with b24260d6ebb5760ed5eb1c0c02589a49303835010022552b104a7438b9f7351e not found: ID does not exist" Sep 10 16:12:44.824395 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:44.824334 2583 scope.go:117] "RemoveContainer" containerID="f6509b2c49e76ca467f07713336e900eee59a24cbb83709a260b6b21e64f3b3c" Sep 10 16:12:44.826010 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:44.825991 2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb00c59d-fe86-4af7-97b2-5d4d81c72330-plugins-config-volume\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:12:44.826090 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:44.826015 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcwhp\" (UniqueName: \"kubernetes.io/projected/bb00c59d-fe86-4af7-97b2-5d4d81c72330-kube-api-access-kcwhp\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:12:44.832422 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:44.832406 2583 scope.go:117] "RemoveContainer" containerID="f6509b2c49e76ca467f07713336e900eee59a24cbb83709a260b6b21e64f3b3c" Sep 10 16:12:44.832673 ip-10-0-139-107 kubenswrapper[2583]: E0910 16:12:44.832657 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6509b2c49e76ca467f07713336e900eee59a24cbb83709a260b6b21e64f3b3c\": container with ID starting with f6509b2c49e76ca467f07713336e900eee59a24cbb83709a260b6b21e64f3b3c not found: ID does not exist" containerID="f6509b2c49e76ca467f07713336e900eee59a24cbb83709a260b6b21e64f3b3c" Sep 10 16:12:44.832725 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:44.832679 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6509b2c49e76ca467f07713336e900eee59a24cbb83709a260b6b21e64f3b3c"} err="failed to get container status \"f6509b2c49e76ca467f07713336e900eee59a24cbb83709a260b6b21e64f3b3c\": rpc error: code = NotFound desc = could not find container \"f6509b2c49e76ca467f07713336e900eee59a24cbb83709a260b6b21e64f3b3c\": container with ID starting with f6509b2c49e76ca467f07713336e900eee59a24cbb83709a260b6b21e64f3b3c not found: ID does not exist" Sep 10 16:12:44.847157 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:44.847137 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b52270-bb595ff74-jg88r"] Sep 10 16:12:44.853781 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:44.853762 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-b52270-bb595ff74-jg88r"] Sep 10 16:12:44.882934 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:44.882909 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b52270-6f9778864c-mmxv2"] Sep 10 16:12:44.895013 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:44.894988 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-b52270-6f9778864c-mmxv2"] Sep 10 16:12:45.084777 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:45.084697 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:45.515758 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:45.515727 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e645ebb-52bc-46c6-9429-b4d7d447649a" path="/var/lib/kubelet/pods/5e645ebb-52bc-46c6-9429-b4d7d447649a/volumes" Sep 10 16:12:45.516112 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:45.516070 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb00c59d-fe86-4af7-97b2-5d4d81c72330" path="/var/lib/kubelet/pods/bb00c59d-fe86-4af7-97b2-5d4d81c72330/volumes" Sep 10 16:12:53.296123 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:53.296089 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:53.447584 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:53.447553 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:53.598659 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:53.598581 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:12:54.158805 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:12:54.158776 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:13:04.680482 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:04.680444 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-108dbb-74d7f9c965-p5hhn"] Sep 10 16:13:04.680980 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:04.680675 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-108dbb-74d7f9c965-p5hhn" podUID="807a838c-5645-422a-9ab2-2d1a0b139189" containerName="maas-api" containerID="cri-o://ede242e6225ca72075e0f4ef872c0f58dfe877ee53569000a8468ef033ec8585" gracePeriod=30 Sep 10 16:13:04.722196 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:04.722167 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:13:04.775267 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:04.775180 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-108dbb-c46946898-7sxn5"] Sep 10 16:13:04.775531 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:04.775498 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-108dbb-c46946898-7sxn5" podUID="6c6c9359-d7ae-4c58-9a62-16104d1deb17" containerName="payload-pre-processing" containerID="cri-o://442b99922db3cdad3ce725935531dfd6fac1f8838a5b0a33331481e57b3a56cf" gracePeriod=30 Sep 10 16:13:04.828159 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:04.828121 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:13:04.900100 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:04.900071 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:13:04.931237 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:04.931179 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-108dbb-74d7f9c965-p5hhn" Sep 10 16:13:04.933806 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:04.933776 2583 generic.go:356] "Generic (PLEG): container finished" podID="6c6c9359-d7ae-4c58-9a62-16104d1deb17" containerID="442b99922db3cdad3ce725935531dfd6fac1f8838a5b0a33331481e57b3a56cf" exitCode=0 Sep 10 16:13:04.933907 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:04.933849 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-108dbb-c46946898-7sxn5" event={"ID":"6c6c9359-d7ae-4c58-9a62-16104d1deb17","Type":"ContainerDied","Data":"442b99922db3cdad3ce725935531dfd6fac1f8838a5b0a33331481e57b3a56cf"} Sep 10 16:13:04.935332 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:04.935297 2583 generic.go:356] "Generic (PLEG): container finished" podID="807a838c-5645-422a-9ab2-2d1a0b139189" containerID="ede242e6225ca72075e0f4ef872c0f58dfe877ee53569000a8468ef033ec8585" exitCode=0 Sep 10 16:13:04.935463 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:04.935329 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-108dbb-74d7f9c965-p5hhn" event={"ID":"807a838c-5645-422a-9ab2-2d1a0b139189","Type":"ContainerDied","Data":"ede242e6225ca72075e0f4ef872c0f58dfe877ee53569000a8468ef033ec8585"} Sep 10 16:13:04.935463 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:04.935351 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-108dbb-74d7f9c965-p5hhn" Sep 10 16:13:04.935463 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:04.935402 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-108dbb-74d7f9c965-p5hhn" event={"ID":"807a838c-5645-422a-9ab2-2d1a0b139189","Type":"ContainerDied","Data":"94de21d441f9c997405d4a2c961ec3d230be9d21cfe4102d8830b0f123183cb6"} Sep 10 16:13:04.935463 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:04.935436 2583 scope.go:117] "RemoveContainer" containerID="ede242e6225ca72075e0f4ef872c0f58dfe877ee53569000a8468ef033ec8585" Sep 10 16:13:04.945088 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:04.945069 2583 scope.go:117] "RemoveContainer" containerID="ede242e6225ca72075e0f4ef872c0f58dfe877ee53569000a8468ef033ec8585" Sep 10 16:13:04.945481 ip-10-0-139-107 kubenswrapper[2583]: E0910 16:13:04.945457 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ede242e6225ca72075e0f4ef872c0f58dfe877ee53569000a8468ef033ec8585\": container with ID starting with ede242e6225ca72075e0f4ef872c0f58dfe877ee53569000a8468ef033ec8585 not found: ID does not exist" containerID="ede242e6225ca72075e0f4ef872c0f58dfe877ee53569000a8468ef033ec8585" Sep 10 16:13:04.945578 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:04.945494 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ede242e6225ca72075e0f4ef872c0f58dfe877ee53569000a8468ef033ec8585"} err="failed to get container status \"ede242e6225ca72075e0f4ef872c0f58dfe877ee53569000a8468ef033ec8585\": rpc error: code = NotFound desc = could not find container \"ede242e6225ca72075e0f4ef872c0f58dfe877ee53569000a8468ef033ec8585\": container with ID starting with ede242e6225ca72075e0f4ef872c0f58dfe877ee53569000a8468ef033ec8585 not found: ID does not exist" Sep 10 16:13:04.999169 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:04.999139 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkf4t\" (UniqueName: \"kubernetes.io/projected/807a838c-5645-422a-9ab2-2d1a0b139189-kube-api-access-pkf4t\") pod \"807a838c-5645-422a-9ab2-2d1a0b139189\" (UID: \"807a838c-5645-422a-9ab2-2d1a0b139189\") " Sep 10 16:13:04.999319 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:04.999222 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/807a838c-5645-422a-9ab2-2d1a0b139189-openshift-service-ca\") pod \"807a838c-5645-422a-9ab2-2d1a0b139189\" (UID: \"807a838c-5645-422a-9ab2-2d1a0b139189\") " Sep 10 16:13:04.999319 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:04.999250 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/807a838c-5645-422a-9ab2-2d1a0b139189-metrics-certs\") pod \"807a838c-5645-422a-9ab2-2d1a0b139189\" (UID: \"807a838c-5645-422a-9ab2-2d1a0b139189\") " Sep 10 16:13:04.999319 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:04.999310 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/807a838c-5645-422a-9ab2-2d1a0b139189-maas-api-tls\") pod \"807a838c-5645-422a-9ab2-2d1a0b139189\" (UID: \"807a838c-5645-422a-9ab2-2d1a0b139189\") " Sep 10 16:13:04.999629 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:04.999596 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/807a838c-5645-422a-9ab2-2d1a0b139189-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "807a838c-5645-422a-9ab2-2d1a0b139189" (UID: "807a838c-5645-422a-9ab2-2d1a0b139189"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:13:05.001345 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:05.001313 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/807a838c-5645-422a-9ab2-2d1a0b139189-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "807a838c-5645-422a-9ab2-2d1a0b139189" (UID: "807a838c-5645-422a-9ab2-2d1a0b139189"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:13:05.001538 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:05.001424 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/807a838c-5645-422a-9ab2-2d1a0b139189-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "807a838c-5645-422a-9ab2-2d1a0b139189" (UID: "807a838c-5645-422a-9ab2-2d1a0b139189"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:13:05.001538 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:05.001473 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/807a838c-5645-422a-9ab2-2d1a0b139189-kube-api-access-pkf4t" (OuterVolumeSpecName: "kube-api-access-pkf4t") pod "807a838c-5645-422a-9ab2-2d1a0b139189" (UID: "807a838c-5645-422a-9ab2-2d1a0b139189"). InnerVolumeSpecName "kube-api-access-pkf4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:13:05.031423 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:05.031373 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-108dbb-c46946898-7sxn5" Sep 10 16:13:05.032000 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:05.031962 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:13:05.100890 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:05.100868 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c6c9359-d7ae-4c58-9a62-16104d1deb17-plugins-config-volume\") pod \"6c6c9359-d7ae-4c58-9a62-16104d1deb17\" (UID: \"6c6c9359-d7ae-4c58-9a62-16104d1deb17\") " Sep 10 16:13:05.101024 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:05.100913 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8p4f\" (UniqueName: \"kubernetes.io/projected/6c6c9359-d7ae-4c58-9a62-16104d1deb17-kube-api-access-d8p4f\") pod \"6c6c9359-d7ae-4c58-9a62-16104d1deb17\" (UID: \"6c6c9359-d7ae-4c58-9a62-16104d1deb17\") " Sep 10 16:13:05.101120 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:05.101105 2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/807a838c-5645-422a-9ab2-2d1a0b139189-metrics-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:13:05.101120 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:05.101120 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/807a838c-5645-422a-9ab2-2d1a0b139189-maas-api-tls\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:13:05.101231 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:05.101129 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkf4t\" (UniqueName: \"kubernetes.io/projected/807a838c-5645-422a-9ab2-2d1a0b139189-kube-api-access-pkf4t\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:13:05.101231 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:05.101138 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/807a838c-5645-422a-9ab2-2d1a0b139189-openshift-service-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:13:05.101231 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:05.101182 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c6c9359-d7ae-4c58-9a62-16104d1deb17-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6c6c9359-d7ae-4c58-9a62-16104d1deb17" (UID: "6c6c9359-d7ae-4c58-9a62-16104d1deb17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:13:05.102873 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:05.102853 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c6c9359-d7ae-4c58-9a62-16104d1deb17-kube-api-access-d8p4f" (OuterVolumeSpecName: "kube-api-access-d8p4f") pod "6c6c9359-d7ae-4c58-9a62-16104d1deb17" (UID: "6c6c9359-d7ae-4c58-9a62-16104d1deb17"). InnerVolumeSpecName "kube-api-access-d8p4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:13:05.108270 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:05.108248 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:13:05.202331 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:05.202266 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8p4f\" (UniqueName: \"kubernetes.io/projected/6c6c9359-d7ae-4c58-9a62-16104d1deb17-kube-api-access-d8p4f\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:13:05.202331 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:05.202289 2583 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6c6c9359-d7ae-4c58-9a62-16104d1deb17-plugins-config-volume\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:13:05.265424 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:05.265397 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-108dbb-74d7f9c965-p5hhn"] Sep 10 16:13:05.270913 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:05.270889 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-108dbb-74d7f9c965-p5hhn"] Sep 10 16:13:05.460271 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:05.460204 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:13:05.516961 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:05.516934 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="807a838c-5645-422a-9ab2-2d1a0b139189" path="/var/lib/kubelet/pods/807a838c-5645-422a-9ab2-2d1a0b139189/volumes" Sep 10 16:13:05.941965 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:05.941935 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-108dbb-c46946898-7sxn5" Sep 10 16:13:05.941965 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:05.941955 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-108dbb-c46946898-7sxn5" event={"ID":"6c6c9359-d7ae-4c58-9a62-16104d1deb17","Type":"ContainerDied","Data":"cf14084d40296ade9e442856c40f60823f89beca08c9b53359fc896f7ce5fa0b"} Sep 10 16:13:05.942480 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:05.942013 2583 scope.go:117] "RemoveContainer" containerID="442b99922db3cdad3ce725935531dfd6fac1f8838a5b0a33331481e57b3a56cf" Sep 10 16:13:05.973236 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:05.973209 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-108dbb-c46946898-7sxn5"] Sep 10 16:13:05.977211 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:05.977191 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-108dbb-c46946898-7sxn5"] Sep 10 16:13:07.517312 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:07.517278 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c6c9359-d7ae-4c58-9a62-16104d1deb17" path="/var/lib/kubelet/pods/6c6c9359-d7ae-4c58-9a62-16104d1deb17/volumes" Sep 10 16:13:12.762237 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:12.762201 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:13:12.944683 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:12.944649 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:13:13.060731 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:13.060643 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:13:23.595060 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:13:23.595027 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:14:36.787601 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:14:36.787565 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:14:37.231537 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:14:37.231505 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:14:38.938929 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:14:38.938898 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:15:03.829594 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:15:03.829516 2583 scope.go:117] "RemoveContainer" containerID="8fadaa333cac9d78934118adbfa59ee0b070c56f91eeeb16c3a07dee8ea641a5" Sep 10 16:15:28.921960 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:15:28.921928 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:16:00.029171 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:16:00.029138 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:17:08.281682 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:17:08.281648 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:17:49.154768 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:17:49.154733 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:01.600812 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:01.600736 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:24.054788 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:24.054752 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:26.059554 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:26.059510 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:18:45.113226 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:45.113196 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-68786f5998-vrxsc"] Sep 10 16:18:45.113636 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:45.113620 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="807a838c-5645-422a-9ab2-2d1a0b139189" containerName="maas-api" Sep 10 16:18:45.113690 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:45.113639 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="807a838c-5645-422a-9ab2-2d1a0b139189" containerName="maas-api" Sep 10 16:18:45.113690 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:45.113665 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acbbca2a-492b-416a-b1ba-73c757caef1c" containerName="payload-processing" Sep 10 16:18:45.113690 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:45.113674 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="acbbca2a-492b-416a-b1ba-73c757caef1c" containerName="payload-processing" Sep 10 16:18:45.113690 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:45.113686 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcddf00e-6d3f-4c5c-99a9-2f50745879c2" containerName="istio-proxy" Sep 10 16:18:45.113690 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:45.113692 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcddf00e-6d3f-4c5c-99a9-2f50745879c2" containerName="istio-proxy" Sep 10 16:18:45.113857 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:45.113701 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcddf00e-6d3f-4c5c-99a9-2f50745879c2" containerName="istio-proxy" Sep 10 16:18:45.113857 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:45.113710 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcddf00e-6d3f-4c5c-99a9-2f50745879c2" containerName="istio-proxy" Sep 10 16:18:45.113857 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:45.113719 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e645ebb-52bc-46c6-9429-b4d7d447649a" containerName="maas-api" Sep 10 16:18:45.113857 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:45.113725 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e645ebb-52bc-46c6-9429-b4d7d447649a" containerName="maas-api" Sep 10 16:18:45.113857 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:45.113731 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c6c9359-d7ae-4c58-9a62-16104d1deb17" containerName="payload-pre-processing" Sep 10 16:18:45.113857 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:45.113736 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c6c9359-d7ae-4c58-9a62-16104d1deb17" containerName="payload-pre-processing" Sep 10 16:18:45.113857 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:45.113746 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caeb4219-f316-40dc-8ca9-5542b58a99d2" containerName="maas-api" Sep 10 16:18:45.113857 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:45.113752 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="caeb4219-f316-40dc-8ca9-5542b58a99d2" containerName="maas-api" Sep 10 16:18:45.113857 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:45.113762 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="801d4560-01db-4a3b-8794-41b3779141b6" containerName="payload-pre-processing" Sep 10 16:18:45.113857 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:45.113767 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="801d4560-01db-4a3b-8794-41b3779141b6" containerName="payload-pre-processing" Sep 10 16:18:45.113857 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:45.113775 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0c29a25-b303-4838-8a4e-0b1564b14e06" containerName="payload-processing" Sep 10 16:18:45.113857 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:45.113780 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0c29a25-b303-4838-8a4e-0b1564b14e06" containerName="payload-processing" Sep 10 16:18:45.113857 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:45.113794 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb00c59d-fe86-4af7-97b2-5d4d81c72330" containerName="payload-pre-processing" Sep 10 16:18:45.113857 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:45.113800 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb00c59d-fe86-4af7-97b2-5d4d81c72330" containerName="payload-pre-processing" Sep 10 16:18:45.113857 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:45.113863 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="807a838c-5645-422a-9ab2-2d1a0b139189" containerName="maas-api" Sep 10 16:18:45.114311 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:45.113877 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c6c9359-d7ae-4c58-9a62-16104d1deb17" containerName="payload-pre-processing" Sep 10 16:18:45.114311 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:45.113883 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="caeb4219-f316-40dc-8ca9-5542b58a99d2" containerName="maas-api" Sep 10 16:18:45.114311 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:45.113890 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="801d4560-01db-4a3b-8794-41b3779141b6" containerName="payload-pre-processing" Sep 10 16:18:45.114311 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:45.113896 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcddf00e-6d3f-4c5c-99a9-2f50745879c2" containerName="istio-proxy" Sep 10 16:18:45.114311 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:45.113902 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0c29a25-b303-4838-8a4e-0b1564b14e06" containerName="payload-processing" Sep 10 16:18:45.114311 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:45.113908 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcddf00e-6d3f-4c5c-99a9-2f50745879c2" containerName="istio-proxy" Sep 10 16:18:45.114311 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:45.113914 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="acbbca2a-492b-416a-b1ba-73c757caef1c" containerName="payload-processing" Sep 10 16:18:45.114311 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:45.113919 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e645ebb-52bc-46c6-9429-b4d7d447649a" containerName="maas-api" Sep 10 16:18:45.114311 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:45.113925 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb00c59d-fe86-4af7-97b2-5d4d81c72330" containerName="payload-pre-processing" Sep 10 16:18:45.117082 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:45.117063 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-68786f5998-vrxsc" Sep 10 16:18:45.128548 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:45.128513 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-68786f5998-vrxsc"] Sep 10 16:18:45.206112 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:45.206076 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc3a7059-c909-4873-aaec-249711e7fa58-openshift-service-ca\") pod \"maas-api-68786f5998-vrxsc\" (UID: \"fc3a7059-c909-4873-aaec-249711e7fa58\") " pod="odh-ai-gateway-infra/maas-api-68786f5998-vrxsc" Sep 10 16:18:45.206263 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:45.206119 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzkgv\" (UniqueName: \"kubernetes.io/projected/fc3a7059-c909-4873-aaec-249711e7fa58-kube-api-access-nzkgv\") pod \"maas-api-68786f5998-vrxsc\" (UID: \"fc3a7059-c909-4873-aaec-249711e7fa58\") " pod="odh-ai-gateway-infra/maas-api-68786f5998-vrxsc" Sep 10 16:18:45.206323 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:45.206252 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc3a7059-c909-4873-aaec-249711e7fa58-maas-api-tls\") pod \"maas-api-68786f5998-vrxsc\" (UID: \"fc3a7059-c909-4873-aaec-249711e7fa58\") " pod="odh-ai-gateway-infra/maas-api-68786f5998-vrxsc" Sep 10 16:18:45.206323 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:45.206283 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc3a7059-c909-4873-aaec-249711e7fa58-metrics-certs\") pod \"maas-api-68786f5998-vrxsc\" (UID: \"fc3a7059-c909-4873-aaec-249711e7fa58\") " pod="odh-ai-gateway-infra/maas-api-68786f5998-vrxsc" Sep 10 16:18:45.307356 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:45.307328 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc3a7059-c909-4873-aaec-249711e7fa58-openshift-service-ca\") pod \"maas-api-68786f5998-vrxsc\" (UID: \"fc3a7059-c909-4873-aaec-249711e7fa58\") " pod="odh-ai-gateway-infra/maas-api-68786f5998-vrxsc" Sep 10 16:18:45.307534 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:45.307365 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzkgv\" (UniqueName: \"kubernetes.io/projected/fc3a7059-c909-4873-aaec-249711e7fa58-kube-api-access-nzkgv\") pod \"maas-api-68786f5998-vrxsc\" (UID: \"fc3a7059-c909-4873-aaec-249711e7fa58\") " pod="odh-ai-gateway-infra/maas-api-68786f5998-vrxsc" Sep 10 16:18:45.307534 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:45.307442 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc3a7059-c909-4873-aaec-249711e7fa58-maas-api-tls\") pod \"maas-api-68786f5998-vrxsc\" (UID: \"fc3a7059-c909-4873-aaec-249711e7fa58\") " pod="odh-ai-gateway-infra/maas-api-68786f5998-vrxsc" Sep 10 16:18:45.307534 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:45.307458 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc3a7059-c909-4873-aaec-249711e7fa58-metrics-certs\") pod \"maas-api-68786f5998-vrxsc\" (UID: \"fc3a7059-c909-4873-aaec-249711e7fa58\") " pod="odh-ai-gateway-infra/maas-api-68786f5998-vrxsc" Sep 10 16:18:45.308027 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:45.308001 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fc3a7059-c909-4873-aaec-249711e7fa58-openshift-service-ca\") pod \"maas-api-68786f5998-vrxsc\" (UID: \"fc3a7059-c909-4873-aaec-249711e7fa58\") " pod="odh-ai-gateway-infra/maas-api-68786f5998-vrxsc" Sep 10 16:18:45.309907 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:45.309883 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc3a7059-c909-4873-aaec-249711e7fa58-metrics-certs\") pod \"maas-api-68786f5998-vrxsc\" (UID: \"fc3a7059-c909-4873-aaec-249711e7fa58\") " pod="odh-ai-gateway-infra/maas-api-68786f5998-vrxsc" Sep 10 16:18:45.309999 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:45.309883 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fc3a7059-c909-4873-aaec-249711e7fa58-maas-api-tls\") pod \"maas-api-68786f5998-vrxsc\" (UID: \"fc3a7059-c909-4873-aaec-249711e7fa58\") " pod="odh-ai-gateway-infra/maas-api-68786f5998-vrxsc" Sep 10 16:18:45.318440 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:45.318420 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzkgv\" (UniqueName: \"kubernetes.io/projected/fc3a7059-c909-4873-aaec-249711e7fa58-kube-api-access-nzkgv\") pod \"maas-api-68786f5998-vrxsc\" (UID: \"fc3a7059-c909-4873-aaec-249711e7fa58\") " pod="odh-ai-gateway-infra/maas-api-68786f5998-vrxsc" Sep 10 16:18:45.430753 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:45.430725 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-68786f5998-vrxsc" Sep 10 16:18:45.713116 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:45.713041 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-68786f5998-vrxsc"] Sep 10 16:18:45.716358 ip-10-0-139-107 kubenswrapper[2583]: W0910 16:18:45.716331 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc3a7059_c909_4873_aaec_249711e7fa58.slice/crio-40709148898a44879933867e1b60eea008e95bab1d02953751c9ccd85faf8fa6 WatchSource:0}: Error finding container 40709148898a44879933867e1b60eea008e95bab1d02953751c9ccd85faf8fa6: Status 404 returned error can't find the container with id 40709148898a44879933867e1b60eea008e95bab1d02953751c9ccd85faf8fa6 Sep 10 16:18:45.717967 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:45.717951 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 10 16:18:45.810350 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:45.810323 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-68786f5998-vrxsc" event={"ID":"fc3a7059-c909-4873-aaec-249711e7fa58","Type":"ContainerStarted","Data":"40709148898a44879933867e1b60eea008e95bab1d02953751c9ccd85faf8fa6"} Sep 10 16:18:46.818060 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:46.818019 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-68786f5998-vrxsc" event={"ID":"fc3a7059-c909-4873-aaec-249711e7fa58","Type":"ContainerStarted","Data":"00305f03c85127916e3d54dff417cfc4e40803b09307e2b8368c8a067e440448"} Sep 10 16:18:46.818469 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:46.818157 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-68786f5998-vrxsc" Sep 10 16:18:46.856155 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:46.856090 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-68786f5998-vrxsc" podStartSLOduration=1.450148094 podStartE2EDuration="1.85607111s" totalImagesPullingTime="405.923016ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:18:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:18:46.84928173 +0000 UTC m=+2203.824541684" watchObservedRunningTime="2026-09-10 16:18:46.85607111 +0000 UTC m=+2203.831331065" Sep 10 16:18:52.829824 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:52.829795 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-68786f5998-vrxsc" Sep 10 16:18:52.886958 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:52.886923 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f5df8d666-pnkxh"] Sep 10 16:18:52.887204 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:52.887178 2583 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7f5df8d666-pnkxh" podUID="c8de5e93-f9f7-4073-a54f-ad64580068f4" containerName="maas-api" containerID="cri-o://82115493d97d9f91fee8a939efd7d10778ce27d618c83bededd12d93e5ba2f97" gracePeriod=30 Sep 10 16:18:53.326492 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:53.326455 2583 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-7f5df8d666-pnkxh" podUID="c8de5e93-f9f7-4073-a54f-ad64580068f4" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.38:8443/health\": dial tcp 10.134.0.38:8443: connect: connection refused" Sep 10 16:18:54.124515 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:54.124492 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f5df8d666-pnkxh" Sep 10 16:18:54.184394 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:54.184364 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8de5e93-f9f7-4073-a54f-ad64580068f4-metrics-certs\") pod \"c8de5e93-f9f7-4073-a54f-ad64580068f4\" (UID: \"c8de5e93-f9f7-4073-a54f-ad64580068f4\") " Sep 10 16:18:54.184557 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:54.184410 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swxrm\" (UniqueName: \"kubernetes.io/projected/c8de5e93-f9f7-4073-a54f-ad64580068f4-kube-api-access-swxrm\") pod \"c8de5e93-f9f7-4073-a54f-ad64580068f4\" (UID: \"c8de5e93-f9f7-4073-a54f-ad64580068f4\") " Sep 10 16:18:54.184557 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:54.184464 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8de5e93-f9f7-4073-a54f-ad64580068f4-maas-api-tls\") pod \"c8de5e93-f9f7-4073-a54f-ad64580068f4\" (UID: \"c8de5e93-f9f7-4073-a54f-ad64580068f4\") " Sep 10 16:18:54.184557 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:54.184495 2583 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8de5e93-f9f7-4073-a54f-ad64580068f4-openshift-service-ca\") pod \"c8de5e93-f9f7-4073-a54f-ad64580068f4\" (UID: \"c8de5e93-f9f7-4073-a54f-ad64580068f4\") " Sep 10 16:18:54.184941 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:54.184894 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8de5e93-f9f7-4073-a54f-ad64580068f4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c8de5e93-f9f7-4073-a54f-ad64580068f4" (UID: "c8de5e93-f9f7-4073-a54f-ad64580068f4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 10 16:18:54.186475 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:54.186447 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8de5e93-f9f7-4073-a54f-ad64580068f4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "c8de5e93-f9f7-4073-a54f-ad64580068f4" (UID: "c8de5e93-f9f7-4073-a54f-ad64580068f4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:18:54.186595 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:54.186505 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8de5e93-f9f7-4073-a54f-ad64580068f4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c8de5e93-f9f7-4073-a54f-ad64580068f4" (UID: "c8de5e93-f9f7-4073-a54f-ad64580068f4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 10 16:18:54.186595 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:54.186532 2583 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8de5e93-f9f7-4073-a54f-ad64580068f4-kube-api-access-swxrm" (OuterVolumeSpecName: "kube-api-access-swxrm") pod "c8de5e93-f9f7-4073-a54f-ad64580068f4" (UID: "c8de5e93-f9f7-4073-a54f-ad64580068f4"). InnerVolumeSpecName "kube-api-access-swxrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 10 16:18:54.286133 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:54.286077 2583 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8de5e93-f9f7-4073-a54f-ad64580068f4-openshift-service-ca\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:18:54.286133 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:54.286097 2583 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c8de5e93-f9f7-4073-a54f-ad64580068f4-metrics-certs\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:18:54.286133 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:54.286132 2583 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swxrm\" (UniqueName: \"kubernetes.io/projected/c8de5e93-f9f7-4073-a54f-ad64580068f4-kube-api-access-swxrm\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:18:54.286347 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:54.286147 2583 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c8de5e93-f9f7-4073-a54f-ad64580068f4-maas-api-tls\") on node \"ip-10-0-139-107.ec2.internal\" DevicePath \"\"" Sep 10 16:18:54.862390 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:54.862344 2583 generic.go:356] "Generic (PLEG): container finished" podID="c8de5e93-f9f7-4073-a54f-ad64580068f4" containerID="82115493d97d9f91fee8a939efd7d10778ce27d618c83bededd12d93e5ba2f97" exitCode=0 Sep 10 16:18:54.862623 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:54.862393 2583 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f5df8d666-pnkxh" Sep 10 16:18:54.862623 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:54.862419 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f5df8d666-pnkxh" event={"ID":"c8de5e93-f9f7-4073-a54f-ad64580068f4","Type":"ContainerDied","Data":"82115493d97d9f91fee8a939efd7d10778ce27d618c83bededd12d93e5ba2f97"} Sep 10 16:18:54.862623 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:54.862465 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f5df8d666-pnkxh" event={"ID":"c8de5e93-f9f7-4073-a54f-ad64580068f4","Type":"ContainerDied","Data":"4aa4c691cac2a2bc26b565113f2d038e07fc1adf1d9062ac139b6ccf9b33e9de"} Sep 10 16:18:54.862623 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:54.862489 2583 scope.go:117] "RemoveContainer" containerID="82115493d97d9f91fee8a939efd7d10778ce27d618c83bededd12d93e5ba2f97" Sep 10 16:18:54.872204 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:54.872187 2583 scope.go:117] "RemoveContainer" containerID="82115493d97d9f91fee8a939efd7d10778ce27d618c83bededd12d93e5ba2f97" Sep 10 16:18:54.872499 ip-10-0-139-107 kubenswrapper[2583]: E0910 16:18:54.872476 2583 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82115493d97d9f91fee8a939efd7d10778ce27d618c83bededd12d93e5ba2f97\": container with ID starting with 82115493d97d9f91fee8a939efd7d10778ce27d618c83bededd12d93e5ba2f97 not found: ID does not exist" containerID="82115493d97d9f91fee8a939efd7d10778ce27d618c83bededd12d93e5ba2f97" Sep 10 16:18:54.872605 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:54.872504 2583 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82115493d97d9f91fee8a939efd7d10778ce27d618c83bededd12d93e5ba2f97"} err="failed to get container status \"82115493d97d9f91fee8a939efd7d10778ce27d618c83bededd12d93e5ba2f97\": rpc error: code = NotFound desc = could not find container \"82115493d97d9f91fee8a939efd7d10778ce27d618c83bededd12d93e5ba2f97\": container with ID starting with 82115493d97d9f91fee8a939efd7d10778ce27d618c83bededd12d93e5ba2f97 not found: ID does not exist" Sep 10 16:18:54.885075 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:54.885052 2583 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f5df8d666-pnkxh"] Sep 10 16:18:54.889660 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:54.889639 2583 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f5df8d666-pnkxh"] Sep 10 16:18:55.515320 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:18:55.515289 2583 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8de5e93-f9f7-4073-a54f-ad64580068f4" path="/var/lib/kubelet/pods/c8de5e93-f9f7-4073-a54f-ad64580068f4/volumes" Sep 10 16:19:00.691688 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:00.691653 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:23.885649 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:23.885616 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-tlfmm_1c7f7317-43f6-43d7-b12d-769c7d0bb5e8/manager/0.log" Sep 10 16:19:24.255910 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:24.255835 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8fbcf4bcf-bt7tx_b2e49b1f-82b5-49ee-b670-8948f7ea4997/manager/0.log" Sep 10 16:19:24.265323 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:24.265304 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8fbcf4bcf-bt7tx_b2e49b1f-82b5-49ee-b670-8948f7ea4997/copy-manifests/0.log" Sep 10 16:19:24.739461 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:24.739428 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-68786f5998-vrxsc_fc3a7059-c909-4873-aaec-249711e7fa58/maas-api/0.log" Sep 10 16:19:26.871372 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:26.871303 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2_4aebac9a-c55c-48af-95fa-4c1431657fe7/istio-proxy/0.log" Sep 10 16:19:26.985194 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:26.985168 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-zdjj4_a33616ed-885f-4af2-9bea-eb1f9e9dcf3c/discovery/0.log" Sep 10 16:19:27.683014 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:27.682984 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-b4f467c4b-xjkv7_e6af5593-a983-4c35-95e9-3c166bdbf385/router/0.log" Sep 10 16:19:28.030633 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:28.030565 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-d4l8v_404be92e-4d9e-4070-a64f-ccf91ba25b8c/main/0.log" Sep 10 16:19:28.591895 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:28.591866 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-v4knc_df2ec437-8304-45bd-80c7-a4f6eaa67e33/main/0.log" Sep 10 16:19:33.718876 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:33.718846 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:33.819256 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:33.819225 2583 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" limit=1024 Sep 10 16:19:36.704462 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:36.704432 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-swxpd_bdf61327-2614-4068-974c-a5c0f7c8e3e0/global-pull-secret-syncer/0.log" Sep 10 16:19:36.803611 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:36.803579 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mcx2v_3448a418-7ccc-4a41-b2f0-0e1fa12cd592/konnectivity-agent/0.log" Sep 10 16:19:36.963821 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:36.963741 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-107.ec2.internal_ef65ebb74ba3516cc857fe0f9e4cef0a/haproxy/0.log" Sep 10 16:19:43.408950 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:43.408920 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_590890d6-a6d7-4868-b14e-f02eb06f758a/alertmanager/0.log" Sep 10 16:19:43.450757 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:43.450730 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_590890d6-a6d7-4868-b14e-f02eb06f758a/config-reloader/0.log" Sep 10 16:19:43.484641 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:43.484616 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_590890d6-a6d7-4868-b14e-f02eb06f758a/kube-rbac-proxy-web/0.log" Sep 10 16:19:43.515225 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:43.515195 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_590890d6-a6d7-4868-b14e-f02eb06f758a/kube-rbac-proxy/0.log" Sep 10 16:19:43.540522 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:43.540497 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_590890d6-a6d7-4868-b14e-f02eb06f758a/kube-rbac-proxy-metric/0.log" Sep 10 16:19:43.570291 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:43.570264 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_590890d6-a6d7-4868-b14e-f02eb06f758a/prom-label-proxy/0.log" Sep 10 16:19:43.596630 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:43.596608 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_590890d6-a6d7-4868-b14e-f02eb06f758a/init-config-reloader/0.log" Sep 10 16:19:43.772021 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:43.771954 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-f9bdf75f9-r2swd_27b3c4fd-2cc4-4298-8ebc-0d93e73097f8/metrics-server/0.log" Sep 10 16:19:43.803145 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:43.803123 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-t4sw4_d807c2fa-db5f-4003-8e8d-0938f21950d5/monitoring-plugin/0.log" Sep 10 16:19:43.846285 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:43.846261 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8hcsc_8b0d03d5-0328-428f-94cf-8f57dba68534/node-exporter/0.log" Sep 10 16:19:43.873479 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:43.873462 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8hcsc_8b0d03d5-0328-428f-94cf-8f57dba68534/kube-rbac-proxy/0.log" Sep 10 16:19:43.898860 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:43.898845 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8hcsc_8b0d03d5-0328-428f-94cf-8f57dba68534/init-textfile/0.log" Sep 10 16:19:44.182761 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:44.182733 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-kzbh5_29dc562d-f7be-4903-bb3a-23ee7b84d3cf/kube-rbac-proxy-main/0.log" Sep 10 16:19:44.226540 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:44.226515 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-kzbh5_29dc562d-f7be-4903-bb3a-23ee7b84d3cf/kube-rbac-proxy-self/0.log" Sep 10 16:19:44.261902 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:44.261882 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-kzbh5_29dc562d-f7be-4903-bb3a-23ee7b84d3cf/openshift-state-metrics/0.log" Sep 10 16:19:44.563675 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:44.563643 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-b8ntr_1fe37db8-3301-406d-b0d8-241dec30c632/prometheus-operator/0.log" Sep 10 16:19:44.610811 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:44.610784 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-6f4946db56-b8ntr_1fe37db8-3301-406d-b0d8-241dec30c632/kube-rbac-proxy/0.log" Sep 10 16:19:44.713535 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:44.713502 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-dcd5cd59c-xhqp9_a1907b32-48aa-4b25-a41e-45c8201a69dd/telemeter-client/0.log" Sep 10 16:19:44.742716 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:44.742690 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-dcd5cd59c-xhqp9_a1907b32-48aa-4b25-a41e-45c8201a69dd/reload/0.log" Sep 10 16:19:44.768712 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:44.768692 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-dcd5cd59c-xhqp9_a1907b32-48aa-4b25-a41e-45c8201a69dd/kube-rbac-proxy/0.log" Sep 10 16:19:44.810710 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:44.810692 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f68bcbf87-57tzb_24f85477-eb7b-4fcc-8c84-d9e89d841b57/thanos-query/0.log" Sep 10 16:19:44.849566 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:44.849516 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f68bcbf87-57tzb_24f85477-eb7b-4fcc-8c84-d9e89d841b57/kube-rbac-proxy-web/0.log" Sep 10 16:19:44.881523 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:44.881500 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f68bcbf87-57tzb_24f85477-eb7b-4fcc-8c84-d9e89d841b57/kube-rbac-proxy/0.log" Sep 10 16:19:44.911560 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:44.911538 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f68bcbf87-57tzb_24f85477-eb7b-4fcc-8c84-d9e89d841b57/prom-label-proxy/0.log" Sep 10 16:19:44.941634 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:44.941606 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f68bcbf87-57tzb_24f85477-eb7b-4fcc-8c84-d9e89d841b57/kube-rbac-proxy-rules/0.log" Sep 10 16:19:44.970173 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:44.970154 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5f68bcbf87-57tzb_24f85477-eb7b-4fcc-8c84-d9e89d841b57/kube-rbac-proxy-metrics/0.log" Sep 10 16:19:47.018331 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:47.018298 2583 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2t5gt/perf-node-gather-daemonset-w4ksx"] Sep 10 16:19:47.018719 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:47.018688 2583 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8de5e93-f9f7-4073-a54f-ad64580068f4" containerName="maas-api" Sep 10 16:19:47.018719 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:47.018700 2583 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8de5e93-f9f7-4073-a54f-ad64580068f4" containerName="maas-api" Sep 10 16:19:47.018795 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:47.018761 2583 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8de5e93-f9f7-4073-a54f-ad64580068f4" containerName="maas-api" Sep 10 16:19:47.021945 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:47.021930 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-w4ksx" Sep 10 16:19:47.024331 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:47.024310 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2t5gt\"/\"openshift-service-ca.crt\"" Sep 10 16:19:47.025071 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:47.025052 2583 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2t5gt\"/\"kube-root-ca.crt\"" Sep 10 16:19:47.025141 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:47.025122 2583 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2t5gt\"/\"default-dockercfg-xmrs6\"" Sep 10 16:19:47.034779 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:47.034749 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2t5gt/perf-node-gather-daemonset-w4ksx"] Sep 10 16:19:47.130592 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:47.130560 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b563bd83-ae31-40ea-8b8b-4f0ceef12747-lib-modules\") pod \"perf-node-gather-daemonset-w4ksx\" (UID: \"b563bd83-ae31-40ea-8b8b-4f0ceef12747\") " pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-w4ksx" Sep 10 16:19:47.130592 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:47.130598 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b563bd83-ae31-40ea-8b8b-4f0ceef12747-podres\") pod \"perf-node-gather-daemonset-w4ksx\" (UID: \"b563bd83-ae31-40ea-8b8b-4f0ceef12747\") " pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-w4ksx" Sep 10 16:19:47.130819 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:47.130645 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpqqw\" (UniqueName: \"kubernetes.io/projected/b563bd83-ae31-40ea-8b8b-4f0ceef12747-kube-api-access-bpqqw\") pod \"perf-node-gather-daemonset-w4ksx\" (UID: \"b563bd83-ae31-40ea-8b8b-4f0ceef12747\") " pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-w4ksx" Sep 10 16:19:47.130819 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:47.130702 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b563bd83-ae31-40ea-8b8b-4f0ceef12747-sys\") pod \"perf-node-gather-daemonset-w4ksx\" (UID: \"b563bd83-ae31-40ea-8b8b-4f0ceef12747\") " pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-w4ksx" Sep 10 16:19:47.130819 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:47.130722 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b563bd83-ae31-40ea-8b8b-4f0ceef12747-proc\") pod \"perf-node-gather-daemonset-w4ksx\" (UID: \"b563bd83-ae31-40ea-8b8b-4f0ceef12747\") " pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-w4ksx" Sep 10 16:19:47.231605 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:47.231568 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b563bd83-ae31-40ea-8b8b-4f0ceef12747-proc\") pod \"perf-node-gather-daemonset-w4ksx\" (UID: \"b563bd83-ae31-40ea-8b8b-4f0ceef12747\") " pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-w4ksx" Sep 10 16:19:47.231787 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:47.231621 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b563bd83-ae31-40ea-8b8b-4f0ceef12747-lib-modules\") pod \"perf-node-gather-daemonset-w4ksx\" (UID: \"b563bd83-ae31-40ea-8b8b-4f0ceef12747\") " pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-w4ksx" Sep 10 16:19:47.231787 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:47.231651 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b563bd83-ae31-40ea-8b8b-4f0ceef12747-podres\") pod \"perf-node-gather-daemonset-w4ksx\" (UID: \"b563bd83-ae31-40ea-8b8b-4f0ceef12747\") " pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-w4ksx" Sep 10 16:19:47.231787 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:47.231701 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpqqw\" (UniqueName: \"kubernetes.io/projected/b563bd83-ae31-40ea-8b8b-4f0ceef12747-kube-api-access-bpqqw\") pod \"perf-node-gather-daemonset-w4ksx\" (UID: \"b563bd83-ae31-40ea-8b8b-4f0ceef12747\") " pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-w4ksx" Sep 10 16:19:47.231787 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:47.231698 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b563bd83-ae31-40ea-8b8b-4f0ceef12747-proc\") pod \"perf-node-gather-daemonset-w4ksx\" (UID: \"b563bd83-ae31-40ea-8b8b-4f0ceef12747\") " pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-w4ksx" Sep 10 16:19:47.232002 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:47.231812 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b563bd83-ae31-40ea-8b8b-4f0ceef12747-sys\") pod \"perf-node-gather-daemonset-w4ksx\" (UID: \"b563bd83-ae31-40ea-8b8b-4f0ceef12747\") " pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-w4ksx" Sep 10 16:19:47.232002 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:47.231820 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b563bd83-ae31-40ea-8b8b-4f0ceef12747-lib-modules\") pod \"perf-node-gather-daemonset-w4ksx\" (UID: \"b563bd83-ae31-40ea-8b8b-4f0ceef12747\") " pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-w4ksx" Sep 10 16:19:47.232002 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:47.231854 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b563bd83-ae31-40ea-8b8b-4f0ceef12747-sys\") pod \"perf-node-gather-daemonset-w4ksx\" (UID: \"b563bd83-ae31-40ea-8b8b-4f0ceef12747\") " pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-w4ksx" Sep 10 16:19:47.232002 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:47.231810 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b563bd83-ae31-40ea-8b8b-4f0ceef12747-podres\") pod \"perf-node-gather-daemonset-w4ksx\" (UID: \"b563bd83-ae31-40ea-8b8b-4f0ceef12747\") " pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-w4ksx" Sep 10 16:19:47.243253 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:47.243226 2583 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpqqw\" (UniqueName: \"kubernetes.io/projected/b563bd83-ae31-40ea-8b8b-4f0ceef12747-kube-api-access-bpqqw\") pod \"perf-node-gather-daemonset-w4ksx\" (UID: \"b563bd83-ae31-40ea-8b8b-4f0ceef12747\") " pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-w4ksx" Sep 10 16:19:47.332440 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:47.332344 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-w4ksx" Sep 10 16:19:47.604395 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:47.604348 2583 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2t5gt/perf-node-gather-daemonset-w4ksx"] Sep 10 16:19:47.606419 ip-10-0-139-107 kubenswrapper[2583]: W0910 16:19:47.606368 2583 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb563bd83_ae31_40ea_8b8b_4f0ceef12747.slice/crio-31c390d2de3285805a3cb9edd80c37e05e7b8650669b5c091d77a153160c4cea WatchSource:0}: Error finding container 31c390d2de3285805a3cb9edd80c37e05e7b8650669b5c091d77a153160c4cea: Status 404 returned error can't find the container with id 31c390d2de3285805a3cb9edd80c37e05e7b8650669b5c091d77a153160c4cea Sep 10 16:19:48.153880 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:48.153846 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-w4ksx" event={"ID":"b563bd83-ae31-40ea-8b8b-4f0ceef12747","Type":"ContainerStarted","Data":"abaf716e3fcb65bb9eaed2a4f8d78263ea1ec28047df1603eb795931333b4aa5"} Sep 10 16:19:48.154292 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:48.153896 2583 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-w4ksx" event={"ID":"b563bd83-ae31-40ea-8b8b-4f0ceef12747","Type":"ContainerStarted","Data":"31c390d2de3285805a3cb9edd80c37e05e7b8650669b5c091d77a153160c4cea"} Sep 10 16:19:48.154292 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:48.153990 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-w4ksx" Sep 10 16:19:48.175195 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:48.175152 2583 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-w4ksx" podStartSLOduration=2.175139404 podStartE2EDuration="2.175139404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 16:19:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:19:48.174715054 +0000 UTC m=+2265.149975008" watchObservedRunningTime="2026-09-10 16:19:48.175139404 +0000 UTC m=+2265.150399358" Sep 10 16:19:49.683596 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:49.683570 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-45ph9_8681b681-0371-4b9d-8c1d-82a431bd3e62/dns/0.log" Sep 10 16:19:49.708371 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:49.708350 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-45ph9_8681b681-0371-4b9d-8c1d-82a431bd3e62/kube-rbac-proxy/0.log" Sep 10 16:19:49.817333 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:49.817314 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8v784_cb7626aa-a9d1-4dec-aab8-b68a3e6adca3/dns-node-resolver/0.log" Sep 10 16:19:50.435478 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:50.435448 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-5lpqq_7dc7aa02-57e8-4cdc-aab6-9875e1427825/node-ca/0.log" Sep 10 16:19:51.457330 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:51.457304 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2_4aebac9a-c55c-48af-95fa-4c1431657fe7/istio-proxy/0.log" Sep 10 16:19:51.693483 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:51.693449 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-zdjj4_a33616ed-885f-4af2-9bea-eb1f9e9dcf3c/discovery/0.log" Sep 10 16:19:51.930431 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:51.930403 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-b4f467c4b-xjkv7_e6af5593-a983-4c35-95e9-3c166bdbf385/router/0.log" Sep 10 16:19:52.904265 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:52.904216 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wqwzs_66ac0bb7-d8cd-43d7-b247-4d5e441f321a/serve-healthcheck-canary/0.log" Sep 10 16:19:53.490953 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:53.490921 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8wtqm_b1949768-9a66-4404-b7f1-c71ae6520f36/kube-rbac-proxy/0.log" Sep 10 16:19:53.516562 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:53.516533 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8wtqm_b1949768-9a66-4404-b7f1-c71ae6520f36/exporter/0.log" Sep 10 16:19:53.540298 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:53.540273 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8wtqm_b1949768-9a66-4404-b7f1-c71ae6520f36/extractor/0.log" Sep 10 16:19:54.168932 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:54.168908 2583 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-w4ksx" Sep 10 16:19:56.203236 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:56.203161 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-tlfmm_1c7f7317-43f6-43d7-b12d-769c7d0bb5e8/manager/0.log" Sep 10 16:19:56.421865 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:56.421836 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8fbcf4bcf-bt7tx_b2e49b1f-82b5-49ee-b670-8948f7ea4997/manager/0.log" Sep 10 16:19:56.445819 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:56.445799 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8fbcf4bcf-bt7tx_b2e49b1f-82b5-49ee-b670-8948f7ea4997/copy-manifests/0.log" Sep 10 16:19:57.670104 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:57.670074 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2_4aebac9a-c55c-48af-95fa-4c1431657fe7/istio-proxy/0.log" Sep 10 16:19:57.792413 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:57.792369 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-f8dc47f5d-rwzg9_9fce4356-4a66-4635-b017-ec95c65229ae/manager/0.log" Sep 10 16:19:57.847349 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:57.847312 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-tktqk_881c5d57-02e1-4ef1-89be-2c4b0a6a0b48/openshift-lws-operator/0.log" Sep 10 16:19:57.894037 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:57.894011 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-zdjj4_a33616ed-885f-4af2-9bea-eb1f9e9dcf3c/discovery/0.log" Sep 10 16:19:58.026632 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:58.026549 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-b4f467c4b-xjkv7_e6af5593-a983-4c35-95e9-3c166bdbf385/router/0.log" Sep 10 16:19:59.538067 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:59.538039 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-jn8n2_4aebac9a-c55c-48af-95fa-4c1431657fe7/istio-proxy/0.log" Sep 10 16:19:59.785169 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:59.785139 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-zdjj4_a33616ed-885f-4af2-9bea-eb1f9e9dcf3c/discovery/0.log" Sep 10 16:19:59.928947 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:19:59.928921 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-b4f467c4b-xjkv7_e6af5593-a983-4c35-95e9-3c166bdbf385/router/0.log" Sep 10 16:20:03.086655 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:20:03.086570 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-8v6zg_815141e9-52d6-47c9-9c59-71e6f3fc385c/migrator/0.log" Sep 10 16:20:03.111292 ip-10-0-139-107 kubenswrapper[2583]: I0910 16:20:03.111262 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-8v6zg_815141e9-52d6-47c9-9c59-71e6f3fc385c/graceful-termination/0.log"