Sep 11 03:53:05.145322 ip-10-0-140-30 systemd[1]: Starting Kubernetes Kubelet... Sep 11 03:53:05.565535 ip-10-0-140-30 kubenswrapper[2568]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 03:53:05.565535 ip-10-0-140-30 kubenswrapper[2568]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 11 03:53:05.565535 ip-10-0-140-30 kubenswrapper[2568]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 03:53:05.565535 ip-10-0-140-30 kubenswrapper[2568]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 11 03:53:05.565535 ip-10-0-140-30 kubenswrapper[2568]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 03:53:05.568108 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.568005 2568 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 11 03:53:05.572165 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572150 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 03:53:05.572165 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572165 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 03:53:05.572228 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572168 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 03:53:05.572228 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572172 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 03:53:05.572228 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572175 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 03:53:05.572228 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572178 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 03:53:05.572228 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572181 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 03:53:05.572228 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572184 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 03:53:05.572228 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572188 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 03:53:05.572228 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572190 2568 feature_gate.go:328] unrecognized feature gate: Example Sep 11 03:53:05.572228 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572193 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 03:53:05.572228 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572196 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 03:53:05.572228 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572198 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 03:53:05.572228 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572201 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 03:53:05.572228 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572203 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 03:53:05.572228 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572207 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 03:53:05.572228 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572209 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 03:53:05.572228 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572212 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 03:53:05.572228 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572215 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 03:53:05.572228 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572222 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 03:53:05.572228 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572225 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 03:53:05.572228 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572228 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 03:53:05.572693 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572230 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 03:53:05.572693 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572233 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 03:53:05.572693 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572236 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 03:53:05.572693 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572239 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 03:53:05.572693 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572241 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 03:53:05.572693 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572244 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 03:53:05.572693 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572247 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 03:53:05.572693 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572249 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 03:53:05.572693 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572252 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 03:53:05.572693 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572254 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 03:53:05.572693 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572257 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 03:53:05.572693 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572260 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 03:53:05.572693 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572262 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 03:53:05.572693 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572265 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 03:53:05.572693 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572267 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 03:53:05.572693 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572270 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 03:53:05.572693 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572272 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 03:53:05.572693 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572275 2568 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 03:53:05.572693 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572278 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 03:53:05.573166 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572282 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 03:53:05.573166 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572285 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 03:53:05.573166 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572288 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 03:53:05.573166 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572290 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 03:53:05.573166 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572293 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 03:53:05.573166 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572296 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 03:53:05.573166 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572299 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 03:53:05.573166 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572301 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 03:53:05.573166 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572304 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 03:53:05.573166 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572307 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 03:53:05.573166 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572309 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 03:53:05.573166 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572311 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 03:53:05.573166 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572314 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 03:53:05.573166 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572317 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 03:53:05.573166 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572321 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 03:53:05.573166 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572324 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 03:53:05.573166 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572327 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 03:53:05.573166 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572329 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 03:53:05.573166 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572332 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 03:53:05.573633 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572334 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 03:53:05.573633 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572337 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 03:53:05.573633 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572340 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 03:53:05.573633 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572342 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 03:53:05.573633 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572345 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 03:53:05.573633 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572347 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 03:53:05.573633 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572350 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 03:53:05.573633 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572352 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 03:53:05.573633 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572354 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 03:53:05.573633 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572357 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 03:53:05.573633 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572359 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 03:53:05.573633 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572362 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 03:53:05.573633 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572365 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 03:53:05.573633 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572367 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 03:53:05.573633 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572370 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 03:53:05.573633 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572372 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 03:53:05.573633 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572375 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 03:53:05.573633 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572377 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 03:53:05.573633 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572380 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 03:53:05.573633 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572382 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 03:53:05.574118 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572385 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 03:53:05.574118 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572387 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 03:53:05.574118 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572390 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 03:53:05.574118 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572394 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 03:53:05.574118 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572397 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 03:53:05.574118 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572400 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 03:53:05.574118 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572403 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 03:53:05.574118 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572407 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 03:53:05.574118 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572411 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 03:53:05.574118 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572413 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 03:53:05.574118 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572416 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 03:53:05.574118 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572419 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 03:53:05.574118 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572422 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 03:53:05.574118 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572424 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 03:53:05.574118 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572427 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 03:53:05.574118 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572430 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 03:53:05.574118 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572432 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 03:53:05.574118 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572435 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 03:53:05.574118 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.572438 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 03:53:05.574583 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573122 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 03:53:05.574583 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573130 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 03:53:05.574583 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573133 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 03:53:05.574583 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573136 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 03:53:05.574583 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573139 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 03:53:05.574583 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573142 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 03:53:05.574583 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573145 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 03:53:05.574583 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573148 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 03:53:05.574583 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573151 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 03:53:05.574583 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573153 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 03:53:05.574583 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573158 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 03:53:05.574583 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573162 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 03:53:05.574583 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573165 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 03:53:05.574583 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573168 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 03:53:05.574583 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573172 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 03:53:05.574583 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573175 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 03:53:05.574583 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573178 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 03:53:05.574583 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573182 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 03:53:05.574583 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573184 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 03:53:05.574583 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573187 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 03:53:05.575101 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573191 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 03:53:05.575101 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573194 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 03:53:05.575101 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573197 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 03:53:05.575101 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573200 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 03:53:05.575101 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573203 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 03:53:05.575101 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573205 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 03:53:05.575101 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573208 2568 feature_gate.go:328] unrecognized feature gate: Example Sep 11 03:53:05.575101 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573210 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 03:53:05.575101 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573213 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 03:53:05.575101 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573215 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 03:53:05.575101 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573218 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 03:53:05.575101 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573221 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 03:53:05.575101 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573223 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 03:53:05.575101 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573226 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 03:53:05.575101 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573228 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 03:53:05.575101 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573231 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 03:53:05.575101 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573235 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 03:53:05.575101 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573237 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 03:53:05.575101 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573240 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 03:53:05.575101 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573243 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 03:53:05.575587 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573246 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 03:53:05.575587 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573248 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 03:53:05.575587 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573250 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 03:53:05.575587 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573254 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 03:53:05.575587 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573256 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 03:53:05.575587 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573259 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 03:53:05.575587 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573261 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 03:53:05.575587 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573264 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 03:53:05.575587 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573266 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 03:53:05.575587 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573269 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 03:53:05.575587 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573272 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 03:53:05.575587 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573275 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 03:53:05.575587 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573278 2568 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 03:53:05.575587 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573281 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 03:53:05.575587 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573283 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 03:53:05.575587 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573286 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 03:53:05.575587 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573289 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 03:53:05.575587 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573291 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 03:53:05.575587 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573294 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 03:53:05.575587 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573296 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 03:53:05.576082 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573299 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 03:53:05.576082 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573301 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 03:53:05.576082 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573304 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 03:53:05.576082 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573307 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 03:53:05.576082 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573309 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 03:53:05.576082 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573312 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 03:53:05.576082 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573314 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 03:53:05.576082 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573317 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 03:53:05.576082 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573320 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 03:53:05.576082 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573322 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 03:53:05.576082 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573325 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 03:53:05.576082 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573328 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 03:53:05.576082 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573332 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 03:53:05.576082 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573334 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 03:53:05.576082 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573337 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 03:53:05.576082 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573339 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 03:53:05.576082 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573342 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 03:53:05.576082 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573344 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 03:53:05.576082 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573347 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 03:53:05.576082 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573349 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 03:53:05.576575 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573352 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 03:53:05.576575 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573354 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 03:53:05.576575 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573357 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 03:53:05.576575 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573359 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 03:53:05.576575 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573362 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 03:53:05.576575 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573365 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 03:53:05.576575 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573369 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 03:53:05.576575 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573372 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 03:53:05.576575 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573375 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 03:53:05.576575 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573379 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 03:53:05.576575 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573382 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 03:53:05.576575 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573385 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 03:53:05.576575 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573388 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 03:53:05.576575 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573390 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 03:53:05.576575 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573393 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 03:53:05.576575 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573396 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 03:53:05.576575 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573398 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 03:53:05.576575 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573401 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 03:53:05.577037 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573403 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 03:53:05.577037 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573780 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 03:53:05.577037 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573785 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 03:53:05.577037 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573788 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 03:53:05.577037 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573791 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 03:53:05.577037 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573799 2568 feature_gate.go:328] unrecognized feature gate: Example Sep 11 03:53:05.577037 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573802 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 03:53:05.577037 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573805 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 03:53:05.577037 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573808 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 03:53:05.577037 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573810 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 03:53:05.577037 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573813 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 03:53:05.577037 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573815 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 03:53:05.577037 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573817 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 03:53:05.577037 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573820 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 03:53:05.577037 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573823 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 03:53:05.577037 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573826 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 03:53:05.577037 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573828 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 03:53:05.577037 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573831 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 03:53:05.577037 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573834 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 03:53:05.577610 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573837 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 03:53:05.577610 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573840 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 03:53:05.577610 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573842 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 03:53:05.577610 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573845 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 03:53:05.577610 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573847 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 03:53:05.577610 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573850 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 03:53:05.577610 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573853 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 03:53:05.577610 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573855 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 03:53:05.577610 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573858 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 03:53:05.577610 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573860 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 03:53:05.577610 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573863 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 03:53:05.577610 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573865 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 03:53:05.577610 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573868 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 03:53:05.577610 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573870 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 03:53:05.577610 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573873 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 03:53:05.577610 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573876 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 03:53:05.577610 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573880 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 03:53:05.577610 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573884 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 03:53:05.577610 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573887 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 03:53:05.578118 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573897 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 03:53:05.578118 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573900 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 03:53:05.578118 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573903 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 03:53:05.578118 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573905 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 03:53:05.578118 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573908 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 03:53:05.578118 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573911 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 03:53:05.578118 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573913 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 03:53:05.578118 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573916 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 03:53:05.578118 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573919 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 03:53:05.578118 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573922 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 03:53:05.578118 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573925 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 03:53:05.578118 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573927 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 03:53:05.578118 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573930 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 03:53:05.578118 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573933 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 03:53:05.578118 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573935 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 03:53:05.578118 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573938 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 03:53:05.578118 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573940 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 03:53:05.578118 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573943 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 03:53:05.578118 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573945 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 03:53:05.578118 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573948 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 03:53:05.578607 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573950 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 03:53:05.578607 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573952 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 03:53:05.578607 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573955 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 03:53:05.578607 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573958 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 03:53:05.578607 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573960 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 03:53:05.578607 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573963 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 03:53:05.578607 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573968 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 03:53:05.578607 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573970 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 03:53:05.578607 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573972 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 03:53:05.578607 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573975 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 03:53:05.578607 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573977 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 03:53:05.578607 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573980 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 03:53:05.578607 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573982 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 03:53:05.578607 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573990 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 03:53:05.578607 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573993 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 03:53:05.578607 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573996 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 03:53:05.578607 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.573998 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 03:53:05.578607 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574001 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 03:53:05.578607 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574003 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 03:53:05.578607 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574006 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 03:53:05.579107 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574008 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 03:53:05.579107 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574011 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 03:53:05.579107 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574014 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 03:53:05.579107 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574016 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 03:53:05.579107 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574021 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 03:53:05.579107 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574024 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 03:53:05.579107 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574027 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 03:53:05.579107 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574031 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 03:53:05.579107 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574034 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 03:53:05.579107 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574037 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 03:53:05.579107 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574040 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 03:53:05.579107 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574054 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 03:53:05.579107 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574058 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 03:53:05.579107 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574060 2568 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 03:53:05.579107 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574063 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 03:53:05.579107 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574067 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 03:53:05.579107 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574069 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 03:53:05.579107 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574072 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 03:53:05.579107 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574076 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 03:53:05.579609 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574079 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 03:53:05.579609 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574082 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 03:53:05.579609 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574084 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 03:53:05.579609 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574164 2568 flags.go:64] FLAG: --address="0.0.0.0" Sep 11 03:53:05.579609 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574209 2568 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 11 03:53:05.579609 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574221 2568 flags.go:64] FLAG: --anonymous-auth="true" Sep 11 03:53:05.579609 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574225 2568 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 11 03:53:05.579609 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574237 2568 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 11 03:53:05.579609 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574240 2568 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 11 03:53:05.579609 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574245 2568 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 11 03:53:05.579609 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574249 2568 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 11 03:53:05.579609 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574252 2568 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 11 03:53:05.579609 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574255 2568 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 11 03:53:05.579609 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574259 2568 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 11 03:53:05.579609 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574262 2568 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 11 03:53:05.579609 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574265 2568 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 11 03:53:05.579609 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574268 2568 flags.go:64] FLAG: --cgroup-root="" Sep 11 03:53:05.579609 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574271 2568 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 11 03:53:05.579609 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574274 2568 flags.go:64] FLAG: --client-ca-file="" Sep 11 03:53:05.579609 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574276 2568 flags.go:64] FLAG: --cloud-provider="external" Sep 11 03:53:05.579609 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574280 2568 flags.go:64] FLAG: --cluster-dns="[]" Sep 11 03:53:05.579609 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574286 2568 flags.go:64] FLAG: --cluster-domain="" Sep 11 03:53:05.579609 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574289 2568 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 11 03:53:05.579609 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574292 2568 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 11 03:53:05.580211 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574295 2568 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 11 03:53:05.580211 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574298 2568 flags.go:64] FLAG: --container-log-max-files="5" Sep 11 03:53:05.580211 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574301 2568 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 11 03:53:05.580211 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574304 2568 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 11 03:53:05.580211 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574307 2568 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 11 03:53:05.580211 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574311 2568 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 11 03:53:05.580211 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574314 2568 flags.go:64] FLAG: --contention-profiling="false" Sep 11 03:53:05.580211 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574317 2568 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 11 03:53:05.580211 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574320 2568 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 11 03:53:05.580211 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574323 2568 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 11 03:53:05.580211 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574326 2568 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 11 03:53:05.580211 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574330 2568 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 11 03:53:05.580211 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574333 2568 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 11 03:53:05.580211 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574335 2568 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 11 03:53:05.580211 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574339 2568 flags.go:64] FLAG: --enable-load-reader="false" Sep 11 03:53:05.580211 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574342 2568 flags.go:64] FLAG: --enable-server="true" Sep 11 03:53:05.580211 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574350 2568 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 11 03:53:05.580211 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574361 2568 flags.go:64] FLAG: --event-burst="100" Sep 11 03:53:05.580211 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574364 2568 flags.go:64] FLAG: --event-qps="50" Sep 11 03:53:05.580211 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574367 2568 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 11 03:53:05.580211 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574370 2568 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 11 03:53:05.580211 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574373 2568 flags.go:64] FLAG: --eviction-hard="" Sep 11 03:53:05.580211 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574377 2568 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 11 03:53:05.580211 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574379 2568 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 11 03:53:05.580843 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574382 2568 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 11 03:53:05.580843 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574386 2568 flags.go:64] FLAG: --eviction-soft="" Sep 11 03:53:05.580843 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574389 2568 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 11 03:53:05.580843 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574392 2568 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 11 03:53:05.580843 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574395 2568 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 11 03:53:05.580843 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574397 2568 flags.go:64] FLAG: --experimental-mounter-path="" Sep 11 03:53:05.580843 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574400 2568 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 11 03:53:05.580843 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574403 2568 flags.go:64] FLAG: --fail-swap-on="true" Sep 11 03:53:05.580843 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574406 2568 flags.go:64] FLAG: --feature-gates="" Sep 11 03:53:05.580843 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574412 2568 flags.go:64] FLAG: --file-check-frequency="20s" Sep 11 03:53:05.580843 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574414 2568 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 11 03:53:05.580843 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574418 2568 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 11 03:53:05.580843 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574421 2568 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 11 03:53:05.580843 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574424 2568 flags.go:64] FLAG: --healthz-port="10248" Sep 11 03:53:05.580843 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574426 2568 flags.go:64] FLAG: --help="false" Sep 11 03:53:05.580843 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574430 2568 flags.go:64] FLAG: --hostname-override="ip-10-0-140-30.ec2.internal" Sep 11 03:53:05.580843 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574433 2568 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 11 03:53:05.580843 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574436 2568 flags.go:64] FLAG: --http-check-frequency="20s" Sep 11 03:53:05.580843 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574439 2568 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 11 03:53:05.580843 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574442 2568 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 11 03:53:05.580843 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574446 2568 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 11 03:53:05.580843 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574449 2568 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 11 03:53:05.580843 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574452 2568 flags.go:64] FLAG: --image-service-endpoint="" Sep 11 03:53:05.581445 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574454 2568 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 11 03:53:05.581445 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574457 2568 flags.go:64] FLAG: --kube-api-burst="100" Sep 11 03:53:05.581445 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574466 2568 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 11 03:53:05.581445 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574469 2568 flags.go:64] FLAG: --kube-api-qps="50" Sep 11 03:53:05.581445 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574472 2568 flags.go:64] FLAG: --kube-reserved="" Sep 11 03:53:05.581445 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574475 2568 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 11 03:53:05.581445 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574478 2568 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 11 03:53:05.581445 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574481 2568 flags.go:64] FLAG: --kubelet-cgroups="" Sep 11 03:53:05.581445 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574484 2568 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 11 03:53:05.581445 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574486 2568 flags.go:64] FLAG: --lock-file="" Sep 11 03:53:05.581445 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574489 2568 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 11 03:53:05.581445 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574492 2568 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 11 03:53:05.581445 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574495 2568 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 11 03:53:05.581445 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574500 2568 flags.go:64] FLAG: --log-json-split-stream="false" Sep 11 03:53:05.581445 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574503 2568 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 11 03:53:05.581445 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574506 2568 flags.go:64] FLAG: --log-text-split-stream="false" Sep 11 03:53:05.581445 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574509 2568 flags.go:64] FLAG: --logging-format="text" Sep 11 03:53:05.581445 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574512 2568 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 11 03:53:05.581445 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574515 2568 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 11 03:53:05.581445 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574518 2568 flags.go:64] FLAG: --manifest-url="" Sep 11 03:53:05.581445 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574520 2568 flags.go:64] FLAG: --manifest-url-header="" Sep 11 03:53:05.581445 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574525 2568 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 11 03:53:05.581445 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574528 2568 flags.go:64] FLAG: --max-open-files="1000000" Sep 11 03:53:05.581445 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574532 2568 flags.go:64] FLAG: --max-pods="110" Sep 11 03:53:05.581445 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574535 2568 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 11 03:53:05.582064 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574538 2568 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 11 03:53:05.582064 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574541 2568 flags.go:64] FLAG: --memory-manager-policy="None" Sep 11 03:53:05.582064 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574544 2568 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 11 03:53:05.582064 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574547 2568 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 11 03:53:05.582064 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574550 2568 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 11 03:53:05.582064 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574553 2568 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 11 03:53:05.582064 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574560 2568 flags.go:64] FLAG: --node-status-max-images="50" Sep 11 03:53:05.582064 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574563 2568 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 11 03:53:05.582064 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574566 2568 flags.go:64] FLAG: --oom-score-adj="-999" Sep 11 03:53:05.582064 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574569 2568 flags.go:64] FLAG: --pod-cidr="" Sep 11 03:53:05.582064 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574577 2568 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 11 03:53:05.582064 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574583 2568 flags.go:64] FLAG: --pod-manifest-path="" Sep 11 03:53:05.582064 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574586 2568 flags.go:64] FLAG: --pod-max-pids="-1" Sep 11 03:53:05.582064 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574589 2568 flags.go:64] FLAG: --pods-per-core="0" Sep 11 03:53:05.582064 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574592 2568 flags.go:64] FLAG: --port="10250" Sep 11 03:53:05.582064 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574595 2568 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 11 03:53:05.582064 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574598 2568 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0eaf8c3df5e012530" Sep 11 03:53:05.582064 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574601 2568 flags.go:64] FLAG: --qos-reserved="" Sep 11 03:53:05.582064 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574604 2568 flags.go:64] FLAG: --read-only-port="10255" Sep 11 03:53:05.582064 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574607 2568 flags.go:64] FLAG: --register-node="true" Sep 11 03:53:05.582064 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574610 2568 flags.go:64] FLAG: --register-with-taints="" Sep 11 03:53:05.582064 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574613 2568 flags.go:64] FLAG: --registry-burst="10" Sep 11 03:53:05.582064 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574616 2568 flags.go:64] FLAG: --registry-qps="5" Sep 11 03:53:05.582064 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574619 2568 flags.go:64] FLAG: --reserved-cpus="" Sep 11 03:53:05.582683 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574621 2568 flags.go:64] FLAG: --reserved-memory="" Sep 11 03:53:05.582683 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574625 2568 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 11 03:53:05.582683 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574628 2568 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 11 03:53:05.582683 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574631 2568 flags.go:64] FLAG: --rotate-certificates="false" Sep 11 03:53:05.582683 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574634 2568 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 11 03:53:05.582683 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574636 2568 flags.go:64] FLAG: --runonce="false" Sep 11 03:53:05.582683 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574639 2568 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 11 03:53:05.582683 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574643 2568 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 11 03:53:05.582683 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574645 2568 flags.go:64] FLAG: --seccomp-default="false" Sep 11 03:53:05.582683 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574648 2568 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 11 03:53:05.582683 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574651 2568 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 11 03:53:05.582683 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574654 2568 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 11 03:53:05.582683 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574657 2568 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 11 03:53:05.582683 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574660 2568 flags.go:64] FLAG: --storage-driver-password="root" Sep 11 03:53:05.582683 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574662 2568 flags.go:64] FLAG: --storage-driver-secure="false" Sep 11 03:53:05.582683 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574665 2568 flags.go:64] FLAG: --storage-driver-table="stats" Sep 11 03:53:05.582683 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574668 2568 flags.go:64] FLAG: --storage-driver-user="root" Sep 11 03:53:05.582683 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574671 2568 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 11 03:53:05.582683 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574676 2568 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 11 03:53:05.582683 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574684 2568 flags.go:64] FLAG: --system-cgroups="" Sep 11 03:53:05.582683 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574687 2568 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 11 03:53:05.582683 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574700 2568 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 11 03:53:05.582683 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574704 2568 flags.go:64] FLAG: --tls-cert-file="" Sep 11 03:53:05.582683 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574707 2568 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 11 03:53:05.582683 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574713 2568 flags.go:64] FLAG: --tls-min-version="" Sep 11 03:53:05.583322 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574716 2568 flags.go:64] FLAG: --tls-private-key-file="" Sep 11 03:53:05.583322 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574719 2568 flags.go:64] FLAG: --topology-manager-policy="none" Sep 11 03:53:05.583322 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574722 2568 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 11 03:53:05.583322 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574725 2568 flags.go:64] FLAG: --topology-manager-scope="container" Sep 11 03:53:05.583322 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574728 2568 flags.go:64] FLAG: --v="2" Sep 11 03:53:05.583322 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574732 2568 flags.go:64] FLAG: --version="false" Sep 11 03:53:05.583322 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574736 2568 flags.go:64] FLAG: --vmodule="" Sep 11 03:53:05.583322 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574740 2568 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 11 03:53:05.583322 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.574743 2568 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 11 03:53:05.583322 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574881 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 03:53:05.583322 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574886 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 03:53:05.583322 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574890 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 03:53:05.583322 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574893 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 03:53:05.583322 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574898 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 03:53:05.583322 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574901 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 03:53:05.583322 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574904 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 03:53:05.583322 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574907 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 03:53:05.583322 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574910 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 03:53:05.583322 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574912 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 03:53:05.583322 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574915 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 03:53:05.583322 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574918 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 03:53:05.583322 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574920 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 03:53:05.583857 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574922 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 03:53:05.583857 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574925 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 03:53:05.583857 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574928 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 03:53:05.583857 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574932 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 03:53:05.583857 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574934 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 03:53:05.583857 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574942 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 03:53:05.583857 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574946 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 03:53:05.583857 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574948 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 03:53:05.583857 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574951 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 03:53:05.583857 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574953 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 03:53:05.583857 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574956 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 03:53:05.583857 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574958 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 03:53:05.583857 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574961 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 03:53:05.583857 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574964 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 03:53:05.583857 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574966 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 03:53:05.583857 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574969 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 03:53:05.583857 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574971 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 03:53:05.583857 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574974 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 03:53:05.583857 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574976 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 03:53:05.583857 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574979 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 03:53:05.584390 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574981 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 03:53:05.584390 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574984 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 03:53:05.584390 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574986 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 03:53:05.584390 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574990 2568 feature_gate.go:328] unrecognized feature gate: Example Sep 11 03:53:05.584390 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574993 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 03:53:05.584390 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574995 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 03:53:05.584390 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.574998 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 03:53:05.584390 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575000 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 03:53:05.584390 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575003 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 03:53:05.584390 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575005 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 03:53:05.584390 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575008 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 03:53:05.584390 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575010 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 03:53:05.584390 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575013 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 03:53:05.584390 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575016 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 03:53:05.584390 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575018 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 03:53:05.584390 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575022 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 03:53:05.584390 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575025 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 03:53:05.584390 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575027 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 03:53:05.584390 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575036 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 03:53:05.584390 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575038 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 03:53:05.584867 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575041 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 03:53:05.584867 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575058 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 03:53:05.584867 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575062 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 03:53:05.584867 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575066 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 03:53:05.584867 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575078 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 03:53:05.584867 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575082 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 03:53:05.584867 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575084 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 03:53:05.584867 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575087 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 03:53:05.584867 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575089 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 03:53:05.584867 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575092 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 03:53:05.584867 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575094 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 03:53:05.584867 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575097 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 03:53:05.584867 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575099 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 03:53:05.584867 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575103 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 03:53:05.584867 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575107 2568 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 03:53:05.584867 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575112 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 03:53:05.584867 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575115 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 03:53:05.584867 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575118 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 03:53:05.584867 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575121 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 03:53:05.584867 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575124 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 03:53:05.585367 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575126 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 03:53:05.585367 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575129 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 03:53:05.585367 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575132 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 03:53:05.585367 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575134 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 03:53:05.585367 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575137 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 03:53:05.585367 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575140 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 03:53:05.585367 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575144 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 03:53:05.585367 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575148 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 03:53:05.585367 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575151 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 03:53:05.585367 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575153 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 03:53:05.585367 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575156 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 03:53:05.585367 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575164 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 03:53:05.585367 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575167 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 03:53:05.585367 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575170 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 03:53:05.585367 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575172 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 03:53:05.585367 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575175 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 03:53:05.585367 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575177 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 03:53:05.585367 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575180 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 03:53:05.585367 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575182 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 03:53:05.585835 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575185 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 03:53:05.585835 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575187 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 03:53:05.585835 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575190 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 03:53:05.585835 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575192 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 03:53:05.585835 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575195 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 03:53:05.585835 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575197 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 03:53:05.585835 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.575200 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 03:53:05.585835 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.575960 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 03:53:05.585835 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.584002 2568 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 11 03:53:05.585835 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.584017 2568 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 11 03:53:05.585835 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.584949 2568 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 11 03:53:05.585835 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.584956 2568 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 11 03:53:05.585835 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585023 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 03:53:05.585835 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585028 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 03:53:05.585835 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585032 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 03:53:05.586227 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585035 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 03:53:05.586227 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585038 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 03:53:05.586227 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585041 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 03:53:05.586227 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585055 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 03:53:05.586227 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585058 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 03:53:05.586227 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585061 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 03:53:05.586227 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585064 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 03:53:05.586227 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585067 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 03:53:05.586227 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585071 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 03:53:05.586227 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585076 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 03:53:05.586227 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585079 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 03:53:05.586227 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585082 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 03:53:05.586227 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585085 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 03:53:05.586227 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585088 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 03:53:05.586227 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585090 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 03:53:05.586227 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585093 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 03:53:05.586227 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585096 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 03:53:05.586227 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585098 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 03:53:05.586227 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585101 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 03:53:05.586709 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585104 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 03:53:05.586709 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585106 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 03:53:05.586709 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585109 2568 feature_gate.go:328] unrecognized feature gate: Example Sep 11 03:53:05.586709 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585111 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 03:53:05.586709 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585114 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 03:53:05.586709 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585117 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 03:53:05.586709 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585120 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 03:53:05.586709 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585124 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 03:53:05.586709 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585129 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 03:53:05.586709 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585132 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 03:53:05.586709 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585135 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 03:53:05.586709 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585138 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 03:53:05.586709 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585141 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 03:53:05.586709 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585144 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 03:53:05.586709 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585146 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 03:53:05.586709 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585149 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 03:53:05.586709 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585151 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 03:53:05.586709 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585154 2568 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 03:53:05.586709 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585157 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 03:53:05.586709 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585159 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 03:53:05.587217 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585162 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 03:53:05.587217 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585166 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 03:53:05.587217 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585170 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 03:53:05.587217 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585173 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 03:53:05.587217 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585176 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 03:53:05.587217 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585179 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 03:53:05.587217 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585182 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 03:53:05.587217 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585184 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 03:53:05.587217 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585187 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 03:53:05.587217 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585190 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 03:53:05.587217 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585192 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 03:53:05.587217 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585195 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 03:53:05.587217 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585197 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 03:53:05.587217 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585200 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 03:53:05.587217 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585202 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 03:53:05.587217 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585205 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 03:53:05.587217 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585208 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 03:53:05.587217 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585212 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 03:53:05.587217 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585215 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 03:53:05.587713 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585218 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 03:53:05.587713 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585221 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 03:53:05.587713 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585224 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 03:53:05.587713 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585227 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 03:53:05.587713 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585230 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 03:53:05.587713 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585233 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 03:53:05.587713 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585235 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 03:53:05.587713 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585238 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 03:53:05.587713 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585241 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 03:53:05.587713 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585243 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 03:53:05.587713 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585246 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 03:53:05.587713 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585249 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 03:53:05.587713 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585251 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 03:53:05.587713 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585254 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 03:53:05.587713 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585256 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 03:53:05.587713 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585259 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 03:53:05.587713 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585261 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 03:53:05.587713 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585264 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 03:53:05.587713 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585267 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 03:53:05.587713 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585269 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 03:53:05.588201 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585272 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 03:53:05.588201 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585274 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 03:53:05.588201 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585277 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 03:53:05.588201 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585279 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 03:53:05.588201 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585282 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 03:53:05.588201 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585285 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 03:53:05.588201 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585287 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 03:53:05.588201 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585290 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 03:53:05.588201 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585292 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 03:53:05.588201 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585295 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 03:53:05.588201 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585297 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 03:53:05.588201 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585300 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 03:53:05.588201 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585303 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 03:53:05.588201 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585306 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 03:53:05.588201 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585309 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 03:53:05.588201 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585311 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 03:53:05.588201 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585314 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 03:53:05.588201 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585316 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 03:53:05.588645 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.585322 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 03:53:05.588645 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585443 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 03:53:05.588645 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585447 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 03:53:05.588645 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585450 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 03:53:05.588645 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585454 2568 feature_gate.go:328] unrecognized feature gate: Example Sep 11 03:53:05.588645 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585457 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 03:53:05.588645 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585460 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 03:53:05.588645 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585463 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 03:53:05.588645 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585465 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 03:53:05.588645 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585468 2568 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 03:53:05.588645 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585470 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 03:53:05.588645 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585473 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 03:53:05.588645 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585476 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 03:53:05.588645 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585478 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 03:53:05.588645 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585482 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 03:53:05.589013 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585486 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 03:53:05.589013 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585489 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 03:53:05.589013 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585492 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 03:53:05.589013 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585495 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 03:53:05.589013 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585498 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 03:53:05.589013 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585501 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 03:53:05.589013 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585503 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 03:53:05.589013 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585506 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 03:53:05.589013 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585509 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 03:53:05.589013 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585511 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 03:53:05.589013 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585514 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 03:53:05.589013 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585517 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 03:53:05.589013 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585519 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 03:53:05.589013 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585522 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 03:53:05.589013 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585524 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 03:53:05.589013 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585527 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 03:53:05.589013 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585530 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 03:53:05.589013 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585532 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 03:53:05.589013 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585535 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 03:53:05.589013 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585537 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 03:53:05.589568 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585540 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 03:53:05.589568 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585542 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 03:53:05.589568 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585545 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 03:53:05.589568 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585547 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 03:53:05.589568 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585550 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 03:53:05.589568 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585553 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 03:53:05.589568 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585555 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 03:53:05.589568 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585558 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 03:53:05.589568 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585560 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 03:53:05.589568 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585563 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 03:53:05.589568 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585566 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 03:53:05.589568 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585568 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 03:53:05.589568 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585571 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 03:53:05.589568 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585573 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 03:53:05.589568 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585576 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 03:53:05.589568 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585578 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 03:53:05.589568 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585581 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 03:53:05.589568 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585585 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 03:53:05.589568 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585587 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 03:53:05.590023 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585590 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 03:53:05.590023 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585593 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 03:53:05.590023 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585595 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 03:53:05.590023 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585598 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 03:53:05.590023 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585601 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 03:53:05.590023 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585604 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 03:53:05.590023 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585606 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 03:53:05.590023 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585609 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 03:53:05.590023 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585611 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 03:53:05.590023 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585614 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 03:53:05.590023 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585616 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 03:53:05.590023 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585619 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 03:53:05.590023 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585621 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 03:53:05.590023 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585624 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 03:53:05.590023 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585626 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 03:53:05.590023 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585629 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 03:53:05.590023 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585631 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 03:53:05.590023 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585634 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 03:53:05.590023 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585636 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 03:53:05.590023 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585639 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 03:53:05.590555 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585641 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 03:53:05.590555 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585644 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 03:53:05.590555 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585646 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 03:53:05.590555 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585649 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 03:53:05.590555 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585651 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 03:53:05.590555 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585654 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 03:53:05.590555 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585656 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 03:53:05.590555 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585660 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 03:53:05.590555 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585664 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 03:53:05.590555 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585667 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 03:53:05.590555 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585670 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 03:53:05.590555 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585673 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 03:53:05.590555 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585676 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 03:53:05.590555 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585678 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 03:53:05.590555 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585681 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 03:53:05.590555 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585683 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 03:53:05.590555 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585686 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 03:53:05.590555 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585689 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 03:53:05.590555 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585692 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 03:53:05.591019 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585695 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 03:53:05.591019 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585698 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 03:53:05.591019 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585700 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 03:53:05.591019 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585703 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 03:53:05.591019 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585705 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 03:53:05.591019 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585708 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 03:53:05.591019 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:05.585710 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 03:53:05.591019 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.585715 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 03:53:05.591019 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.585819 2568 server.go:962] "Client rotation is on, will bootstrap in background" Sep 11 03:53:05.591019 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.590123 2568 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 11 03:53:05.591279 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.591267 2568 server.go:1019] "Starting client certificate rotation" Sep 11 03:53:05.591392 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.591373 2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 03:53:05.591431 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.591413 2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 03:53:05.594098 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.594084 2568 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 03:53:05.597939 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.597909 2568 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 03:53:05.609329 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.609306 2568 log.go:25] "Validated CRI v1 runtime API" Sep 11 03:53:05.612903 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.612887 2568 log.go:25] "Validated CRI v1 image API" Sep 11 03:53:05.615492 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.615472 2568 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 11 03:53:05.619314 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.619295 2568 fs.go:135] Filesystem UUIDs: map[18dc3283-a90d-4b8c-89f5-13f09963fc3d:/dev/nvme0n1p4 31bc6a4e-3178-4d28-9991-d103728d4b87:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Sep 11 03:53:05.619416 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.619312 2568 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 11 03:53:05.620994 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.620978 2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 03:53:05.624861 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.624751 2568 manager.go:216] Machine: {Timestamp:2026-09-11 03:53:05.623314637 +0000 UTC m=+0.376946030 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100032 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec255a89fda80fd178e056581d14f587 SystemUUID:ec255a89-fda8-0fd1-78e0-56581d14f587 BootID:b2e5c9e9-3db6-46d3-9c71-bc045cdf7ecc Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:3a:66:ad:47:91 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:3a:66:ad:47:91 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b6:3a:19:91:00:39 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 11 03:53:05.624861 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.624854 2568 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 11 03:53:05.624990 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.624937 2568 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 11 03:53:05.628488 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.628458 2568 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 11 03:53:05.628651 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.628491 2568 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-30.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 11 03:53:05.630108 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.630094 2568 topology_manager.go:138] "Creating topology manager with none policy" Sep 11 03:53:05.630165 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.630113 2568 container_manager_linux.go:309] "Creating device plugin manager" Sep 11 03:53:05.630165 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.630124 2568 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 03:53:05.631021 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.631008 2568 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 03:53:05.632407 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.632395 2568 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 11 03:53:05.636996 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.636982 2568 state_mem.go:36] "Initialized new in-memory state store" Sep 11 03:53:05.637143 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.637131 2568 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 11 03:53:05.637209 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.637198 2568 kubelet.go:486] "Attempting to sync node with API server" Sep 11 03:53:05.637261 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.637214 2568 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 11 03:53:05.637261 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.637230 2568 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 11 03:53:05.637261 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.637247 2568 kubelet.go:398] "Adding apiserver pod source" Sep 11 03:53:05.637261 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.637260 2568 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 11 03:53:05.637670 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.637655 2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 03:53:05.637737 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.637680 2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 03:53:05.640150 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.640129 2568 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 11 03:53:05.643035 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.643018 2568 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 11 03:53:05.643112 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.643063 2568 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 11 03:53:05.645221 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.645207 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 11 03:53:05.645288 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.645224 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 11 03:53:05.645288 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.645231 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 11 03:53:05.645288 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.645239 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 11 03:53:05.645288 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.645250 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 11 03:53:05.645288 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.645256 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 11 03:53:05.645288 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.645269 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 11 03:53:05.645288 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.645274 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 11 03:53:05.645288 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.645280 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 11 03:53:05.645288 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.645285 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 11 03:53:05.645519 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.645294 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 11 03:53:05.645519 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.645299 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 11 03:53:05.646233 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.646218 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 11 03:53:05.646233 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.646230 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 11 03:53:05.648744 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.648725 2568 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-30.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 11 03:53:05.649487 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:05.649464 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 11 03:53:05.649538 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:05.649464 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-30.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 11 03:53:05.650176 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.650164 2568 server.go:1268] "Started kubelet" Sep 11 03:53:05.650259 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.650226 2568 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 11 03:53:05.650296 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.650246 2568 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 11 03:53:05.650343 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.650313 2568 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 11 03:53:05.655218 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.655196 2568 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 11 03:53:05.655548 ip-10-0-140-30 systemd[1]: Started Kubernetes Kubelet. Sep 11 03:53:05.656847 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:05.656785 2568 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 11 03:53:05.657400 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.657374 2568 server.go:310] "Adding debug handlers to kubelet server" Sep 11 03:53:05.662812 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:05.661704 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-30.ec2.internal.18d427ddd2c258a6 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-30.ec2.internal,UID:ip-10-0-140-30.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-30.ec2.internal,},FirstTimestamp:2026-09-11 03:53:05.650137254 +0000 UTC m=+0.403768647,LastTimestamp:2026-09-11 03:53:05.650137254 +0000 UTC m=+0.403768647,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-30.ec2.internal,}" Sep 11 03:53:05.665472 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.665457 2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 11 03:53:05.666151 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.666135 2568 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 11 03:53:05.666359 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:05.666332 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-30.ec2.internal\" not found" Sep 11 03:53:05.666866 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.666845 2568 factory.go:55] Registering systemd factory Sep 11 03:53:05.666943 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.666869 2568 factory.go:223] Registration of the systemd container factory successfully Sep 11 03:53:05.667042 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.666971 2568 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 11 03:53:05.667101 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.666970 2568 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 11 03:53:05.667144 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.667112 2568 factory.go:153] Registering CRI-O factory Sep 11 03:53:05.667144 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.667123 2568 factory.go:223] Registration of the crio container factory successfully Sep 11 03:53:05.667196 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.667163 2568 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 11 03:53:05.667196 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.667180 2568 factory.go:103] Registering Raw factory Sep 11 03:53:05.667196 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.667191 2568 manager.go:1196] Started watching for new ooms in manager Sep 11 03:53:05.667528 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.667518 2568 manager.go:349] Starting recovery of all containers Sep 11 03:53:05.667618 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.667600 2568 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 11 03:53:05.667740 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.667712 2568 reconstruct.go:97] "Volume reconstruction finished" Sep 11 03:53:05.667740 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.667720 2568 reconciler.go:29] "Reconciler: start to sync state" Sep 11 03:53:05.669624 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:05.669598 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 11 03:53:05.669725 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:05.669704 2568 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-30.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 11 03:53:05.676457 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.676433 2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-r7w2z" Sep 11 03:53:05.677287 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.677267 2568 manager.go:354] Recovery completed Sep 11 03:53:05.683201 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.683186 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 03:53:05.683699 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.683686 2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-r7w2z" Sep 11 03:53:05.685102 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.685084 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-30.ec2.internal" event="NodeHasSufficientMemory" Sep 11 03:53:05.685179 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.685111 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-30.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 03:53:05.685179 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.685121 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-30.ec2.internal" event="NodeHasSufficientPID" Sep 11 03:53:05.685665 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.685647 2568 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 11 03:53:05.685665 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.685663 2568 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 11 03:53:05.685780 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.685680 2568 state_mem.go:36] "Initialized new in-memory state store" Sep 11 03:53:05.685780 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.685680 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 03:53:05.686449 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.686430 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-30.ec2.internal" event="NodeHasSufficientMemory" Sep 11 03:53:05.686527 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.686461 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-30.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 03:53:05.686527 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.686494 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-30.ec2.internal" event="NodeHasSufficientPID" Sep 11 03:53:05.687238 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:05.687152 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-30.ec2.internal.18d427ddd4d7d03a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-30.ec2.internal,UID:ip-10-0-140-30.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-30.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-30.ec2.internal,},FirstTimestamp:2026-09-11 03:53:05.685098554 +0000 UTC m=+0.438729946,LastTimestamp:2026-09-11 03:53:05.685098554 +0000 UTC m=+0.438729946,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-30.ec2.internal,}" Sep 11 03:53:05.687920 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.687902 2568 policy_none.go:49] "None policy: Start" Sep 11 03:53:05.687992 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.687924 2568 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 11 03:53:05.687992 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.687937 2568 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 11 03:53:05.690027 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.690010 2568 policy_none.go:47] "Start" Sep 11 03:53:05.722784 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.722764 2568 manager.go:337] "Starting Device Plugin manager" Sep 11 03:53:05.741117 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:05.722811 2568 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 11 03:53:05.741117 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.722825 2568 server.go:87] "Starting device plugin registration server" Sep 11 03:53:05.741117 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.723237 2568 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 11 03:53:05.741117 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.723250 2568 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 11 03:53:05.741117 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.723353 2568 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 11 03:53:05.741117 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.723433 2568 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 11 03:53:05.741117 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.723443 2568 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 11 03:53:05.741117 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:05.723966 2568 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 11 03:53:05.741117 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:05.724013 2568 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-30.ec2.internal\" not found" Sep 11 03:53:05.760367 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.760340 2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 11 03:53:05.761590 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.761564 2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 11 03:53:05.761689 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.761627 2568 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 11 03:53:05.761689 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.761656 2568 kubelet.go:2448] "Starting kubelet main sync loop" Sep 11 03:53:05.761786 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:05.761701 2568 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 11 03:53:05.763846 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.763820 2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 03:53:05.823739 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.823691 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 03:53:05.824951 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.824936 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-30.ec2.internal" event="NodeHasSufficientMemory" Sep 11 03:53:05.825039 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.824968 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-30.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 03:53:05.825039 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.824984 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-30.ec2.internal" event="NodeHasSufficientPID" Sep 11 03:53:05.825039 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.825014 2568 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-30.ec2.internal" Sep 11 03:53:05.833386 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.833368 2568 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-30.ec2.internal" Sep 11 03:53:05.833386 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:05.833387 2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-30.ec2.internal\": node \"ip-10-0-140-30.ec2.internal\" not found" Sep 11 03:53:05.861772 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.861750 2568 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-30.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-30.ec2.internal"] Sep 11 03:53:05.861861 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:05.861783 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-30.ec2.internal\" not found" Sep 11 03:53:05.861861 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.861816 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 03:53:05.863089 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.863075 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-30.ec2.internal" event="NodeHasSufficientMemory" Sep 11 03:53:05.863165 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.863100 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-30.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 03:53:05.863165 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.863111 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-30.ec2.internal" event="NodeHasSufficientPID" Sep 11 03:53:05.865404 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.865392 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 03:53:05.865568 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.865555 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-30.ec2.internal" Sep 11 03:53:05.865616 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.865606 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 03:53:05.866115 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.866102 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-30.ec2.internal" event="NodeHasSufficientMemory" Sep 11 03:53:05.866180 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.866123 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-30.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 03:53:05.866180 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.866129 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-30.ec2.internal" event="NodeHasSufficientMemory" Sep 11 03:53:05.866180 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.866154 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-30.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 03:53:05.866180 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.866163 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-30.ec2.internal" event="NodeHasSufficientPID" Sep 11 03:53:05.866302 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.866133 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-30.ec2.internal" event="NodeHasSufficientPID" Sep 11 03:53:05.868359 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.868346 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-30.ec2.internal" Sep 11 03:53:05.868415 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.868369 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 03:53:05.868662 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.868646 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/72105b6990dc29a2d8255d18f1b5a345-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-30.ec2.internal\" (UID: \"72105b6990dc29a2d8255d18f1b5a345\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-30.ec2.internal" Sep 11 03:53:05.868702 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.868669 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72105b6990dc29a2d8255d18f1b5a345-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-30.ec2.internal\" (UID: \"72105b6990dc29a2d8255d18f1b5a345\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-30.ec2.internal" Sep 11 03:53:05.868702 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.868685 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f90f3e1f70d21477b755e9266cecc76b-config\") pod \"kube-apiserver-proxy-ip-10-0-140-30.ec2.internal\" (UID: \"f90f3e1f70d21477b755e9266cecc76b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-30.ec2.internal" Sep 11 03:53:05.869031 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.869016 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-30.ec2.internal" event="NodeHasSufficientMemory" Sep 11 03:53:05.869139 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.869039 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-30.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 03:53:05.869139 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.869070 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-30.ec2.internal" event="NodeHasSufficientPID" Sep 11 03:53:05.889546 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:05.889521 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-30.ec2.internal\" not found" node="ip-10-0-140-30.ec2.internal" Sep 11 03:53:05.893694 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:05.893678 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-30.ec2.internal\" not found" node="ip-10-0-140-30.ec2.internal" Sep 11 03:53:05.962609 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:05.962582 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-30.ec2.internal\" not found" Sep 11 03:53:05.968912 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.968875 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/72105b6990dc29a2d8255d18f1b5a345-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-30.ec2.internal\" (UID: \"72105b6990dc29a2d8255d18f1b5a345\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-30.ec2.internal" Sep 11 03:53:05.968968 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.968916 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72105b6990dc29a2d8255d18f1b5a345-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-30.ec2.internal\" (UID: \"72105b6990dc29a2d8255d18f1b5a345\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-30.ec2.internal" Sep 11 03:53:05.968968 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.968931 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f90f3e1f70d21477b755e9266cecc76b-config\") pod \"kube-apiserver-proxy-ip-10-0-140-30.ec2.internal\" (UID: \"f90f3e1f70d21477b755e9266cecc76b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-30.ec2.internal" Sep 11 03:53:05.969062 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.968980 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/72105b6990dc29a2d8255d18f1b5a345-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-30.ec2.internal\" (UID: \"72105b6990dc29a2d8255d18f1b5a345\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-30.ec2.internal" Sep 11 03:53:05.969062 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.969003 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f90f3e1f70d21477b755e9266cecc76b-config\") pod \"kube-apiserver-proxy-ip-10-0-140-30.ec2.internal\" (UID: \"f90f3e1f70d21477b755e9266cecc76b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-30.ec2.internal" Sep 11 03:53:05.969062 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:05.968983 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/72105b6990dc29a2d8255d18f1b5a345-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-30.ec2.internal\" (UID: \"72105b6990dc29a2d8255d18f1b5a345\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-30.ec2.internal" Sep 11 03:53:06.063126 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:06.063095 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-30.ec2.internal\" not found" Sep 11 03:53:06.163882 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:06.163831 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-30.ec2.internal\" not found" Sep 11 03:53:06.192377 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:06.192348 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-30.ec2.internal" Sep 11 03:53:06.195994 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:06.195973 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-30.ec2.internal" Sep 11 03:53:06.264711 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:06.264671 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-30.ec2.internal\" not found" Sep 11 03:53:06.365154 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:06.365120 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-30.ec2.internal\" not found" Sep 11 03:53:06.465529 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:06.465475 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-30.ec2.internal\" not found" Sep 11 03:53:06.545434 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:06.545411 2568 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 03:53:06.565957 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:06.565935 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-30.ec2.internal\" not found" Sep 11 03:53:06.591336 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:06.591312 2568 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 11 03:53:06.591469 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:06.591422 2568 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 11 03:53:06.591513 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:06.591460 2568 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 11 03:53:06.666639 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:06.666611 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-30.ec2.internal\" not found" Sep 11 03:53:06.689797 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:06.689765 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 03:48:05 +0000 UTC" deadline="2028-03-12 03:37:58.264519031 +0000 UTC" Sep 11 03:53:06.689797 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:06.689792 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13151h44m51.574729346s" Sep 11 03:53:06.748938 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:06.748911 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72105b6990dc29a2d8255d18f1b5a345.slice/crio-55fc9cfd90d5e6d87e21a9861775254230c9b9c1e9fba9cb57f355432b5b4dc8 WatchSource:0}: Error finding container 55fc9cfd90d5e6d87e21a9861775254230c9b9c1e9fba9cb57f355432b5b4dc8: Status 404 returned error can't find the container with id 55fc9cfd90d5e6d87e21a9861775254230c9b9c1e9fba9cb57f355432b5b4dc8 Sep 11 03:53:06.749179 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:06.749149 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf90f3e1f70d21477b755e9266cecc76b.slice/crio-9501bb1b7f284037396ab9f804f9b3e4532503c59bf7d45d82d941cbf72cc750 WatchSource:0}: Error finding container 9501bb1b7f284037396ab9f804f9b3e4532503c59bf7d45d82d941cbf72cc750: Status 404 returned error can't find the container with id 9501bb1b7f284037396ab9f804f9b3e4532503c59bf7d45d82d941cbf72cc750 Sep 11 03:53:06.753663 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:06.753650 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 03:53:06.764727 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:06.764688 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-30.ec2.internal" event={"ID":"f90f3e1f70d21477b755e9266cecc76b","Type":"ContainerStarted","Data":"9501bb1b7f284037396ab9f804f9b3e4532503c59bf7d45d82d941cbf72cc750"} Sep 11 03:53:06.765705 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:06.765686 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-30.ec2.internal" event={"ID":"72105b6990dc29a2d8255d18f1b5a345","Type":"ContainerStarted","Data":"55fc9cfd90d5e6d87e21a9861775254230c9b9c1e9fba9cb57f355432b5b4dc8"} Sep 11 03:53:06.766742 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:06.766727 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-30.ec2.internal\" not found" Sep 11 03:53:06.866986 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:06.866955 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-30.ec2.internal\" not found" Sep 11 03:53:06.967310 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:06.967286 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-30.ec2.internal\" not found" Sep 11 03:53:07.067750 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:07.067674 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-30.ec2.internal\" not found" Sep 11 03:53:07.091712 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.091684 2568 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 03:53:07.166582 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.166553 2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-30.ec2.internal" Sep 11 03:53:07.175170 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.175152 2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 03:53:07.175932 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.175920 2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-30.ec2.internal" Sep 11 03:53:07.186313 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.186296 2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 03:53:07.215379 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.215363 2568 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 03:53:07.638235 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.638186 2568 apiserver.go:52] "Watching apiserver" Sep 11 03:53:07.645841 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.645814 2568 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 11 03:53:07.647769 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.647740 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqk2s","openshift-image-registry/node-ca-gmdt4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-30.ec2.internal","openshift-multus/network-metrics-daemon-tk8q7","openshift-network-diagnostics/network-check-target-vssnm","kube-system/konnectivity-agent-c5drs","kube-system/kube-apiserver-proxy-ip-10-0-140-30.ec2.internal","openshift-cluster-node-tuning-operator/tuned-wvrb9","openshift-dns/node-resolver-7f284","openshift-multus/multus-additional-cni-plugins-9r65j","openshift-multus/multus-r8rsb","openshift-network-operator/iptables-alerter-cvjtt","openshift-ovn-kubernetes/ovnkube-node-htjph","kube-system/global-pull-secret-syncer-t5fb2"] Sep 11 03:53:07.650222 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.650202 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9r65j" Sep 11 03:53:07.652566 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.652545 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 11 03:53:07.652695 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.652549 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 11 03:53:07.652695 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.652551 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 11 03:53:07.652885 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.652810 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 11 03:53:07.652885 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.652831 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kmsfr\"" Sep 11 03:53:07.653001 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.652920 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 11 03:53:07.653208 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.653191 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.655130 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.655111 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-hvmx4\"" Sep 11 03:53:07.655130 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.655111 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 11 03:53:07.655484 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.655465 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wvrb9" Sep 11 03:53:07.657545 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.657520 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qcwkd\"" Sep 11 03:53:07.657648 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.657521 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 11 03:53:07.658304 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.657797 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk8q7" Sep 11 03:53:07.658304 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.657826 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 11 03:53:07.658304 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:07.657866 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk8q7" podUID="fe1486cf-6276-41a8-a685-025064c4abca" Sep 11 03:53:07.659903 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.659882 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gmdt4" Sep 11 03:53:07.661808 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.661792 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 11 03:53:07.662016 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.662001 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 11 03:53:07.662120 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.662017 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dkmqp\"" Sep 11 03:53:07.662286 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.662272 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 11 03:53:07.662339 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.662326 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-c5drs" Sep 11 03:53:07.664405 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.664274 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 11 03:53:07.664405 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.664368 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-c74w6\"" Sep 11 03:53:07.664549 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.664433 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.664549 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.664368 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 11 03:53:07.665795 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.665704 2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 11 03:53:07.666819 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.666725 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 11 03:53:07.666819 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.666740 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 11 03:53:07.666819 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.666754 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 11 03:53:07.666819 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.666767 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xt2qp\"" Sep 11 03:53:07.667478 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.667450 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 11 03:53:07.667824 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.667808 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 11 03:53:07.668233 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.668215 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 11 03:53:07.668592 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.668576 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7f284" Sep 11 03:53:07.671587 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.670662 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 11 03:53:07.671587 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.671173 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 11 03:53:07.671587 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.671386 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vxlrv\"" Sep 11 03:53:07.672748 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.672732 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vssnm" Sep 11 03:53:07.672839 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:07.672803 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vssnm" podUID="fb9f553c-03c7-4489-b43f-b62d630e7f50" Sep 11 03:53:07.675112 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.675092 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqk2s" Sep 11 03:53:07.676481 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.676461 2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 03:53:07.677228 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.677122 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/26ea58d8-26d9-4191-87b4-03d1efda477a-run\") pod \"tuned-wvrb9\" (UID: \"26ea58d8-26d9-4191-87b4-03d1efda477a\") " pod="openshift-cluster-node-tuning-operator/tuned-wvrb9" Sep 11 03:53:07.677228 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.677147 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aa66b45c-400f-46fa-b216-0e063413439b-agent-certs\") pod \"konnectivity-agent-c5drs\" (UID: \"aa66b45c-400f-46fa-b216-0e063413439b\") " pod="kube-system/konnectivity-agent-c5drs" Sep 11 03:53:07.677228 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.677194 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 11 03:53:07.677228 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.677194 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/44fdee40-4032-43ac-aabc-2393a05a4ef2-host-run-netns\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.677517 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.677236 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e35bc33f-65c8-4fc4-abef-6e652686ec22-host-run-k8s-cni-cncf-io\") pod \"multus-r8rsb\" (UID: \"e35bc33f-65c8-4fc4-abef-6e652686ec22\") " pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.677517 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.677260 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62v5d\" (UniqueName: \"kubernetes.io/projected/e35bc33f-65c8-4fc4-abef-6e652686ec22-kube-api-access-62v5d\") pod \"multus-r8rsb\" (UID: \"e35bc33f-65c8-4fc4-abef-6e652686ec22\") " pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.677517 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.677265 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-s8qr2\"" Sep 11 03:53:07.677517 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.677283 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6906bf8-7095-4f6a-8969-d6f28167548b-system-cni-dir\") pod \"multus-additional-cni-plugins-9r65j\" (UID: \"d6906bf8-7095-4f6a-8969-d6f28167548b\") " pod="openshift-multus/multus-additional-cni-plugins-9r65j" Sep 11 03:53:07.677517 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.677307 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6906bf8-7095-4f6a-8969-d6f28167548b-cnibin\") pod \"multus-additional-cni-plugins-9r65j\" (UID: \"d6906bf8-7095-4f6a-8969-d6f28167548b\") " pod="openshift-multus/multus-additional-cni-plugins-9r65j" Sep 11 03:53:07.677517 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.677325 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 11 03:53:07.677517 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.677329 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/44fdee40-4032-43ac-aabc-2393a05a4ef2-etc-openvswitch\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.677517 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.677351 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/44fdee40-4032-43ac-aabc-2393a05a4ef2-host-cni-netd\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.677517 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.677285 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 11 03:53:07.677517 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.677400 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a48e2404-e37c-43d5-8902-b139cb2e07d9-tmp-dir\") pod \"node-resolver-7f284\" (UID: \"a48e2404-e37c-43d5-8902-b139cb2e07d9\") " pod="openshift-dns/node-resolver-7f284" Sep 11 03:53:07.677517 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.677429 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e35bc33f-65c8-4fc4-abef-6e652686ec22-cni-binary-copy\") pod \"multus-r8rsb\" (UID: \"e35bc33f-65c8-4fc4-abef-6e652686ec22\") " pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.677517 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.677458 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aa66b45c-400f-46fa-b216-0e063413439b-konnectivity-ca\") pod \"konnectivity-agent-c5drs\" (UID: \"aa66b45c-400f-46fa-b216-0e063413439b\") " pod="kube-system/konnectivity-agent-c5drs" Sep 11 03:53:07.677517 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.677481 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44fdee40-4032-43ac-aabc-2393a05a4ef2-host-run-ovn-kubernetes\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.677517 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.677506 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e35bc33f-65c8-4fc4-abef-6e652686ec22-host-run-multus-certs\") pod \"multus-r8rsb\" (UID: \"e35bc33f-65c8-4fc4-abef-6e652686ec22\") " pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.678173 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.677530 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44fdee40-4032-43ac-aabc-2393a05a4ef2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.678173 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.677563 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e35bc33f-65c8-4fc4-abef-6e652686ec22-multus-socket-dir-parent\") pod \"multus-r8rsb\" (UID: \"e35bc33f-65c8-4fc4-abef-6e652686ec22\") " pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.678173 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.677597 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/44fdee40-4032-43ac-aabc-2393a05a4ef2-ovnkube-script-lib\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.678173 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.677630 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/26ea58d8-26d9-4191-87b4-03d1efda477a-lib-modules\") pod \"tuned-wvrb9\" (UID: \"26ea58d8-26d9-4191-87b4-03d1efda477a\") " pod="openshift-cluster-node-tuning-operator/tuned-wvrb9" Sep 11 03:53:07.678173 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.677674 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe1486cf-6276-41a8-a685-025064c4abca-metrics-certs\") pod \"network-metrics-daemon-tk8q7\" (UID: \"fe1486cf-6276-41a8-a685-025064c4abca\") " pod="openshift-multus/network-metrics-daemon-tk8q7" Sep 11 03:53:07.678173 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.677711 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/44fdee40-4032-43ac-aabc-2393a05a4ef2-log-socket\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.678173 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.677734 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/44fdee40-4032-43ac-aabc-2393a05a4ef2-ovnkube-config\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.678173 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.677745 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cvjtt" Sep 11 03:53:07.678173 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.677761 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb849\" (UniqueName: \"kubernetes.io/projected/44fdee40-4032-43ac-aabc-2393a05a4ef2-kube-api-access-sb849\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.678173 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.677790 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjh77\" (UniqueName: \"kubernetes.io/projected/a48e2404-e37c-43d5-8902-b139cb2e07d9-kube-api-access-mjh77\") pod \"node-resolver-7f284\" (UID: \"a48e2404-e37c-43d5-8902-b139cb2e07d9\") " pod="openshift-dns/node-resolver-7f284" Sep 11 03:53:07.678173 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.677819 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/26ea58d8-26d9-4191-87b4-03d1efda477a-etc-sysconfig\") pod \"tuned-wvrb9\" (UID: \"26ea58d8-26d9-4191-87b4-03d1efda477a\") " pod="openshift-cluster-node-tuning-operator/tuned-wvrb9" Sep 11 03:53:07.678173 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.677839 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/26ea58d8-26d9-4191-87b4-03d1efda477a-tmp\") pod \"tuned-wvrb9\" (UID: \"26ea58d8-26d9-4191-87b4-03d1efda477a\") " pod="openshift-cluster-node-tuning-operator/tuned-wvrb9" Sep 11 03:53:07.678173 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.677860 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6clwl\" (UniqueName: \"kubernetes.io/projected/fe1486cf-6276-41a8-a685-025064c4abca-kube-api-access-6clwl\") pod \"network-metrics-daemon-tk8q7\" (UID: \"fe1486cf-6276-41a8-a685-025064c4abca\") " pod="openshift-multus/network-metrics-daemon-tk8q7" Sep 11 03:53:07.678173 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.677885 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/44fdee40-4032-43ac-aabc-2393a05a4ef2-host-kubelet\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.678173 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.677905 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/44fdee40-4032-43ac-aabc-2393a05a4ef2-host-cni-bin\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.678173 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.677927 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnmts\" (UniqueName: \"kubernetes.io/projected/fb9f553c-03c7-4489-b43f-b62d630e7f50-kube-api-access-lnmts\") pod \"network-check-target-vssnm\" (UID: \"fb9f553c-03c7-4489-b43f-b62d630e7f50\") " pod="openshift-network-diagnostics/network-check-target-vssnm" Sep 11 03:53:07.678173 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.677949 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e35bc33f-65c8-4fc4-abef-6e652686ec22-host-var-lib-cni-bin\") pod \"multus-r8rsb\" (UID: \"e35bc33f-65c8-4fc4-abef-6e652686ec22\") " pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.678937 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.677997 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/db235f27-7ffe-4c12-9d60-53fb8b810d97-serviceca\") pod \"node-ca-gmdt4\" (UID: \"db235f27-7ffe-4c12-9d60-53fb8b810d97\") " pod="openshift-image-registry/node-ca-gmdt4" Sep 11 03:53:07.678937 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.678019 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/44fdee40-4032-43ac-aabc-2393a05a4ef2-systemd-units\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.678937 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.678066 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/44fdee40-4032-43ac-aabc-2393a05a4ef2-var-lib-openvswitch\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.678937 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.678113 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/44fdee40-4032-43ac-aabc-2393a05a4ef2-env-overrides\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.678937 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.678143 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/44fdee40-4032-43ac-aabc-2393a05a4ef2-ovn-node-metrics-cert\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.678937 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.678169 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d6906bf8-7095-4f6a-8969-d6f28167548b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9r65j\" (UID: \"d6906bf8-7095-4f6a-8969-d6f28167548b\") " pod="openshift-multus/multus-additional-cni-plugins-9r65j" Sep 11 03:53:07.678937 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.678192 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7xdg\" (UniqueName: \"kubernetes.io/projected/d6906bf8-7095-4f6a-8969-d6f28167548b-kube-api-access-v7xdg\") pod \"multus-additional-cni-plugins-9r65j\" (UID: \"d6906bf8-7095-4f6a-8969-d6f28167548b\") " pod="openshift-multus/multus-additional-cni-plugins-9r65j" Sep 11 03:53:07.678937 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.678214 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e35bc33f-65c8-4fc4-abef-6e652686ec22-multus-cni-dir\") pod \"multus-r8rsb\" (UID: \"e35bc33f-65c8-4fc4-abef-6e652686ec22\") " pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.678937 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.678234 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e35bc33f-65c8-4fc4-abef-6e652686ec22-cnibin\") pod \"multus-r8rsb\" (UID: \"e35bc33f-65c8-4fc4-abef-6e652686ec22\") " pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.678937 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.678252 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e35bc33f-65c8-4fc4-abef-6e652686ec22-multus-daemon-config\") pod \"multus-r8rsb\" (UID: \"e35bc33f-65c8-4fc4-abef-6e652686ec22\") " pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.678937 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.678272 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e35bc33f-65c8-4fc4-abef-6e652686ec22-etc-kubernetes\") pod \"multus-r8rsb\" (UID: \"e35bc33f-65c8-4fc4-abef-6e652686ec22\") " pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.678937 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.678292 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sfdp\" (UniqueName: \"kubernetes.io/projected/26ea58d8-26d9-4191-87b4-03d1efda477a-kube-api-access-9sfdp\") pod \"tuned-wvrb9\" (UID: \"26ea58d8-26d9-4191-87b4-03d1efda477a\") " pod="openshift-cluster-node-tuning-operator/tuned-wvrb9" Sep 11 03:53:07.678937 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.678315 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d6906bf8-7095-4f6a-8969-d6f28167548b-cni-binary-copy\") pod \"multus-additional-cni-plugins-9r65j\" (UID: \"d6906bf8-7095-4f6a-8969-d6f28167548b\") " pod="openshift-multus/multus-additional-cni-plugins-9r65j" Sep 11 03:53:07.678937 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.678338 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e35bc33f-65c8-4fc4-abef-6e652686ec22-host-var-lib-cni-multus\") pod \"multus-r8rsb\" (UID: \"e35bc33f-65c8-4fc4-abef-6e652686ec22\") " pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.678937 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.678352 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e35bc33f-65c8-4fc4-abef-6e652686ec22-multus-conf-dir\") pod \"multus-r8rsb\" (UID: \"e35bc33f-65c8-4fc4-abef-6e652686ec22\") " pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.678937 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.678371 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26ea58d8-26d9-4191-87b4-03d1efda477a-etc-kubernetes\") pod \"tuned-wvrb9\" (UID: \"26ea58d8-26d9-4191-87b4-03d1efda477a\") " pod="openshift-cluster-node-tuning-operator/tuned-wvrb9" Sep 11 03:53:07.679597 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.678405 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d6906bf8-7095-4f6a-8969-d6f28167548b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9r65j\" (UID: \"d6906bf8-7095-4f6a-8969-d6f28167548b\") " pod="openshift-multus/multus-additional-cni-plugins-9r65j" Sep 11 03:53:07.679597 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.678429 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/44fdee40-4032-43ac-aabc-2393a05a4ef2-run-openvswitch\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.679597 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.678448 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a48e2404-e37c-43d5-8902-b139cb2e07d9-hosts-file\") pod \"node-resolver-7f284\" (UID: \"a48e2404-e37c-43d5-8902-b139cb2e07d9\") " pod="openshift-dns/node-resolver-7f284" Sep 11 03:53:07.679597 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.678469 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e35bc33f-65c8-4fc4-abef-6e652686ec22-system-cni-dir\") pod \"multus-r8rsb\" (UID: \"e35bc33f-65c8-4fc4-abef-6e652686ec22\") " pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.679597 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.678484 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e35bc33f-65c8-4fc4-abef-6e652686ec22-host-var-lib-kubelet\") pod \"multus-r8rsb\" (UID: \"e35bc33f-65c8-4fc4-abef-6e652686ec22\") " pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.679597 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.678498 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/26ea58d8-26d9-4191-87b4-03d1efda477a-etc-sysctl-d\") pod \"tuned-wvrb9\" (UID: \"26ea58d8-26d9-4191-87b4-03d1efda477a\") " pod="openshift-cluster-node-tuning-operator/tuned-wvrb9" Sep 11 03:53:07.679597 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.678515 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26ea58d8-26d9-4191-87b4-03d1efda477a-var-lib-kubelet\") pod \"tuned-wvrb9\" (UID: \"26ea58d8-26d9-4191-87b4-03d1efda477a\") " pod="openshift-cluster-node-tuning-operator/tuned-wvrb9" Sep 11 03:53:07.679597 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.678558 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6906bf8-7095-4f6a-8969-d6f28167548b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9r65j\" (UID: \"d6906bf8-7095-4f6a-8969-d6f28167548b\") " pod="openshift-multus/multus-additional-cni-plugins-9r65j" Sep 11 03:53:07.679597 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.678581 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/db235f27-7ffe-4c12-9d60-53fb8b810d97-host\") pod \"node-ca-gmdt4\" (UID: \"db235f27-7ffe-4c12-9d60-53fb8b810d97\") " pod="openshift-image-registry/node-ca-gmdt4" Sep 11 03:53:07.679597 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.678626 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/26ea58d8-26d9-4191-87b4-03d1efda477a-etc-modprobe-d\") pod \"tuned-wvrb9\" (UID: \"26ea58d8-26d9-4191-87b4-03d1efda477a\") " pod="openshift-cluster-node-tuning-operator/tuned-wvrb9" Sep 11 03:53:07.679597 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.678660 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/26ea58d8-26d9-4191-87b4-03d1efda477a-etc-systemd\") pod \"tuned-wvrb9\" (UID: \"26ea58d8-26d9-4191-87b4-03d1efda477a\") " pod="openshift-cluster-node-tuning-operator/tuned-wvrb9" Sep 11 03:53:07.679597 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.678706 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/44fdee40-4032-43ac-aabc-2393a05a4ef2-node-log\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.679597 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.678728 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/44fdee40-4032-43ac-aabc-2393a05a4ef2-run-systemd\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.679597 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.678749 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e35bc33f-65c8-4fc4-abef-6e652686ec22-hostroot\") pod \"multus-r8rsb\" (UID: \"e35bc33f-65c8-4fc4-abef-6e652686ec22\") " pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.679597 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.678780 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/26ea58d8-26d9-4191-87b4-03d1efda477a-sys\") pod \"tuned-wvrb9\" (UID: \"26ea58d8-26d9-4191-87b4-03d1efda477a\") " pod="openshift-cluster-node-tuning-operator/tuned-wvrb9" Sep 11 03:53:07.679597 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.678830 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/26ea58d8-26d9-4191-87b4-03d1efda477a-etc-sysctl-conf\") pod \"tuned-wvrb9\" (UID: \"26ea58d8-26d9-4191-87b4-03d1efda477a\") " pod="openshift-cluster-node-tuning-operator/tuned-wvrb9" Sep 11 03:53:07.679597 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.678863 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/26ea58d8-26d9-4191-87b4-03d1efda477a-etc-tuned\") pod \"tuned-wvrb9\" (UID: \"26ea58d8-26d9-4191-87b4-03d1efda477a\") " pod="openshift-cluster-node-tuning-operator/tuned-wvrb9" Sep 11 03:53:07.680245 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.678885 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzj9q\" (UniqueName: \"kubernetes.io/projected/db235f27-7ffe-4c12-9d60-53fb8b810d97-kube-api-access-vzj9q\") pod \"node-ca-gmdt4\" (UID: \"db235f27-7ffe-4c12-9d60-53fb8b810d97\") " pod="openshift-image-registry/node-ca-gmdt4" Sep 11 03:53:07.680245 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.678914 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/44fdee40-4032-43ac-aabc-2393a05a4ef2-host-slash\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.680245 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.678934 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/44fdee40-4032-43ac-aabc-2393a05a4ef2-run-ovn\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.680245 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.678968 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e35bc33f-65c8-4fc4-abef-6e652686ec22-host-run-netns\") pod \"multus-r8rsb\" (UID: \"e35bc33f-65c8-4fc4-abef-6e652686ec22\") " pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.680245 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.678996 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/26ea58d8-26d9-4191-87b4-03d1efda477a-host\") pod \"tuned-wvrb9\" (UID: \"26ea58d8-26d9-4191-87b4-03d1efda477a\") " pod="openshift-cluster-node-tuning-operator/tuned-wvrb9" Sep 11 03:53:07.680245 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.679581 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 11 03:53:07.680245 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.679907 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pp9lb\"" Sep 11 03:53:07.680245 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.679992 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 11 03:53:07.680245 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.680088 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5fb2" Sep 11 03:53:07.680245 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:07.680147 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5fb2" podUID="9fc48d3a-bf43-44f7-b142-cd4937cea66b" Sep 11 03:53:07.680698 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.680314 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 11 03:53:07.695496 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.695438 2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4c5xj" Sep 11 03:53:07.702731 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.702712 2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4c5xj" Sep 11 03:53:07.739235 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.739213 2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 03:53:07.767658 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.767608 2568 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 11 03:53:07.779212 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.779180 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/26ea58d8-26d9-4191-87b4-03d1efda477a-etc-sysctl-conf\") pod \"tuned-wvrb9\" (UID: \"26ea58d8-26d9-4191-87b4-03d1efda477a\") " pod="openshift-cluster-node-tuning-operator/tuned-wvrb9" Sep 11 03:53:07.779212 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.779214 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/26ea58d8-26d9-4191-87b4-03d1efda477a-etc-tuned\") pod \"tuned-wvrb9\" (UID: \"26ea58d8-26d9-4191-87b4-03d1efda477a\") " pod="openshift-cluster-node-tuning-operator/tuned-wvrb9" Sep 11 03:53:07.779360 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.779230 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzj9q\" (UniqueName: \"kubernetes.io/projected/db235f27-7ffe-4c12-9d60-53fb8b810d97-kube-api-access-vzj9q\") pod \"node-ca-gmdt4\" (UID: \"db235f27-7ffe-4c12-9d60-53fb8b810d97\") " pod="openshift-image-registry/node-ca-gmdt4" Sep 11 03:53:07.779360 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.779253 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/44fdee40-4032-43ac-aabc-2393a05a4ef2-host-slash\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.779360 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.779277 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/44fdee40-4032-43ac-aabc-2393a05a4ef2-run-ovn\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.779360 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.779297 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e35bc33f-65c8-4fc4-abef-6e652686ec22-host-run-netns\") pod \"multus-r8rsb\" (UID: \"e35bc33f-65c8-4fc4-abef-6e652686ec22\") " pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.779360 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.779315 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/26ea58d8-26d9-4191-87b4-03d1efda477a-host\") pod \"tuned-wvrb9\" (UID: \"26ea58d8-26d9-4191-87b4-03d1efda477a\") " pod="openshift-cluster-node-tuning-operator/tuned-wvrb9" Sep 11 03:53:07.779360 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.779329 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/26ea58d8-26d9-4191-87b4-03d1efda477a-run\") pod \"tuned-wvrb9\" (UID: \"26ea58d8-26d9-4191-87b4-03d1efda477a\") " pod="openshift-cluster-node-tuning-operator/tuned-wvrb9" Sep 11 03:53:07.779360 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.779347 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aa66b45c-400f-46fa-b216-0e063413439b-agent-certs\") pod \"konnectivity-agent-c5drs\" (UID: \"aa66b45c-400f-46fa-b216-0e063413439b\") " pod="kube-system/konnectivity-agent-c5drs" Sep 11 03:53:07.779360 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.779363 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/44fdee40-4032-43ac-aabc-2393a05a4ef2-host-run-netns\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.779614 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.779376 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e35bc33f-65c8-4fc4-abef-6e652686ec22-host-run-k8s-cni-cncf-io\") pod \"multus-r8rsb\" (UID: \"e35bc33f-65c8-4fc4-abef-6e652686ec22\") " pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.779614 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.779391 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62v5d\" (UniqueName: \"kubernetes.io/projected/e35bc33f-65c8-4fc4-abef-6e652686ec22-kube-api-access-62v5d\") pod \"multus-r8rsb\" (UID: \"e35bc33f-65c8-4fc4-abef-6e652686ec22\") " pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.779614 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.779412 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6d855dff-6bf9-486f-aa54-479f9ba1fedc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hqk2s\" (UID: \"6d855dff-6bf9-486f-aa54-479f9ba1fedc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqk2s" Sep 11 03:53:07.779614 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.779429 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6906bf8-7095-4f6a-8969-d6f28167548b-system-cni-dir\") pod \"multus-additional-cni-plugins-9r65j\" (UID: \"d6906bf8-7095-4f6a-8969-d6f28167548b\") " pod="openshift-multus/multus-additional-cni-plugins-9r65j" Sep 11 03:53:07.779614 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.779442 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6906bf8-7095-4f6a-8969-d6f28167548b-cnibin\") pod \"multus-additional-cni-plugins-9r65j\" (UID: \"d6906bf8-7095-4f6a-8969-d6f28167548b\") " pod="openshift-multus/multus-additional-cni-plugins-9r65j" Sep 11 03:53:07.779614 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.779445 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e35bc33f-65c8-4fc4-abef-6e652686ec22-host-run-netns\") pod \"multus-r8rsb\" (UID: \"e35bc33f-65c8-4fc4-abef-6e652686ec22\") " pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.779614 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.779456 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/44fdee40-4032-43ac-aabc-2393a05a4ef2-etc-openvswitch\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.779614 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.779470 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/44fdee40-4032-43ac-aabc-2393a05a4ef2-host-cni-netd\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.779614 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.779491 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a48e2404-e37c-43d5-8902-b139cb2e07d9-tmp-dir\") pod \"node-resolver-7f284\" (UID: \"a48e2404-e37c-43d5-8902-b139cb2e07d9\") " pod="openshift-dns/node-resolver-7f284" Sep 11 03:53:07.779614 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.779506 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e35bc33f-65c8-4fc4-abef-6e652686ec22-cni-binary-copy\") pod \"multus-r8rsb\" (UID: \"e35bc33f-65c8-4fc4-abef-6e652686ec22\") " pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.779614 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.779521 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aa66b45c-400f-46fa-b216-0e063413439b-konnectivity-ca\") pod \"konnectivity-agent-c5drs\" (UID: \"aa66b45c-400f-46fa-b216-0e063413439b\") " pod="kube-system/konnectivity-agent-c5drs" Sep 11 03:53:07.779614 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.779535 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44fdee40-4032-43ac-aabc-2393a05a4ef2-host-run-ovn-kubernetes\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.779614 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.779550 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e35bc33f-65c8-4fc4-abef-6e652686ec22-host-run-multus-certs\") pod \"multus-r8rsb\" (UID: \"e35bc33f-65c8-4fc4-abef-6e652686ec22\") " pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.779614 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.779554 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/26ea58d8-26d9-4191-87b4-03d1efda477a-etc-sysctl-conf\") pod \"tuned-wvrb9\" (UID: \"26ea58d8-26d9-4191-87b4-03d1efda477a\") " pod="openshift-cluster-node-tuning-operator/tuned-wvrb9" Sep 11 03:53:07.779614 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.779567 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6d855dff-6bf9-486f-aa54-479f9ba1fedc-socket-dir\") pod \"aws-ebs-csi-driver-node-hqk2s\" (UID: \"6d855dff-6bf9-486f-aa54-479f9ba1fedc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqk2s" Sep 11 03:53:07.779614 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.779583 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44fdee40-4032-43ac-aabc-2393a05a4ef2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.779614 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.779600 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e35bc33f-65c8-4fc4-abef-6e652686ec22-multus-socket-dir-parent\") pod \"multus-r8rsb\" (UID: \"e35bc33f-65c8-4fc4-abef-6e652686ec22\") " pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.780248 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.779614 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/44fdee40-4032-43ac-aabc-2393a05a4ef2-ovnkube-script-lib\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.780248 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.779628 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/26ea58d8-26d9-4191-87b4-03d1efda477a-lib-modules\") pod \"tuned-wvrb9\" (UID: \"26ea58d8-26d9-4191-87b4-03d1efda477a\") " pod="openshift-cluster-node-tuning-operator/tuned-wvrb9" Sep 11 03:53:07.780248 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.779644 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe1486cf-6276-41a8-a685-025064c4abca-metrics-certs\") pod \"network-metrics-daemon-tk8q7\" (UID: \"fe1486cf-6276-41a8-a685-025064c4abca\") " pod="openshift-multus/network-metrics-daemon-tk8q7" Sep 11 03:53:07.780248 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.779659 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rldj\" (UniqueName: \"kubernetes.io/projected/6d855dff-6bf9-486f-aa54-479f9ba1fedc-kube-api-access-6rldj\") pod \"aws-ebs-csi-driver-node-hqk2s\" (UID: \"6d855dff-6bf9-486f-aa54-479f9ba1fedc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqk2s" Sep 11 03:53:07.780248 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.779674 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5729\" (UniqueName: \"kubernetes.io/projected/328d250b-1183-4ce5-8f21-f263e78a8b95-kube-api-access-k5729\") pod \"iptables-alerter-cvjtt\" (UID: \"328d250b-1183-4ce5-8f21-f263e78a8b95\") " pod="openshift-network-operator/iptables-alerter-cvjtt" Sep 11 03:53:07.780248 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.779692 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/44fdee40-4032-43ac-aabc-2393a05a4ef2-log-socket\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.780248 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.779716 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/44fdee40-4032-43ac-aabc-2393a05a4ef2-ovnkube-config\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.780248 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.779740 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb849\" (UniqueName: \"kubernetes.io/projected/44fdee40-4032-43ac-aabc-2393a05a4ef2-kube-api-access-sb849\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.780248 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.779764 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjh77\" (UniqueName: \"kubernetes.io/projected/a48e2404-e37c-43d5-8902-b139cb2e07d9-kube-api-access-mjh77\") pod \"node-resolver-7f284\" (UID: \"a48e2404-e37c-43d5-8902-b139cb2e07d9\") " pod="openshift-dns/node-resolver-7f284" Sep 11 03:53:07.780248 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.779791 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/26ea58d8-26d9-4191-87b4-03d1efda477a-etc-sysconfig\") pod \"tuned-wvrb9\" (UID: \"26ea58d8-26d9-4191-87b4-03d1efda477a\") " pod="openshift-cluster-node-tuning-operator/tuned-wvrb9" Sep 11 03:53:07.780248 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.779818 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/26ea58d8-26d9-4191-87b4-03d1efda477a-tmp\") pod \"tuned-wvrb9\" (UID: \"26ea58d8-26d9-4191-87b4-03d1efda477a\") " pod="openshift-cluster-node-tuning-operator/tuned-wvrb9" Sep 11 03:53:07.780248 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.779844 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6clwl\" (UniqueName: \"kubernetes.io/projected/fe1486cf-6276-41a8-a685-025064c4abca-kube-api-access-6clwl\") pod \"network-metrics-daemon-tk8q7\" (UID: \"fe1486cf-6276-41a8-a685-025064c4abca\") " pod="openshift-multus/network-metrics-daemon-tk8q7" Sep 11 03:53:07.780248 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.779868 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/44fdee40-4032-43ac-aabc-2393a05a4ef2-host-kubelet\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.780248 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.779891 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/44fdee40-4032-43ac-aabc-2393a05a4ef2-host-cni-bin\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.780248 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.779915 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnmts\" (UniqueName: \"kubernetes.io/projected/fb9f553c-03c7-4489-b43f-b62d630e7f50-kube-api-access-lnmts\") pod \"network-check-target-vssnm\" (UID: \"fb9f553c-03c7-4489-b43f-b62d630e7f50\") " pod="openshift-network-diagnostics/network-check-target-vssnm" Sep 11 03:53:07.780248 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.779940 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e35bc33f-65c8-4fc4-abef-6e652686ec22-host-var-lib-cni-bin\") pod \"multus-r8rsb\" (UID: \"e35bc33f-65c8-4fc4-abef-6e652686ec22\") " pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.780248 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.779966 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6d855dff-6bf9-486f-aa54-479f9ba1fedc-device-dir\") pod \"aws-ebs-csi-driver-node-hqk2s\" (UID: \"6d855dff-6bf9-486f-aa54-479f9ba1fedc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqk2s" Sep 11 03:53:07.780900 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.779999 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a48e2404-e37c-43d5-8902-b139cb2e07d9-tmp-dir\") pod \"node-resolver-7f284\" (UID: \"a48e2404-e37c-43d5-8902-b139cb2e07d9\") " pod="openshift-dns/node-resolver-7f284" Sep 11 03:53:07.780900 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.779999 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/db235f27-7ffe-4c12-9d60-53fb8b810d97-serviceca\") pod \"node-ca-gmdt4\" (UID: \"db235f27-7ffe-4c12-9d60-53fb8b810d97\") " pod="openshift-image-registry/node-ca-gmdt4" Sep 11 03:53:07.780900 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.780041 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/44fdee40-4032-43ac-aabc-2393a05a4ef2-systemd-units\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.780900 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.780082 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/44fdee40-4032-43ac-aabc-2393a05a4ef2-var-lib-openvswitch\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.780900 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.780104 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/44fdee40-4032-43ac-aabc-2393a05a4ef2-env-overrides\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.780900 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.780109 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/26ea58d8-26d9-4191-87b4-03d1efda477a-host\") pod \"tuned-wvrb9\" (UID: \"26ea58d8-26d9-4191-87b4-03d1efda477a\") " pod="openshift-cluster-node-tuning-operator/tuned-wvrb9" Sep 11 03:53:07.780900 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.780119 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/44fdee40-4032-43ac-aabc-2393a05a4ef2-ovn-node-metrics-cert\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.780900 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.780165 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/26ea58d8-26d9-4191-87b4-03d1efda477a-run\") pod \"tuned-wvrb9\" (UID: \"26ea58d8-26d9-4191-87b4-03d1efda477a\") " pod="openshift-cluster-node-tuning-operator/tuned-wvrb9" Sep 11 03:53:07.780900 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.780182 2568 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 11 03:53:07.780900 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.780465 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/44fdee40-4032-43ac-aabc-2393a05a4ef2-host-run-netns\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.780900 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.780519 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e35bc33f-65c8-4fc4-abef-6e652686ec22-host-run-k8s-cni-cncf-io\") pod \"multus-r8rsb\" (UID: \"e35bc33f-65c8-4fc4-abef-6e652686ec22\") " pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.780900 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.780803 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/44fdee40-4032-43ac-aabc-2393a05a4ef2-host-slash\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.781473 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.780977 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d6906bf8-7095-4f6a-8969-d6f28167548b-cnibin\") pod \"multus-additional-cni-plugins-9r65j\" (UID: \"d6906bf8-7095-4f6a-8969-d6f28167548b\") " pod="openshift-multus/multus-additional-cni-plugins-9r65j" Sep 11 03:53:07.781473 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.781195 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d6906bf8-7095-4f6a-8969-d6f28167548b-system-cni-dir\") pod \"multus-additional-cni-plugins-9r65j\" (UID: \"d6906bf8-7095-4f6a-8969-d6f28167548b\") " pod="openshift-multus/multus-additional-cni-plugins-9r65j" Sep 11 03:53:07.781473 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.781241 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/44fdee40-4032-43ac-aabc-2393a05a4ef2-host-cni-bin\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.781797 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.781629 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/26ea58d8-26d9-4191-87b4-03d1efda477a-etc-sysconfig\") pod \"tuned-wvrb9\" (UID: \"26ea58d8-26d9-4191-87b4-03d1efda477a\") " pod="openshift-cluster-node-tuning-operator/tuned-wvrb9" Sep 11 03:53:07.781797 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.781703 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/44fdee40-4032-43ac-aabc-2393a05a4ef2-ovnkube-config\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.781797 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.780196 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9fc48d3a-bf43-44f7-b142-cd4937cea66b-dbus\") pod \"global-pull-secret-syncer-t5fb2\" (UID: \"9fc48d3a-bf43-44f7-b142-cd4937cea66b\") " pod="kube-system/global-pull-secret-syncer-t5fb2" Sep 11 03:53:07.781965 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.781818 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d6906bf8-7095-4f6a-8969-d6f28167548b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9r65j\" (UID: \"d6906bf8-7095-4f6a-8969-d6f28167548b\") " pod="openshift-multus/multus-additional-cni-plugins-9r65j" Sep 11 03:53:07.781965 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.781851 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7xdg\" (UniqueName: \"kubernetes.io/projected/d6906bf8-7095-4f6a-8969-d6f28167548b-kube-api-access-v7xdg\") pod \"multus-additional-cni-plugins-9r65j\" (UID: \"d6906bf8-7095-4f6a-8969-d6f28167548b\") " pod="openshift-multus/multus-additional-cni-plugins-9r65j" Sep 11 03:53:07.781965 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.781879 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e35bc33f-65c8-4fc4-abef-6e652686ec22-multus-cni-dir\") pod \"multus-r8rsb\" (UID: \"e35bc33f-65c8-4fc4-abef-6e652686ec22\") " pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.781965 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.781910 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e35bc33f-65c8-4fc4-abef-6e652686ec22-cnibin\") pod \"multus-r8rsb\" (UID: \"e35bc33f-65c8-4fc4-abef-6e652686ec22\") " pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.781965 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.781937 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e35bc33f-65c8-4fc4-abef-6e652686ec22-multus-daemon-config\") pod \"multus-r8rsb\" (UID: \"e35bc33f-65c8-4fc4-abef-6e652686ec22\") " pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.781965 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.781960 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e35bc33f-65c8-4fc4-abef-6e652686ec22-etc-kubernetes\") pod \"multus-r8rsb\" (UID: \"e35bc33f-65c8-4fc4-abef-6e652686ec22\") " pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.782295 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.781987 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6d855dff-6bf9-486f-aa54-479f9ba1fedc-etc-selinux\") pod \"aws-ebs-csi-driver-node-hqk2s\" (UID: \"6d855dff-6bf9-486f-aa54-479f9ba1fedc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqk2s" Sep 11 03:53:07.782295 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.782014 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sfdp\" (UniqueName: \"kubernetes.io/projected/26ea58d8-26d9-4191-87b4-03d1efda477a-kube-api-access-9sfdp\") pod \"tuned-wvrb9\" (UID: \"26ea58d8-26d9-4191-87b4-03d1efda477a\") " pod="openshift-cluster-node-tuning-operator/tuned-wvrb9" Sep 11 03:53:07.782295 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.782084 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d6906bf8-7095-4f6a-8969-d6f28167548b-cni-binary-copy\") pod \"multus-additional-cni-plugins-9r65j\" (UID: \"d6906bf8-7095-4f6a-8969-d6f28167548b\") " pod="openshift-multus/multus-additional-cni-plugins-9r65j" Sep 11 03:53:07.782295 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.782111 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e35bc33f-65c8-4fc4-abef-6e652686ec22-host-var-lib-cni-multus\") pod \"multus-r8rsb\" (UID: \"e35bc33f-65c8-4fc4-abef-6e652686ec22\") " pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.782295 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.782137 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e35bc33f-65c8-4fc4-abef-6e652686ec22-multus-conf-dir\") pod \"multus-r8rsb\" (UID: \"e35bc33f-65c8-4fc4-abef-6e652686ec22\") " pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.782295 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.782162 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9fc48d3a-bf43-44f7-b142-cd4937cea66b-kubelet-config\") pod \"global-pull-secret-syncer-t5fb2\" (UID: \"9fc48d3a-bf43-44f7-b142-cd4937cea66b\") " pod="kube-system/global-pull-secret-syncer-t5fb2" Sep 11 03:53:07.782295 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.782185 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26ea58d8-26d9-4191-87b4-03d1efda477a-etc-kubernetes\") pod \"tuned-wvrb9\" (UID: \"26ea58d8-26d9-4191-87b4-03d1efda477a\") " pod="openshift-cluster-node-tuning-operator/tuned-wvrb9" Sep 11 03:53:07.782295 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.782217 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d6906bf8-7095-4f6a-8969-d6f28167548b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9r65j\" (UID: \"d6906bf8-7095-4f6a-8969-d6f28167548b\") " pod="openshift-multus/multus-additional-cni-plugins-9r65j" Sep 11 03:53:07.782295 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.782244 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e35bc33f-65c8-4fc4-abef-6e652686ec22-cni-binary-copy\") pod \"multus-r8rsb\" (UID: \"e35bc33f-65c8-4fc4-abef-6e652686ec22\") " pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.782295 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.782245 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/44fdee40-4032-43ac-aabc-2393a05a4ef2-run-openvswitch\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.782295 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.782296 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a48e2404-e37c-43d5-8902-b139cb2e07d9-hosts-file\") pod \"node-resolver-7f284\" (UID: \"a48e2404-e37c-43d5-8902-b139cb2e07d9\") " pod="openshift-dns/node-resolver-7f284" Sep 11 03:53:07.782831 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.782318 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/44fdee40-4032-43ac-aabc-2393a05a4ef2-run-ovn\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.782831 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.782325 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e35bc33f-65c8-4fc4-abef-6e652686ec22-system-cni-dir\") pod \"multus-r8rsb\" (UID: \"e35bc33f-65c8-4fc4-abef-6e652686ec22\") " pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.782831 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.782353 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e35bc33f-65c8-4fc4-abef-6e652686ec22-host-var-lib-cni-bin\") pod \"multus-r8rsb\" (UID: \"e35bc33f-65c8-4fc4-abef-6e652686ec22\") " pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.782831 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.782404 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e35bc33f-65c8-4fc4-abef-6e652686ec22-host-var-lib-kubelet\") pod \"multus-r8rsb\" (UID: \"e35bc33f-65c8-4fc4-abef-6e652686ec22\") " pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.783030 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.782858 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/db235f27-7ffe-4c12-9d60-53fb8b810d97-serviceca\") pod \"node-ca-gmdt4\" (UID: \"db235f27-7ffe-4c12-9d60-53fb8b810d97\") " pod="openshift-image-registry/node-ca-gmdt4" Sep 11 03:53:07.783030 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.782897 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/aa66b45c-400f-46fa-b216-0e063413439b-konnectivity-ca\") pod \"konnectivity-agent-c5drs\" (UID: \"aa66b45c-400f-46fa-b216-0e063413439b\") " pod="kube-system/konnectivity-agent-c5drs" Sep 11 03:53:07.783030 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.782920 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d6906bf8-7095-4f6a-8969-d6f28167548b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9r65j\" (UID: \"d6906bf8-7095-4f6a-8969-d6f28167548b\") " pod="openshift-multus/multus-additional-cni-plugins-9r65j" Sep 11 03:53:07.783030 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.782921 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/44fdee40-4032-43ac-aabc-2393a05a4ef2-systemd-units\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.783030 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.782958 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44fdee40-4032-43ac-aabc-2393a05a4ef2-host-run-ovn-kubernetes\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.783030 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.783006 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e35bc33f-65c8-4fc4-abef-6e652686ec22-host-run-multus-certs\") pod \"multus-r8rsb\" (UID: \"e35bc33f-65c8-4fc4-abef-6e652686ec22\") " pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.783340 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.783079 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44fdee40-4032-43ac-aabc-2393a05a4ef2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.783340 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.783171 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e35bc33f-65c8-4fc4-abef-6e652686ec22-multus-socket-dir-parent\") pod \"multus-r8rsb\" (UID: \"e35bc33f-65c8-4fc4-abef-6e652686ec22\") " pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.783340 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.783173 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e35bc33f-65c8-4fc4-abef-6e652686ec22-multus-cni-dir\") pod \"multus-r8rsb\" (UID: \"e35bc33f-65c8-4fc4-abef-6e652686ec22\") " pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.783340 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.783228 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e35bc33f-65c8-4fc4-abef-6e652686ec22-cnibin\") pod \"multus-r8rsb\" (UID: \"e35bc33f-65c8-4fc4-abef-6e652686ec22\") " pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.783340 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.783327 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/44fdee40-4032-43ac-aabc-2393a05a4ef2-env-overrides\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.783592 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.782360 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e35bc33f-65c8-4fc4-abef-6e652686ec22-host-var-lib-kubelet\") pod \"multus-r8rsb\" (UID: \"e35bc33f-65c8-4fc4-abef-6e652686ec22\") " pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.783592 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.783385 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6d855dff-6bf9-486f-aa54-479f9ba1fedc-registration-dir\") pod \"aws-ebs-csi-driver-node-hqk2s\" (UID: \"6d855dff-6bf9-486f-aa54-479f9ba1fedc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqk2s" Sep 11 03:53:07.783592 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.783414 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/328d250b-1183-4ce5-8f21-f263e78a8b95-host-slash\") pod \"iptables-alerter-cvjtt\" (UID: \"328d250b-1183-4ce5-8f21-f263e78a8b95\") " pod="openshift-network-operator/iptables-alerter-cvjtt" Sep 11 03:53:07.783592 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.783437 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d6906bf8-7095-4f6a-8969-d6f28167548b-cni-binary-copy\") pod \"multus-additional-cni-plugins-9r65j\" (UID: \"d6906bf8-7095-4f6a-8969-d6f28167548b\") " pod="openshift-multus/multus-additional-cni-plugins-9r65j" Sep 11 03:53:07.783592 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.783445 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/26ea58d8-26d9-4191-87b4-03d1efda477a-etc-sysctl-d\") pod \"tuned-wvrb9\" (UID: \"26ea58d8-26d9-4191-87b4-03d1efda477a\") " pod="openshift-cluster-node-tuning-operator/tuned-wvrb9" Sep 11 03:53:07.783592 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.783474 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26ea58d8-26d9-4191-87b4-03d1efda477a-var-lib-kubelet\") pod \"tuned-wvrb9\" (UID: \"26ea58d8-26d9-4191-87b4-03d1efda477a\") " pod="openshift-cluster-node-tuning-operator/tuned-wvrb9" Sep 11 03:53:07.783592 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.783494 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/44fdee40-4032-43ac-aabc-2393a05a4ef2-var-lib-openvswitch\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.783592 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.783498 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6d855dff-6bf9-486f-aa54-479f9ba1fedc-sys-fs\") pod \"aws-ebs-csi-driver-node-hqk2s\" (UID: \"6d855dff-6bf9-486f-aa54-479f9ba1fedc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqk2s" Sep 11 03:53:07.783592 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.783544 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e35bc33f-65c8-4fc4-abef-6e652686ec22-host-var-lib-cni-multus\") pod \"multus-r8rsb\" (UID: \"e35bc33f-65c8-4fc4-abef-6e652686ec22\") " pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.783592 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.783549 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6906bf8-7095-4f6a-8969-d6f28167548b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9r65j\" (UID: \"d6906bf8-7095-4f6a-8969-d6f28167548b\") " pod="openshift-multus/multus-additional-cni-plugins-9r65j" Sep 11 03:53:07.783592 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.783578 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/db235f27-7ffe-4c12-9d60-53fb8b810d97-host\") pod \"node-ca-gmdt4\" (UID: \"db235f27-7ffe-4c12-9d60-53fb8b810d97\") " pod="openshift-image-registry/node-ca-gmdt4" Sep 11 03:53:07.783592 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.783590 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e35bc33f-65c8-4fc4-abef-6e652686ec22-multus-conf-dir\") pod \"multus-r8rsb\" (UID: \"e35bc33f-65c8-4fc4-abef-6e652686ec22\") " pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.784190 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.783605 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/26ea58d8-26d9-4191-87b4-03d1efda477a-etc-modprobe-d\") pod \"tuned-wvrb9\" (UID: \"26ea58d8-26d9-4191-87b4-03d1efda477a\") " pod="openshift-cluster-node-tuning-operator/tuned-wvrb9" Sep 11 03:53:07.784190 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.783632 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/26ea58d8-26d9-4191-87b4-03d1efda477a-etc-systemd\") pod \"tuned-wvrb9\" (UID: \"26ea58d8-26d9-4191-87b4-03d1efda477a\") " pod="openshift-cluster-node-tuning-operator/tuned-wvrb9" Sep 11 03:53:07.784190 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.783654 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/44fdee40-4032-43ac-aabc-2393a05a4ef2-node-log\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.784190 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.783672 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e35bc33f-65c8-4fc4-abef-6e652686ec22-multus-daemon-config\") pod \"multus-r8rsb\" (UID: \"e35bc33f-65c8-4fc4-abef-6e652686ec22\") " pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.784190 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.783679 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/328d250b-1183-4ce5-8f21-f263e78a8b95-iptables-alerter-script\") pod \"iptables-alerter-cvjtt\" (UID: \"328d250b-1183-4ce5-8f21-f263e78a8b95\") " pod="openshift-network-operator/iptables-alerter-cvjtt" Sep 11 03:53:07.784190 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.783679 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/44fdee40-4032-43ac-aabc-2393a05a4ef2-ovnkube-script-lib\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.784190 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.783726 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/44fdee40-4032-43ac-aabc-2393a05a4ef2-run-systemd\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.784190 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.783755 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e35bc33f-65c8-4fc4-abef-6e652686ec22-hostroot\") pod \"multus-r8rsb\" (UID: \"e35bc33f-65c8-4fc4-abef-6e652686ec22\") " pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.784190 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.783763 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/26ea58d8-26d9-4191-87b4-03d1efda477a-lib-modules\") pod \"tuned-wvrb9\" (UID: \"26ea58d8-26d9-4191-87b4-03d1efda477a\") " pod="openshift-cluster-node-tuning-operator/tuned-wvrb9" Sep 11 03:53:07.784190 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.783783 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9fc48d3a-bf43-44f7-b142-cd4937cea66b-original-pull-secret\") pod \"global-pull-secret-syncer-t5fb2\" (UID: \"9fc48d3a-bf43-44f7-b142-cd4937cea66b\") " pod="kube-system/global-pull-secret-syncer-t5fb2" Sep 11 03:53:07.784190 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.783791 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/26ea58d8-26d9-4191-87b4-03d1efda477a-etc-sysctl-d\") pod \"tuned-wvrb9\" (UID: \"26ea58d8-26d9-4191-87b4-03d1efda477a\") " pod="openshift-cluster-node-tuning-operator/tuned-wvrb9" Sep 11 03:53:07.784190 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.783811 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/26ea58d8-26d9-4191-87b4-03d1efda477a-sys\") pod \"tuned-wvrb9\" (UID: \"26ea58d8-26d9-4191-87b4-03d1efda477a\") " pod="openshift-cluster-node-tuning-operator/tuned-wvrb9" Sep 11 03:53:07.784190 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:07.783850 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 03:53:07.784190 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.783891 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/26ea58d8-26d9-4191-87b4-03d1efda477a-sys\") pod \"tuned-wvrb9\" (UID: \"26ea58d8-26d9-4191-87b4-03d1efda477a\") " pod="openshift-cluster-node-tuning-operator/tuned-wvrb9" Sep 11 03:53:07.784190 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:07.783917 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe1486cf-6276-41a8-a685-025064c4abca-metrics-certs podName:fe1486cf-6276-41a8-a685-025064c4abca nodeName:}" failed. No retries permitted until 2026-09-11 03:53:08.283899348 +0000 UTC m=+3.037530739 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe1486cf-6276-41a8-a685-025064c4abca-metrics-certs") pod "network-metrics-daemon-tk8q7" (UID: "fe1486cf-6276-41a8-a685-025064c4abca") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 03:53:07.784190 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.783933 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e35bc33f-65c8-4fc4-abef-6e652686ec22-etc-kubernetes\") pod \"multus-r8rsb\" (UID: \"e35bc33f-65c8-4fc4-abef-6e652686ec22\") " pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.784190 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.783942 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d6906bf8-7095-4f6a-8969-d6f28167548b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9r65j\" (UID: \"d6906bf8-7095-4f6a-8969-d6f28167548b\") " pod="openshift-multus/multus-additional-cni-plugins-9r65j" Sep 11 03:53:07.784761 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.783984 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e35bc33f-65c8-4fc4-abef-6e652686ec22-hostroot\") pod \"multus-r8rsb\" (UID: \"e35bc33f-65c8-4fc4-abef-6e652686ec22\") " pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.784761 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.783984 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/44fdee40-4032-43ac-aabc-2393a05a4ef2-log-socket\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.784761 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.783986 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/db235f27-7ffe-4c12-9d60-53fb8b810d97-host\") pod \"node-ca-gmdt4\" (UID: \"db235f27-7ffe-4c12-9d60-53fb8b810d97\") " pod="openshift-image-registry/node-ca-gmdt4" Sep 11 03:53:07.784761 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.784022 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a48e2404-e37c-43d5-8902-b139cb2e07d9-hosts-file\") pod \"node-resolver-7f284\" (UID: \"a48e2404-e37c-43d5-8902-b139cb2e07d9\") " pod="openshift-dns/node-resolver-7f284" Sep 11 03:53:07.784761 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.784084 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/44fdee40-4032-43ac-aabc-2393a05a4ef2-run-systemd\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.784761 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.784085 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/26ea58d8-26d9-4191-87b4-03d1efda477a-etc-modprobe-d\") pod \"tuned-wvrb9\" (UID: \"26ea58d8-26d9-4191-87b4-03d1efda477a\") " pod="openshift-cluster-node-tuning-operator/tuned-wvrb9" Sep 11 03:53:07.784761 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.784088 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d6906bf8-7095-4f6a-8969-d6f28167548b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9r65j\" (UID: \"d6906bf8-7095-4f6a-8969-d6f28167548b\") " pod="openshift-multus/multus-additional-cni-plugins-9r65j" Sep 11 03:53:07.784761 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.784137 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/26ea58d8-26d9-4191-87b4-03d1efda477a-etc-kubernetes\") pod \"tuned-wvrb9\" (UID: \"26ea58d8-26d9-4191-87b4-03d1efda477a\") " pod="openshift-cluster-node-tuning-operator/tuned-wvrb9" Sep 11 03:53:07.784761 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.784155 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/26ea58d8-26d9-4191-87b4-03d1efda477a-var-lib-kubelet\") pod \"tuned-wvrb9\" (UID: \"26ea58d8-26d9-4191-87b4-03d1efda477a\") " pod="openshift-cluster-node-tuning-operator/tuned-wvrb9" Sep 11 03:53:07.784761 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.784191 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/26ea58d8-26d9-4191-87b4-03d1efda477a-etc-systemd\") pod \"tuned-wvrb9\" (UID: \"26ea58d8-26d9-4191-87b4-03d1efda477a\") " pod="openshift-cluster-node-tuning-operator/tuned-wvrb9" Sep 11 03:53:07.784761 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.784203 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e35bc33f-65c8-4fc4-abef-6e652686ec22-system-cni-dir\") pod \"multus-r8rsb\" (UID: \"e35bc33f-65c8-4fc4-abef-6e652686ec22\") " pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.784761 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.784209 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/44fdee40-4032-43ac-aabc-2393a05a4ef2-run-openvswitch\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.784761 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.784229 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/44fdee40-4032-43ac-aabc-2393a05a4ef2-node-log\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.784761 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.784263 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/44fdee40-4032-43ac-aabc-2393a05a4ef2-etc-openvswitch\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.784761 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.784268 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/44fdee40-4032-43ac-aabc-2393a05a4ef2-host-cni-netd\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.784761 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.784266 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/44fdee40-4032-43ac-aabc-2393a05a4ef2-host-kubelet\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.784761 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.784451 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/26ea58d8-26d9-4191-87b4-03d1efda477a-etc-tuned\") pod \"tuned-wvrb9\" (UID: \"26ea58d8-26d9-4191-87b4-03d1efda477a\") " pod="openshift-cluster-node-tuning-operator/tuned-wvrb9" Sep 11 03:53:07.784761 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.784527 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/44fdee40-4032-43ac-aabc-2393a05a4ef2-ovn-node-metrics-cert\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.785777 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.785746 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/aa66b45c-400f-46fa-b216-0e063413439b-agent-certs\") pod \"konnectivity-agent-c5drs\" (UID: \"aa66b45c-400f-46fa-b216-0e063413439b\") " pod="kube-system/konnectivity-agent-c5drs" Sep 11 03:53:07.785896 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.785755 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/26ea58d8-26d9-4191-87b4-03d1efda477a-tmp\") pod \"tuned-wvrb9\" (UID: \"26ea58d8-26d9-4191-87b4-03d1efda477a\") " pod="openshift-cluster-node-tuning-operator/tuned-wvrb9" Sep 11 03:53:07.788945 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:07.788920 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 03:53:07.788945 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:07.788942 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 03:53:07.789183 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:07.788955 2568 projected.go:196] Error preparing data for projected volume kube-api-access-lnmts for pod openshift-network-diagnostics/network-check-target-vssnm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 03:53:07.789183 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:07.789014 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb9f553c-03c7-4489-b43f-b62d630e7f50-kube-api-access-lnmts podName:fb9f553c-03c7-4489-b43f-b62d630e7f50 nodeName:}" failed. No retries permitted until 2026-09-11 03:53:08.28899598 +0000 UTC m=+3.042627380 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lnmts" (UniqueName: "kubernetes.io/projected/fb9f553c-03c7-4489-b43f-b62d630e7f50-kube-api-access-lnmts") pod "network-check-target-vssnm" (UID: "fb9f553c-03c7-4489-b43f-b62d630e7f50") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 03:53:07.791079 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.791041 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzj9q\" (UniqueName: \"kubernetes.io/projected/db235f27-7ffe-4c12-9d60-53fb8b810d97-kube-api-access-vzj9q\") pod \"node-ca-gmdt4\" (UID: \"db235f27-7ffe-4c12-9d60-53fb8b810d97\") " pod="openshift-image-registry/node-ca-gmdt4" Sep 11 03:53:07.791712 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.791659 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62v5d\" (UniqueName: \"kubernetes.io/projected/e35bc33f-65c8-4fc4-abef-6e652686ec22-kube-api-access-62v5d\") pod \"multus-r8rsb\" (UID: \"e35bc33f-65c8-4fc4-abef-6e652686ec22\") " pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.791903 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.791883 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sfdp\" (UniqueName: \"kubernetes.io/projected/26ea58d8-26d9-4191-87b4-03d1efda477a-kube-api-access-9sfdp\") pod \"tuned-wvrb9\" (UID: \"26ea58d8-26d9-4191-87b4-03d1efda477a\") " pod="openshift-cluster-node-tuning-operator/tuned-wvrb9" Sep 11 03:53:07.793741 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.793718 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6clwl\" (UniqueName: \"kubernetes.io/projected/fe1486cf-6276-41a8-a685-025064c4abca-kube-api-access-6clwl\") pod \"network-metrics-daemon-tk8q7\" (UID: \"fe1486cf-6276-41a8-a685-025064c4abca\") " pod="openshift-multus/network-metrics-daemon-tk8q7" Sep 11 03:53:07.793864 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.793840 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7xdg\" (UniqueName: \"kubernetes.io/projected/d6906bf8-7095-4f6a-8969-d6f28167548b-kube-api-access-v7xdg\") pod \"multus-additional-cni-plugins-9r65j\" (UID: \"d6906bf8-7095-4f6a-8969-d6f28167548b\") " pod="openshift-multus/multus-additional-cni-plugins-9r65j" Sep 11 03:53:07.793864 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.793849 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb849\" (UniqueName: \"kubernetes.io/projected/44fdee40-4032-43ac-aabc-2393a05a4ef2-kube-api-access-sb849\") pod \"ovnkube-node-htjph\" (UID: \"44fdee40-4032-43ac-aabc-2393a05a4ef2\") " pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:07.795408 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.795387 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjh77\" (UniqueName: \"kubernetes.io/projected/a48e2404-e37c-43d5-8902-b139cb2e07d9-kube-api-access-mjh77\") pod \"node-resolver-7f284\" (UID: \"a48e2404-e37c-43d5-8902-b139cb2e07d9\") " pod="openshift-dns/node-resolver-7f284" Sep 11 03:53:07.884348 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.884320 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6d855dff-6bf9-486f-aa54-479f9ba1fedc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hqk2s\" (UID: \"6d855dff-6bf9-486f-aa54-479f9ba1fedc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqk2s" Sep 11 03:53:07.884501 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.884354 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6d855dff-6bf9-486f-aa54-479f9ba1fedc-socket-dir\") pod \"aws-ebs-csi-driver-node-hqk2s\" (UID: \"6d855dff-6bf9-486f-aa54-479f9ba1fedc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqk2s" Sep 11 03:53:07.884501 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.884389 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rldj\" (UniqueName: \"kubernetes.io/projected/6d855dff-6bf9-486f-aa54-479f9ba1fedc-kube-api-access-6rldj\") pod \"aws-ebs-csi-driver-node-hqk2s\" (UID: \"6d855dff-6bf9-486f-aa54-479f9ba1fedc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqk2s" Sep 11 03:53:07.884501 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.884415 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5729\" (UniqueName: \"kubernetes.io/projected/328d250b-1183-4ce5-8f21-f263e78a8b95-kube-api-access-k5729\") pod \"iptables-alerter-cvjtt\" (UID: \"328d250b-1183-4ce5-8f21-f263e78a8b95\") " pod="openshift-network-operator/iptables-alerter-cvjtt" Sep 11 03:53:07.884501 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.884445 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6d855dff-6bf9-486f-aa54-479f9ba1fedc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hqk2s\" (UID: \"6d855dff-6bf9-486f-aa54-479f9ba1fedc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqk2s" Sep 11 03:53:07.884501 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.884446 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6d855dff-6bf9-486f-aa54-479f9ba1fedc-device-dir\") pod \"aws-ebs-csi-driver-node-hqk2s\" (UID: \"6d855dff-6bf9-486f-aa54-479f9ba1fedc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqk2s" Sep 11 03:53:07.884501 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.884500 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9fc48d3a-bf43-44f7-b142-cd4937cea66b-dbus\") pod \"global-pull-secret-syncer-t5fb2\" (UID: \"9fc48d3a-bf43-44f7-b142-cd4937cea66b\") " pod="kube-system/global-pull-secret-syncer-t5fb2" Sep 11 03:53:07.884763 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.884528 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6d855dff-6bf9-486f-aa54-479f9ba1fedc-etc-selinux\") pod \"aws-ebs-csi-driver-node-hqk2s\" (UID: \"6d855dff-6bf9-486f-aa54-479f9ba1fedc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqk2s" Sep 11 03:53:07.884763 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.884530 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6d855dff-6bf9-486f-aa54-479f9ba1fedc-socket-dir\") pod \"aws-ebs-csi-driver-node-hqk2s\" (UID: \"6d855dff-6bf9-486f-aa54-479f9ba1fedc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqk2s" Sep 11 03:53:07.884763 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.884593 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9fc48d3a-bf43-44f7-b142-cd4937cea66b-dbus\") pod \"global-pull-secret-syncer-t5fb2\" (UID: \"9fc48d3a-bf43-44f7-b142-cd4937cea66b\") " pod="kube-system/global-pull-secret-syncer-t5fb2" Sep 11 03:53:07.884763 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.884649 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6d855dff-6bf9-486f-aa54-479f9ba1fedc-etc-selinux\") pod \"aws-ebs-csi-driver-node-hqk2s\" (UID: \"6d855dff-6bf9-486f-aa54-479f9ba1fedc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqk2s" Sep 11 03:53:07.884763 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.884646 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9fc48d3a-bf43-44f7-b142-cd4937cea66b-kubelet-config\") pod \"global-pull-secret-syncer-t5fb2\" (UID: \"9fc48d3a-bf43-44f7-b142-cd4937cea66b\") " pod="kube-system/global-pull-secret-syncer-t5fb2" Sep 11 03:53:07.884763 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.884687 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6d855dff-6bf9-486f-aa54-479f9ba1fedc-registration-dir\") pod \"aws-ebs-csi-driver-node-hqk2s\" (UID: \"6d855dff-6bf9-486f-aa54-479f9ba1fedc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqk2s" Sep 11 03:53:07.884763 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.884705 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9fc48d3a-bf43-44f7-b142-cd4937cea66b-kubelet-config\") pod \"global-pull-secret-syncer-t5fb2\" (UID: \"9fc48d3a-bf43-44f7-b142-cd4937cea66b\") " pod="kube-system/global-pull-secret-syncer-t5fb2" Sep 11 03:53:07.884763 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.884721 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/328d250b-1183-4ce5-8f21-f263e78a8b95-host-slash\") pod \"iptables-alerter-cvjtt\" (UID: \"328d250b-1183-4ce5-8f21-f263e78a8b95\") " pod="openshift-network-operator/iptables-alerter-cvjtt" Sep 11 03:53:07.884763 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.884754 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/328d250b-1183-4ce5-8f21-f263e78a8b95-host-slash\") pod \"iptables-alerter-cvjtt\" (UID: \"328d250b-1183-4ce5-8f21-f263e78a8b95\") " pod="openshift-network-operator/iptables-alerter-cvjtt" Sep 11 03:53:07.885119 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.884769 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6d855dff-6bf9-486f-aa54-479f9ba1fedc-sys-fs\") pod \"aws-ebs-csi-driver-node-hqk2s\" (UID: \"6d855dff-6bf9-486f-aa54-479f9ba1fedc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqk2s" Sep 11 03:53:07.885119 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.884779 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6d855dff-6bf9-486f-aa54-479f9ba1fedc-registration-dir\") pod \"aws-ebs-csi-driver-node-hqk2s\" (UID: \"6d855dff-6bf9-486f-aa54-479f9ba1fedc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqk2s" Sep 11 03:53:07.885119 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.884818 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/328d250b-1183-4ce5-8f21-f263e78a8b95-iptables-alerter-script\") pod \"iptables-alerter-cvjtt\" (UID: \"328d250b-1183-4ce5-8f21-f263e78a8b95\") " pod="openshift-network-operator/iptables-alerter-cvjtt" Sep 11 03:53:07.885119 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.884820 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6d855dff-6bf9-486f-aa54-479f9ba1fedc-device-dir\") pod \"aws-ebs-csi-driver-node-hqk2s\" (UID: \"6d855dff-6bf9-486f-aa54-479f9ba1fedc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqk2s" Sep 11 03:53:07.885119 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.884869 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6d855dff-6bf9-486f-aa54-479f9ba1fedc-sys-fs\") pod \"aws-ebs-csi-driver-node-hqk2s\" (UID: \"6d855dff-6bf9-486f-aa54-479f9ba1fedc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqk2s" Sep 11 03:53:07.885119 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.884927 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9fc48d3a-bf43-44f7-b142-cd4937cea66b-original-pull-secret\") pod \"global-pull-secret-syncer-t5fb2\" (UID: \"9fc48d3a-bf43-44f7-b142-cd4937cea66b\") " pod="kube-system/global-pull-secret-syncer-t5fb2" Sep 11 03:53:07.885119 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:07.885010 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 03:53:07.885394 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:07.885154 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fc48d3a-bf43-44f7-b142-cd4937cea66b-original-pull-secret podName:9fc48d3a-bf43-44f7-b142-cd4937cea66b nodeName:}" failed. No retries permitted until 2026-09-11 03:53:08.385135027 +0000 UTC m=+3.138766411 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9fc48d3a-bf43-44f7-b142-cd4937cea66b-original-pull-secret") pod "global-pull-secret-syncer-t5fb2" (UID: "9fc48d3a-bf43-44f7-b142-cd4937cea66b") : object "kube-system"/"original-pull-secret" not registered Sep 11 03:53:07.885394 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.885351 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/328d250b-1183-4ce5-8f21-f263e78a8b95-iptables-alerter-script\") pod \"iptables-alerter-cvjtt\" (UID: \"328d250b-1183-4ce5-8f21-f263e78a8b95\") " pod="openshift-network-operator/iptables-alerter-cvjtt" Sep 11 03:53:07.897215 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.897162 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5729\" (UniqueName: \"kubernetes.io/projected/328d250b-1183-4ce5-8f21-f263e78a8b95-kube-api-access-k5729\") pod \"iptables-alerter-cvjtt\" (UID: \"328d250b-1183-4ce5-8f21-f263e78a8b95\") " pod="openshift-network-operator/iptables-alerter-cvjtt" Sep 11 03:53:07.897595 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.897572 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rldj\" (UniqueName: \"kubernetes.io/projected/6d855dff-6bf9-486f-aa54-479f9ba1fedc-kube-api-access-6rldj\") pod \"aws-ebs-csi-driver-node-hqk2s\" (UID: \"6d855dff-6bf9-486f-aa54-479f9ba1fedc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqk2s" Sep 11 03:53:07.961381 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.961357 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9r65j" Sep 11 03:53:07.972764 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.972746 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r8rsb" Sep 11 03:53:07.981337 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.981318 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wvrb9" Sep 11 03:53:07.986990 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.986968 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gmdt4" Sep 11 03:53:07.995613 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:07.995596 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-c5drs" Sep 11 03:53:08.003324 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:08.003295 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:08.010936 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:08.010918 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7f284" Sep 11 03:53:08.018579 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:08.018563 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqk2s" Sep 11 03:53:08.025248 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:08.025228 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-cvjtt" Sep 11 03:53:08.288538 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:08.288499 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe1486cf-6276-41a8-a685-025064c4abca-metrics-certs\") pod \"network-metrics-daemon-tk8q7\" (UID: \"fe1486cf-6276-41a8-a685-025064c4abca\") " pod="openshift-multus/network-metrics-daemon-tk8q7" Sep 11 03:53:08.288727 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:08.288665 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 03:53:08.288785 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:08.288733 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe1486cf-6276-41a8-a685-025064c4abca-metrics-certs podName:fe1486cf-6276-41a8-a685-025064c4abca nodeName:}" failed. No retries permitted until 2026-09-11 03:53:09.288715234 +0000 UTC m=+4.042346619 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe1486cf-6276-41a8-a685-025064c4abca-metrics-certs") pod "network-metrics-daemon-tk8q7" (UID: "fe1486cf-6276-41a8-a685-025064c4abca") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 03:53:08.389569 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:08.389535 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9fc48d3a-bf43-44f7-b142-cd4937cea66b-original-pull-secret\") pod \"global-pull-secret-syncer-t5fb2\" (UID: \"9fc48d3a-bf43-44f7-b142-cd4937cea66b\") " pod="kube-system/global-pull-secret-syncer-t5fb2" Sep 11 03:53:08.389736 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:08.389610 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnmts\" (UniqueName: \"kubernetes.io/projected/fb9f553c-03c7-4489-b43f-b62d630e7f50-kube-api-access-lnmts\") pod \"network-check-target-vssnm\" (UID: \"fb9f553c-03c7-4489-b43f-b62d630e7f50\") " pod="openshift-network-diagnostics/network-check-target-vssnm" Sep 11 03:53:08.389736 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:08.389675 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 03:53:08.389736 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:08.389737 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fc48d3a-bf43-44f7-b142-cd4937cea66b-original-pull-secret podName:9fc48d3a-bf43-44f7-b142-cd4937cea66b nodeName:}" failed. No retries permitted until 2026-09-11 03:53:09.389723213 +0000 UTC m=+4.143354606 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9fc48d3a-bf43-44f7-b142-cd4937cea66b-original-pull-secret") pod "global-pull-secret-syncer-t5fb2" (UID: "9fc48d3a-bf43-44f7-b142-cd4937cea66b") : object "kube-system"/"original-pull-secret" not registered Sep 11 03:53:08.389900 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:08.389743 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 03:53:08.389900 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:08.389760 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 03:53:08.389900 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:08.389773 2568 projected.go:196] Error preparing data for projected volume kube-api-access-lnmts for pod openshift-network-diagnostics/network-check-target-vssnm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 03:53:08.389900 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:08.389832 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb9f553c-03c7-4489-b43f-b62d630e7f50-kube-api-access-lnmts podName:fb9f553c-03c7-4489-b43f-b62d630e7f50 nodeName:}" failed. No retries permitted until 2026-09-11 03:53:09.389814738 +0000 UTC m=+4.143446137 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lnmts" (UniqueName: "kubernetes.io/projected/fb9f553c-03c7-4489-b43f-b62d630e7f50-kube-api-access-lnmts") pod "network-check-target-vssnm" (UID: "fb9f553c-03c7-4489-b43f-b62d630e7f50") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 03:53:08.706271 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:08.706218 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 03:48:07 +0000 UTC" deadline="2028-05-05 00:35:19.465591217 +0000 UTC" Sep 11 03:53:08.706271 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:08.706266 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14444h42m10.759328385s" Sep 11 03:53:08.738125 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:08.738089 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6906bf8_7095_4f6a_8969_d6f28167548b.slice/crio-3ad5f7ab5ef19f84251f8e90cfad88fcb0cfff4e51ce863b52c34ca77729579f WatchSource:0}: Error finding container 3ad5f7ab5ef19f84251f8e90cfad88fcb0cfff4e51ce863b52c34ca77729579f: Status 404 returned error can't find the container with id 3ad5f7ab5ef19f84251f8e90cfad88fcb0cfff4e51ce863b52c34ca77729579f Sep 11 03:53:08.738720 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:08.738698 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode35bc33f_65c8_4fc4_abef_6e652686ec22.slice/crio-67f4ed53d12ddc1501f1ef8ce2c09b0dd34e798e1be89c488e679ee1f4d28944 WatchSource:0}: Error finding container 67f4ed53d12ddc1501f1ef8ce2c09b0dd34e798e1be89c488e679ee1f4d28944: Status 404 returned error can't find the container with id 67f4ed53d12ddc1501f1ef8ce2c09b0dd34e798e1be89c488e679ee1f4d28944 Sep 11 03:53:08.744157 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:08.744129 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda48e2404_e37c_43d5_8902_b139cb2e07d9.slice/crio-accb255f0b704018342e2949f12e9e9b331cc4c73dfd92e2df531801ef070cbf WatchSource:0}: Error finding container accb255f0b704018342e2949f12e9e9b331cc4c73dfd92e2df531801ef070cbf: Status 404 returned error can't find the container with id accb255f0b704018342e2949f12e9e9b331cc4c73dfd92e2df531801ef070cbf Sep 11 03:53:08.745643 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:08.745606 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d855dff_6bf9_486f_aa54_479f9ba1fedc.slice/crio-9ef66c76ca75f9dfc28ad7240798254b4f57bdda4917fa1f7b8fb8e85fc4485a WatchSource:0}: Error finding container 9ef66c76ca75f9dfc28ad7240798254b4f57bdda4917fa1f7b8fb8e85fc4485a: Status 404 returned error can't find the container with id 9ef66c76ca75f9dfc28ad7240798254b4f57bdda4917fa1f7b8fb8e85fc4485a Sep 11 03:53:08.746487 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:08.746438 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44fdee40_4032_43ac_aabc_2393a05a4ef2.slice/crio-540ba26b53dd7cc525733eda1e2ad20b2eab3d8193d472278a107ba5efbbab72 WatchSource:0}: Error finding container 540ba26b53dd7cc525733eda1e2ad20b2eab3d8193d472278a107ba5efbbab72: Status 404 returned error can't find the container with id 540ba26b53dd7cc525733eda1e2ad20b2eab3d8193d472278a107ba5efbbab72 Sep 11 03:53:08.748759 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:08.748730 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod328d250b_1183_4ce5_8f21_f263e78a8b95.slice/crio-925d641fc37569e5a324a68791b8ec03e54ac65b8e0f7cfc86340982129cdda0 WatchSource:0}: Error finding container 925d641fc37569e5a324a68791b8ec03e54ac65b8e0f7cfc86340982129cdda0: Status 404 returned error can't find the container with id 925d641fc37569e5a324a68791b8ec03e54ac65b8e0f7cfc86340982129cdda0 Sep 11 03:53:08.750468 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:08.750449 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa66b45c_400f_46fa_b216_0e063413439b.slice/crio-05f600083d510413fefa60a72bda7fd15e85f02dd8eb31f83fc9c5eca094194a WatchSource:0}: Error finding container 05f600083d510413fefa60a72bda7fd15e85f02dd8eb31f83fc9c5eca094194a: Status 404 returned error can't find the container with id 05f600083d510413fefa60a72bda7fd15e85f02dd8eb31f83fc9c5eca094194a Sep 11 03:53:08.750973 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:08.750849 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb235f27_7ffe_4c12_9d60_53fb8b810d97.slice/crio-cb14f9c143512d17d4f30d483703fe5510c7e9e146523d8badd1984d486ddecc WatchSource:0}: Error finding container cb14f9c143512d17d4f30d483703fe5510c7e9e146523d8badd1984d486ddecc: Status 404 returned error can't find the container with id cb14f9c143512d17d4f30d483703fe5510c7e9e146523d8badd1984d486ddecc Sep 11 03:53:08.751978 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:08.751921 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26ea58d8_26d9_4191_87b4_03d1efda477a.slice/crio-565b087cb778c652c83948e27689ac5508dac87b01d81c472f956713d1128828 WatchSource:0}: Error finding container 565b087cb778c652c83948e27689ac5508dac87b01d81c472f956713d1128828: Status 404 returned error can't find the container with id 565b087cb778c652c83948e27689ac5508dac87b01d81c472f956713d1128828 Sep 11 03:53:08.769301 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:08.769276 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wvrb9" event={"ID":"26ea58d8-26d9-4191-87b4-03d1efda477a","Type":"ContainerStarted","Data":"565b087cb778c652c83948e27689ac5508dac87b01d81c472f956713d1128828"} Sep 11 03:53:08.770317 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:08.770296 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-c5drs" event={"ID":"aa66b45c-400f-46fa-b216-0e063413439b","Type":"ContainerStarted","Data":"05f600083d510413fefa60a72bda7fd15e85f02dd8eb31f83fc9c5eca094194a"} Sep 11 03:53:08.771347 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:08.771328 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqk2s" event={"ID":"6d855dff-6bf9-486f-aa54-479f9ba1fedc","Type":"ContainerStarted","Data":"9ef66c76ca75f9dfc28ad7240798254b4f57bdda4917fa1f7b8fb8e85fc4485a"} Sep 11 03:53:08.772388 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:08.772364 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7f284" event={"ID":"a48e2404-e37c-43d5-8902-b139cb2e07d9","Type":"ContainerStarted","Data":"accb255f0b704018342e2949f12e9e9b331cc4c73dfd92e2df531801ef070cbf"} Sep 11 03:53:08.773294 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:08.773272 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9r65j" event={"ID":"d6906bf8-7095-4f6a-8969-d6f28167548b","Type":"ContainerStarted","Data":"3ad5f7ab5ef19f84251f8e90cfad88fcb0cfff4e51ce863b52c34ca77729579f"} Sep 11 03:53:08.774699 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:08.774599 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gmdt4" event={"ID":"db235f27-7ffe-4c12-9d60-53fb8b810d97","Type":"ContainerStarted","Data":"cb14f9c143512d17d4f30d483703fe5510c7e9e146523d8badd1984d486ddecc"} Sep 11 03:53:08.775897 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:08.775818 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cvjtt" event={"ID":"328d250b-1183-4ce5-8f21-f263e78a8b95","Type":"ContainerStarted","Data":"925d641fc37569e5a324a68791b8ec03e54ac65b8e0f7cfc86340982129cdda0"} Sep 11 03:53:08.777001 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:08.776982 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-htjph" event={"ID":"44fdee40-4032-43ac-aabc-2393a05a4ef2","Type":"ContainerStarted","Data":"540ba26b53dd7cc525733eda1e2ad20b2eab3d8193d472278a107ba5efbbab72"} Sep 11 03:53:08.778281 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:08.778264 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r8rsb" event={"ID":"e35bc33f-65c8-4fc4-abef-6e652686ec22","Type":"ContainerStarted","Data":"67f4ed53d12ddc1501f1ef8ce2c09b0dd34e798e1be89c488e679ee1f4d28944"} Sep 11 03:53:08.984034 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:08.983958 2568 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 03:53:09.295890 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:09.295811 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe1486cf-6276-41a8-a685-025064c4abca-metrics-certs\") pod \"network-metrics-daemon-tk8q7\" (UID: \"fe1486cf-6276-41a8-a685-025064c4abca\") " pod="openshift-multus/network-metrics-daemon-tk8q7" Sep 11 03:53:09.296065 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:09.295959 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 03:53:09.296065 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:09.296026 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe1486cf-6276-41a8-a685-025064c4abca-metrics-certs podName:fe1486cf-6276-41a8-a685-025064c4abca nodeName:}" failed. No retries permitted until 2026-09-11 03:53:11.296007555 +0000 UTC m=+6.049638942 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe1486cf-6276-41a8-a685-025064c4abca-metrics-certs") pod "network-metrics-daemon-tk8q7" (UID: "fe1486cf-6276-41a8-a685-025064c4abca") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 03:53:09.396659 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:09.396262 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnmts\" (UniqueName: \"kubernetes.io/projected/fb9f553c-03c7-4489-b43f-b62d630e7f50-kube-api-access-lnmts\") pod \"network-check-target-vssnm\" (UID: \"fb9f553c-03c7-4489-b43f-b62d630e7f50\") " pod="openshift-network-diagnostics/network-check-target-vssnm" Sep 11 03:53:09.396659 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:09.396329 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9fc48d3a-bf43-44f7-b142-cd4937cea66b-original-pull-secret\") pod \"global-pull-secret-syncer-t5fb2\" (UID: \"9fc48d3a-bf43-44f7-b142-cd4937cea66b\") " pod="kube-system/global-pull-secret-syncer-t5fb2" Sep 11 03:53:09.396659 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:09.396451 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 03:53:09.396659 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:09.396511 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fc48d3a-bf43-44f7-b142-cd4937cea66b-original-pull-secret podName:9fc48d3a-bf43-44f7-b142-cd4937cea66b nodeName:}" failed. No retries permitted until 2026-09-11 03:53:11.39649494 +0000 UTC m=+6.150126320 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9fc48d3a-bf43-44f7-b142-cd4937cea66b-original-pull-secret") pod "global-pull-secret-syncer-t5fb2" (UID: "9fc48d3a-bf43-44f7-b142-cd4937cea66b") : object "kube-system"/"original-pull-secret" not registered Sep 11 03:53:09.396990 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:09.396918 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 03:53:09.396990 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:09.396937 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 03:53:09.396990 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:09.396949 2568 projected.go:196] Error preparing data for projected volume kube-api-access-lnmts for pod openshift-network-diagnostics/network-check-target-vssnm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 03:53:09.396990 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:09.396991 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb9f553c-03c7-4489-b43f-b62d630e7f50-kube-api-access-lnmts podName:fb9f553c-03c7-4489-b43f-b62d630e7f50 nodeName:}" failed. No retries permitted until 2026-09-11 03:53:11.396978058 +0000 UTC m=+6.150609445 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lnmts" (UniqueName: "kubernetes.io/projected/fb9f553c-03c7-4489-b43f-b62d630e7f50-kube-api-access-lnmts") pod "network-check-target-vssnm" (UID: "fb9f553c-03c7-4489-b43f-b62d630e7f50") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 03:53:09.706844 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:09.706799 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 03:48:07 +0000 UTC" deadline="2028-03-14 10:04:35.720109078 +0000 UTC" Sep 11 03:53:09.706844 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:09.706842 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13206h11m26.013271651s" Sep 11 03:53:09.765335 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:09.762206 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5fb2" Sep 11 03:53:09.765335 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:09.762331 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5fb2" podUID="9fc48d3a-bf43-44f7-b142-cd4937cea66b" Sep 11 03:53:09.765335 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:09.762415 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vssnm" Sep 11 03:53:09.765335 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:09.762482 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vssnm" podUID="fb9f553c-03c7-4489-b43f-b62d630e7f50" Sep 11 03:53:09.765335 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:09.762549 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk8q7" Sep 11 03:53:09.765335 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:09.762628 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk8q7" podUID="fe1486cf-6276-41a8-a685-025064c4abca" Sep 11 03:53:09.796668 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:09.796633 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-30.ec2.internal" event={"ID":"f90f3e1f70d21477b755e9266cecc76b","Type":"ContainerStarted","Data":"ef6a7246d26b2299d7258e91ecce2b11087666a67a693a073647120000033bcf"} Sep 11 03:53:09.813523 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:09.812943 2568 generic.go:356] "Generic (PLEG): container finished" podID="72105b6990dc29a2d8255d18f1b5a345" containerID="5262b0118b831fc0e50ae078082b65de0fb3bbe4b198041870280eb190741087" exitCode=0 Sep 11 03:53:09.813523 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:09.812981 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-30.ec2.internal" event={"ID":"72105b6990dc29a2d8255d18f1b5a345","Type":"ContainerDied","Data":"5262b0118b831fc0e50ae078082b65de0fb3bbe4b198041870280eb190741087"} Sep 11 03:53:09.825409 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:09.823751 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-30.ec2.internal" podStartSLOduration=2.823733785 podStartE2EDuration="2.823733785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:53:09.822306122 +0000 UTC m=+4.575937525" watchObservedRunningTime="2026-09-11 03:53:09.823733785 +0000 UTC m=+4.577365189" Sep 11 03:53:10.834618 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:10.834281 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-30.ec2.internal" event={"ID":"72105b6990dc29a2d8255d18f1b5a345","Type":"ContainerStarted","Data":"7a8698774e93d681dc9b16f58d05493f238e37fa7b80f73537640d3546d98b7d"} Sep 11 03:53:11.311399 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:11.311355 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe1486cf-6276-41a8-a685-025064c4abca-metrics-certs\") pod \"network-metrics-daemon-tk8q7\" (UID: \"fe1486cf-6276-41a8-a685-025064c4abca\") " pod="openshift-multus/network-metrics-daemon-tk8q7" Sep 11 03:53:11.311587 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:11.311567 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 03:53:11.311666 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:11.311638 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe1486cf-6276-41a8-a685-025064c4abca-metrics-certs podName:fe1486cf-6276-41a8-a685-025064c4abca nodeName:}" failed. No retries permitted until 2026-09-11 03:53:15.311620654 +0000 UTC m=+10.065252040 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe1486cf-6276-41a8-a685-025064c4abca-metrics-certs") pod "network-metrics-daemon-tk8q7" (UID: "fe1486cf-6276-41a8-a685-025064c4abca") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 03:53:11.412275 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:11.412230 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9fc48d3a-bf43-44f7-b142-cd4937cea66b-original-pull-secret\") pod \"global-pull-secret-syncer-t5fb2\" (UID: \"9fc48d3a-bf43-44f7-b142-cd4937cea66b\") " pod="kube-system/global-pull-secret-syncer-t5fb2" Sep 11 03:53:11.412456 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:11.412327 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnmts\" (UniqueName: \"kubernetes.io/projected/fb9f553c-03c7-4489-b43f-b62d630e7f50-kube-api-access-lnmts\") pod \"network-check-target-vssnm\" (UID: \"fb9f553c-03c7-4489-b43f-b62d630e7f50\") " pod="openshift-network-diagnostics/network-check-target-vssnm" Sep 11 03:53:11.412456 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:11.412435 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 03:53:11.412574 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:11.412516 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fc48d3a-bf43-44f7-b142-cd4937cea66b-original-pull-secret podName:9fc48d3a-bf43-44f7-b142-cd4937cea66b nodeName:}" failed. No retries permitted until 2026-09-11 03:53:15.412495856 +0000 UTC m=+10.166127245 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9fc48d3a-bf43-44f7-b142-cd4937cea66b-original-pull-secret") pod "global-pull-secret-syncer-t5fb2" (UID: "9fc48d3a-bf43-44f7-b142-cd4937cea66b") : object "kube-system"/"original-pull-secret" not registered Sep 11 03:53:11.412574 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:11.412447 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 03:53:11.412574 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:11.412551 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 03:53:11.412574 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:11.412567 2568 projected.go:196] Error preparing data for projected volume kube-api-access-lnmts for pod openshift-network-diagnostics/network-check-target-vssnm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 03:53:11.412766 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:11.412600 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb9f553c-03c7-4489-b43f-b62d630e7f50-kube-api-access-lnmts podName:fb9f553c-03c7-4489-b43f-b62d630e7f50 nodeName:}" failed. No retries permitted until 2026-09-11 03:53:15.41258988 +0000 UTC m=+10.166221264 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lnmts" (UniqueName: "kubernetes.io/projected/fb9f553c-03c7-4489-b43f-b62d630e7f50-kube-api-access-lnmts") pod "network-check-target-vssnm" (UID: "fb9f553c-03c7-4489-b43f-b62d630e7f50") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 03:53:11.766162 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:11.766131 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vssnm" Sep 11 03:53:11.766356 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:11.766254 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vssnm" podUID="fb9f553c-03c7-4489-b43f-b62d630e7f50" Sep 11 03:53:11.766633 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:11.766612 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5fb2" Sep 11 03:53:11.766749 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:11.766717 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5fb2" podUID="9fc48d3a-bf43-44f7-b142-cd4937cea66b" Sep 11 03:53:11.766749 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:11.766612 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk8q7" Sep 11 03:53:11.766924 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:11.766877 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk8q7" podUID="fe1486cf-6276-41a8-a685-025064c4abca" Sep 11 03:53:13.767489 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:13.767119 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vssnm" Sep 11 03:53:13.767489 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:13.767255 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vssnm" podUID="fb9f553c-03c7-4489-b43f-b62d630e7f50" Sep 11 03:53:13.767489 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:13.767263 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5fb2" Sep 11 03:53:13.767489 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:13.767353 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5fb2" podUID="9fc48d3a-bf43-44f7-b142-cd4937cea66b" Sep 11 03:53:13.767489 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:13.767265 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk8q7" Sep 11 03:53:13.767489 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:13.767450 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk8q7" podUID="fe1486cf-6276-41a8-a685-025064c4abca" Sep 11 03:53:15.346634 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:15.346312 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe1486cf-6276-41a8-a685-025064c4abca-metrics-certs\") pod \"network-metrics-daemon-tk8q7\" (UID: \"fe1486cf-6276-41a8-a685-025064c4abca\") " pod="openshift-multus/network-metrics-daemon-tk8q7" Sep 11 03:53:15.346634 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:15.346565 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 03:53:15.346634 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:15.346638 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe1486cf-6276-41a8-a685-025064c4abca-metrics-certs podName:fe1486cf-6276-41a8-a685-025064c4abca nodeName:}" failed. No retries permitted until 2026-09-11 03:53:23.346617649 +0000 UTC m=+18.100249037 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe1486cf-6276-41a8-a685-025064c4abca-metrics-certs") pod "network-metrics-daemon-tk8q7" (UID: "fe1486cf-6276-41a8-a685-025064c4abca") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 03:53:15.447807 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:15.447772 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnmts\" (UniqueName: \"kubernetes.io/projected/fb9f553c-03c7-4489-b43f-b62d630e7f50-kube-api-access-lnmts\") pod \"network-check-target-vssnm\" (UID: \"fb9f553c-03c7-4489-b43f-b62d630e7f50\") " pod="openshift-network-diagnostics/network-check-target-vssnm" Sep 11 03:53:15.447982 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:15.447836 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9fc48d3a-bf43-44f7-b142-cd4937cea66b-original-pull-secret\") pod \"global-pull-secret-syncer-t5fb2\" (UID: \"9fc48d3a-bf43-44f7-b142-cd4937cea66b\") " pod="kube-system/global-pull-secret-syncer-t5fb2" Sep 11 03:53:15.447982 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:15.447939 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 03:53:15.447982 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:15.447953 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 03:53:15.447982 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:15.447976 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 03:53:15.448203 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:15.447987 2568 projected.go:196] Error preparing data for projected volume kube-api-access-lnmts for pod openshift-network-diagnostics/network-check-target-vssnm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 03:53:15.448203 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:15.448000 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fc48d3a-bf43-44f7-b142-cd4937cea66b-original-pull-secret podName:9fc48d3a-bf43-44f7-b142-cd4937cea66b nodeName:}" failed. No retries permitted until 2026-09-11 03:53:23.447983857 +0000 UTC m=+18.201615243 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9fc48d3a-bf43-44f7-b142-cd4937cea66b-original-pull-secret") pod "global-pull-secret-syncer-t5fb2" (UID: "9fc48d3a-bf43-44f7-b142-cd4937cea66b") : object "kube-system"/"original-pull-secret" not registered Sep 11 03:53:15.448203 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:15.448037 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb9f553c-03c7-4489-b43f-b62d630e7f50-kube-api-access-lnmts podName:fb9f553c-03c7-4489-b43f-b62d630e7f50 nodeName:}" failed. No retries permitted until 2026-09-11 03:53:23.448021691 +0000 UTC m=+18.201653089 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lnmts" (UniqueName: "kubernetes.io/projected/fb9f553c-03c7-4489-b43f-b62d630e7f50-kube-api-access-lnmts") pod "network-check-target-vssnm" (UID: "fb9f553c-03c7-4489-b43f-b62d630e7f50") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 03:53:15.764175 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:15.763719 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vssnm" Sep 11 03:53:15.764175 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:15.763815 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk8q7" Sep 11 03:53:15.764175 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:15.763824 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vssnm" podUID="fb9f553c-03c7-4489-b43f-b62d630e7f50" Sep 11 03:53:15.764175 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:15.763845 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5fb2" Sep 11 03:53:15.764175 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:15.763958 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk8q7" podUID="fe1486cf-6276-41a8-a685-025064c4abca" Sep 11 03:53:15.764175 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:15.764062 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5fb2" podUID="9fc48d3a-bf43-44f7-b142-cd4937cea66b" Sep 11 03:53:17.763792 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:17.763142 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk8q7" Sep 11 03:53:17.763792 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:17.763283 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk8q7" podUID="fe1486cf-6276-41a8-a685-025064c4abca" Sep 11 03:53:17.763792 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:17.763656 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vssnm" Sep 11 03:53:17.763792 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:17.763740 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vssnm" podUID="fb9f553c-03c7-4489-b43f-b62d630e7f50" Sep 11 03:53:17.764416 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:17.763887 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5fb2" Sep 11 03:53:17.764416 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:17.764000 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5fb2" podUID="9fc48d3a-bf43-44f7-b142-cd4937cea66b" Sep 11 03:53:19.762895 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:19.762860 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vssnm" Sep 11 03:53:19.763396 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:19.762870 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5fb2" Sep 11 03:53:19.763396 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:19.762982 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk8q7" Sep 11 03:53:19.763396 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:19.762983 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vssnm" podUID="fb9f553c-03c7-4489-b43f-b62d630e7f50" Sep 11 03:53:19.763396 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:19.763072 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5fb2" podUID="9fc48d3a-bf43-44f7-b142-cd4937cea66b" Sep 11 03:53:19.763396 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:19.763159 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk8q7" podUID="fe1486cf-6276-41a8-a685-025064c4abca" Sep 11 03:53:21.762038 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:21.762007 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vssnm" Sep 11 03:53:21.762038 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:21.762027 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk8q7" Sep 11 03:53:21.762578 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:21.762135 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vssnm" podUID="fb9f553c-03c7-4489-b43f-b62d630e7f50" Sep 11 03:53:21.762578 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:21.762205 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk8q7" podUID="fe1486cf-6276-41a8-a685-025064c4abca" Sep 11 03:53:21.762578 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:21.762218 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5fb2" Sep 11 03:53:21.762578 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:21.762313 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5fb2" podUID="9fc48d3a-bf43-44f7-b142-cd4937cea66b" Sep 11 03:53:23.411581 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:23.411543 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe1486cf-6276-41a8-a685-025064c4abca-metrics-certs\") pod \"network-metrics-daemon-tk8q7\" (UID: \"fe1486cf-6276-41a8-a685-025064c4abca\") " pod="openshift-multus/network-metrics-daemon-tk8q7" Sep 11 03:53:23.412092 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:23.411678 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 03:53:23.412092 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:23.411755 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe1486cf-6276-41a8-a685-025064c4abca-metrics-certs podName:fe1486cf-6276-41a8-a685-025064c4abca nodeName:}" failed. No retries permitted until 2026-09-11 03:53:39.411734446 +0000 UTC m=+34.165365834 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe1486cf-6276-41a8-a685-025064c4abca-metrics-certs") pod "network-metrics-daemon-tk8q7" (UID: "fe1486cf-6276-41a8-a685-025064c4abca") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 03:53:23.512882 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:23.512847 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9fc48d3a-bf43-44f7-b142-cd4937cea66b-original-pull-secret\") pod \"global-pull-secret-syncer-t5fb2\" (UID: \"9fc48d3a-bf43-44f7-b142-cd4937cea66b\") " pod="kube-system/global-pull-secret-syncer-t5fb2" Sep 11 03:53:23.513086 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:23.512925 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnmts\" (UniqueName: \"kubernetes.io/projected/fb9f553c-03c7-4489-b43f-b62d630e7f50-kube-api-access-lnmts\") pod \"network-check-target-vssnm\" (UID: \"fb9f553c-03c7-4489-b43f-b62d630e7f50\") " pod="openshift-network-diagnostics/network-check-target-vssnm" Sep 11 03:53:23.513086 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:23.513020 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 03:53:23.513086 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:23.513067 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 03:53:23.513086 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:23.513084 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 03:53:23.513263 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:23.513097 2568 projected.go:196] Error preparing data for projected volume kube-api-access-lnmts for pod openshift-network-diagnostics/network-check-target-vssnm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 03:53:23.513263 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:23.513115 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fc48d3a-bf43-44f7-b142-cd4937cea66b-original-pull-secret podName:9fc48d3a-bf43-44f7-b142-cd4937cea66b nodeName:}" failed. No retries permitted until 2026-09-11 03:53:39.513093616 +0000 UTC m=+34.266725002 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9fc48d3a-bf43-44f7-b142-cd4937cea66b-original-pull-secret") pod "global-pull-secret-syncer-t5fb2" (UID: "9fc48d3a-bf43-44f7-b142-cd4937cea66b") : object "kube-system"/"original-pull-secret" not registered Sep 11 03:53:23.513263 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:23.513144 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb9f553c-03c7-4489-b43f-b62d630e7f50-kube-api-access-lnmts podName:fb9f553c-03c7-4489-b43f-b62d630e7f50 nodeName:}" failed. No retries permitted until 2026-09-11 03:53:39.513128708 +0000 UTC m=+34.266760107 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lnmts" (UniqueName: "kubernetes.io/projected/fb9f553c-03c7-4489-b43f-b62d630e7f50-kube-api-access-lnmts") pod "network-check-target-vssnm" (UID: "fb9f553c-03c7-4489-b43f-b62d630e7f50") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 03:53:23.762732 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:23.762648 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk8q7" Sep 11 03:53:23.762897 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:23.762648 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5fb2" Sep 11 03:53:23.762897 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:23.762809 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk8q7" podUID="fe1486cf-6276-41a8-a685-025064c4abca" Sep 11 03:53:23.762897 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:23.762661 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vssnm" Sep 11 03:53:23.762897 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:23.762876 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5fb2" podUID="9fc48d3a-bf43-44f7-b142-cd4937cea66b" Sep 11 03:53:23.763122 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:23.762946 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vssnm" podUID="fb9f553c-03c7-4489-b43f-b62d630e7f50" Sep 11 03:53:25.765218 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:25.765188 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vssnm" Sep 11 03:53:25.765681 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:25.765295 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vssnm" podUID="fb9f553c-03c7-4489-b43f-b62d630e7f50" Sep 11 03:53:25.765681 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:25.765306 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk8q7" Sep 11 03:53:25.765681 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:25.765322 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5fb2" Sep 11 03:53:25.765681 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:25.765437 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk8q7" podUID="fe1486cf-6276-41a8-a685-025064c4abca" Sep 11 03:53:25.765681 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:25.765535 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5fb2" podUID="9fc48d3a-bf43-44f7-b142-cd4937cea66b" Sep 11 03:53:26.867240 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:26.867022 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gmdt4" event={"ID":"db235f27-7ffe-4c12-9d60-53fb8b810d97","Type":"ContainerStarted","Data":"ef799c2137b3353ffe39c8736e0965cbe9a21a34def90d8f93b5294cadd9e8bd"} Sep 11 03:53:26.869311 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:26.869283 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-htjph" event={"ID":"44fdee40-4032-43ac-aabc-2393a05a4ef2","Type":"ContainerStarted","Data":"1b9b7c36483a395e31c0dfc65f91eed7e019f68bccf809d54986ac6d4a9ca98d"} Sep 11 03:53:26.869409 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:26.869370 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-htjph" event={"ID":"44fdee40-4032-43ac-aabc-2393a05a4ef2","Type":"ContainerStarted","Data":"b77845945d349dd0ad69b416eb3086b7598fe21de596728f7c8edb9a15422a0c"} Sep 11 03:53:26.869450 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:26.869422 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-htjph" event={"ID":"44fdee40-4032-43ac-aabc-2393a05a4ef2","Type":"ContainerStarted","Data":"fe1365f6ec13ebb3acdf8b22b19dac274ff27c648c20c85650847f4cb5773c08"} Sep 11 03:53:26.869495 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:26.869471 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-htjph" event={"ID":"44fdee40-4032-43ac-aabc-2393a05a4ef2","Type":"ContainerStarted","Data":"b66292bade7debefdb97bb00b4948594db65aefce1a437adedebeef60edad8f7"} Sep 11 03:53:26.870765 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:26.870738 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r8rsb" event={"ID":"e35bc33f-65c8-4fc4-abef-6e652686ec22","Type":"ContainerStarted","Data":"98c16252ced6558543501fcdb52df1bc0819444bbe9af2276e510d151dfe4108"} Sep 11 03:53:26.872200 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:26.872181 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wvrb9" event={"ID":"26ea58d8-26d9-4191-87b4-03d1efda477a","Type":"ContainerStarted","Data":"1e9527508dfe80fbb9e20c239f8ecc5f6d5ce8cb4354d6e5b44e45024d4599b1"} Sep 11 03:53:26.873433 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:26.873414 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-c5drs" event={"ID":"aa66b45c-400f-46fa-b216-0e063413439b","Type":"ContainerStarted","Data":"679470059d870c3278af9fa18735f778111b5c6e5120ba4f90776e93e943d85d"} Sep 11 03:53:26.874743 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:26.874722 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqk2s" event={"ID":"6d855dff-6bf9-486f-aa54-479f9ba1fedc","Type":"ContainerStarted","Data":"006033fb1acbe0ff182c7762dbf879b84aa49a344a5ddece96b0a0a36c627004"} Sep 11 03:53:26.875881 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:26.875859 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7f284" event={"ID":"a48e2404-e37c-43d5-8902-b139cb2e07d9","Type":"ContainerStarted","Data":"1301c71bb4ad63a842c760a5996b2daa196e3128b6610e2dc12c6730fa5e1a75"} Sep 11 03:53:26.877071 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:26.877033 2568 generic.go:356] "Generic (PLEG): container finished" podID="d6906bf8-7095-4f6a-8969-d6f28167548b" containerID="9eb8d8340a55e59313dc2601570b40600dd213d7eaf2ef3d79b515e4f3438b3d" exitCode=0 Sep 11 03:53:26.877141 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:26.877078 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9r65j" event={"ID":"d6906bf8-7095-4f6a-8969-d6f28167548b","Type":"ContainerDied","Data":"9eb8d8340a55e59313dc2601570b40600dd213d7eaf2ef3d79b515e4f3438b3d"} Sep 11 03:53:26.881140 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:26.881102 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-gmdt4" podStartSLOduration=4.315094116 podStartE2EDuration="21.881089382s" totalImagesPullingTime="17.565995266s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:53:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:53:26.880878666 +0000 UTC m=+21.634510069" watchObservedRunningTime="2026-09-11 03:53:26.881089382 +0000 UTC m=+21.634720784" Sep 11 03:53:26.881202 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:26.881178 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-30.ec2.internal" podStartSLOduration=19.881172379 podStartE2EDuration="19.881172379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:53:10.849667062 +0000 UTC m=+5.603298465" watchObservedRunningTime="2026-09-11 03:53:26.881172379 +0000 UTC m=+21.634803781" Sep 11 03:53:26.899436 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:26.899399 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-r8rsb" podStartSLOduration=4.247324337 podStartE2EDuration="21.899384513s" totalImagesPullingTime="17.652060176s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:53:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:53:26.899106747 +0000 UTC m=+21.652738146" watchObservedRunningTime="2026-09-11 03:53:26.899384513 +0000 UTC m=+21.653015979" Sep 11 03:53:26.915240 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:26.915203 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-c5drs" podStartSLOduration=8.357078294 podStartE2EDuration="21.915193054s" totalImagesPullingTime="13.55811476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:53:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:53:26.915128996 +0000 UTC m=+21.668760397" watchObservedRunningTime="2026-09-11 03:53:26.915193054 +0000 UTC m=+21.668824457" Sep 11 03:53:26.934896 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:26.934856 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-7f284" podStartSLOduration=3.34411293 podStartE2EDuration="20.934842181s" totalImagesPullingTime="17.590729251s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:53:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:53:26.93433073 +0000 UTC m=+21.687962133" watchObservedRunningTime="2026-09-11 03:53:26.934842181 +0000 UTC m=+21.688473583" Sep 11 03:53:26.974293 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:26.974124 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wvrb9" podStartSLOduration=4.406366803 podStartE2EDuration="21.974111912s" totalImagesPullingTime="17.567745109s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:53:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:53:26.974094807 +0000 UTC m=+21.727726206" watchObservedRunningTime="2026-09-11 03:53:26.974111912 +0000 UTC m=+21.727743314" Sep 11 03:53:27.762402 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:27.762375 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5fb2" Sep 11 03:53:27.762554 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:27.762374 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk8q7" Sep 11 03:53:27.762554 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:27.762493 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5fb2" podUID="9fc48d3a-bf43-44f7-b142-cd4937cea66b" Sep 11 03:53:27.762669 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:27.762601 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk8q7" podUID="fe1486cf-6276-41a8-a685-025064c4abca" Sep 11 03:53:27.762669 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:27.762375 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vssnm" Sep 11 03:53:27.762771 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:27.762691 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vssnm" podUID="fb9f553c-03c7-4489-b43f-b62d630e7f50" Sep 11 03:53:27.881174 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:27.881140 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-cvjtt" event={"ID":"328d250b-1183-4ce5-8f21-f263e78a8b95","Type":"ContainerStarted","Data":"79a45f712f17d230030f441dc5579e6fed072010a0abc2ee7a32b319d0f56341"} Sep 11 03:53:27.884535 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:27.884461 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-htjph" event={"ID":"44fdee40-4032-43ac-aabc-2393a05a4ef2","Type":"ContainerStarted","Data":"4c3a78d6892ee689d02279141ba36fa75b173379025a624750794e4200f0acb2"} Sep 11 03:53:27.884654 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:27.884547 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-htjph" event={"ID":"44fdee40-4032-43ac-aabc-2393a05a4ef2","Type":"ContainerStarted","Data":"1cff176c0686eec2a4b154e2e27f5f7d264c6e731fa6c512c1691b55de5a5938"} Sep 11 03:53:27.894628 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:27.894582 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-cvjtt" podStartSLOduration=8.312181295 podStartE2EDuration="21.894565475s" totalImagesPullingTime="13.58238418s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:53:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:53:27.894111046 +0000 UTC m=+22.647742452" watchObservedRunningTime="2026-09-11 03:53:27.894565475 +0000 UTC m=+22.648196878" Sep 11 03:53:27.940170 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:27.940147 2568 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 11 03:53:28.735289 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:28.735190 2568 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T03:53:27.940166889Z","UUID":"a5971aaf-f0e6-4cee-a0de-efefa9dd360d","Handler":null,"Name":"","Endpoint":""} Sep 11 03:53:28.736810 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:28.736788 2568 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 11 03:53:28.736810 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:28.736824 2568 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 11 03:53:28.890327 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:28.890294 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqk2s" event={"ID":"6d855dff-6bf9-486f-aa54-479f9ba1fedc","Type":"ContainerStarted","Data":"2840212ad0269ba18ecad9136a7f0ccb062d0f4bccf117fbcada9be10654268a"} Sep 11 03:53:29.590430 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:29.590339 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-c5drs" Sep 11 03:53:29.591347 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:29.591297 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-c5drs" Sep 11 03:53:29.762755 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:29.762722 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5fb2" Sep 11 03:53:29.762914 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:29.762722 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk8q7" Sep 11 03:53:29.762914 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:29.762853 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5fb2" podUID="9fc48d3a-bf43-44f7-b142-cd4937cea66b" Sep 11 03:53:29.763033 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:29.762934 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk8q7" podUID="fe1486cf-6276-41a8-a685-025064c4abca" Sep 11 03:53:29.763033 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:29.762722 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vssnm" Sep 11 03:53:29.763033 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:29.762998 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vssnm" podUID="fb9f553c-03c7-4489-b43f-b62d630e7f50" Sep 11 03:53:29.894499 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:29.894414 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqk2s" event={"ID":"6d855dff-6bf9-486f-aa54-479f9ba1fedc","Type":"ContainerStarted","Data":"38eb924409c6645c2757d143014fc5e77988621afe9e0b6fd62402ce8b7de6f8"} Sep 11 03:53:29.898564 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:29.898527 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-htjph" event={"ID":"44fdee40-4032-43ac-aabc-2393a05a4ef2","Type":"ContainerStarted","Data":"c9ad836367b6ae48a8626c1504f96287acf4289d525bc32bc103699ff7adfd4f"} Sep 11 03:53:29.898777 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:29.898758 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-c5drs" Sep 11 03:53:29.899366 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:29.899342 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-c5drs" Sep 11 03:53:29.911236 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:29.911198 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hqk2s" podStartSLOduration=4.053662281 podStartE2EDuration="23.911182496s" totalImagesPullingTime="19.857520215s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:53:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:53:29.910796839 +0000 UTC m=+24.664428243" watchObservedRunningTime="2026-09-11 03:53:29.911182496 +0000 UTC m=+24.664813898" Sep 11 03:53:31.766056 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:31.765846 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5fb2" Sep 11 03:53:31.766668 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:31.765904 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vssnm" Sep 11 03:53:31.766668 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:31.766093 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5fb2" podUID="9fc48d3a-bf43-44f7-b142-cd4937cea66b" Sep 11 03:53:31.766668 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:31.765910 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk8q7" Sep 11 03:53:31.766668 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:31.766159 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vssnm" podUID="fb9f553c-03c7-4489-b43f-b62d630e7f50" Sep 11 03:53:31.766668 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:31.766222 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk8q7" podUID="fe1486cf-6276-41a8-a685-025064c4abca" Sep 11 03:53:31.904344 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:31.904311 2568 generic.go:356] "Generic (PLEG): container finished" podID="d6906bf8-7095-4f6a-8969-d6f28167548b" containerID="c87268a721311934cd58d669a92a85dd35714c89dfdc69650e2d5dd277824716" exitCode=0 Sep 11 03:53:31.904498 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:31.904401 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9r65j" event={"ID":"d6906bf8-7095-4f6a-8969-d6f28167548b","Type":"ContainerDied","Data":"c87268a721311934cd58d669a92a85dd35714c89dfdc69650e2d5dd277824716"} Sep 11 03:53:31.908012 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:31.907983 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-htjph" event={"ID":"44fdee40-4032-43ac-aabc-2393a05a4ef2","Type":"ContainerStarted","Data":"396e89efd2a7a7668d92ea7ea4476fa67a33c1972189aeacbf955fa04513291f"} Sep 11 03:53:31.908283 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:31.908251 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:31.908400 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:31.908334 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:31.923526 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:31.923503 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:31.948921 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:31.948873 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-htjph" podStartSLOduration=9.309874458 podStartE2EDuration="26.94886032s" totalImagesPullingTime="17.638985862s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:53:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:53:31.948351946 +0000 UTC m=+26.701983358" watchObservedRunningTime="2026-09-11 03:53:31.94886032 +0000 UTC m=+26.702491722" Sep 11 03:53:32.912302 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:32.912118 2568 generic.go:356] "Generic (PLEG): container finished" podID="d6906bf8-7095-4f6a-8969-d6f28167548b" containerID="02bd2e0d7f0c314658cccf70acb2914ac5ce0067208dc564df304c69d0eb077f" exitCode=0 Sep 11 03:53:32.912667 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:32.912204 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9r65j" event={"ID":"d6906bf8-7095-4f6a-8969-d6f28167548b","Type":"ContainerDied","Data":"02bd2e0d7f0c314658cccf70acb2914ac5ce0067208dc564df304c69d0eb077f"} Sep 11 03:53:32.912813 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:32.912787 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:32.926964 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:32.926944 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:53:33.480561 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:33.480531 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-t5fb2"] Sep 11 03:53:33.480703 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:33.480637 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5fb2" Sep 11 03:53:33.480747 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:33.480715 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5fb2" podUID="9fc48d3a-bf43-44f7-b142-cd4937cea66b" Sep 11 03:53:33.483647 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:33.483619 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vssnm"] Sep 11 03:53:33.483747 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:33.483727 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vssnm" Sep 11 03:53:33.483839 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:33.483812 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vssnm" podUID="fb9f553c-03c7-4489-b43f-b62d630e7f50" Sep 11 03:53:33.484325 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:33.484301 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tk8q7"] Sep 11 03:53:33.484419 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:33.484404 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk8q7" Sep 11 03:53:33.484516 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:33.484493 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk8q7" podUID="fe1486cf-6276-41a8-a685-025064c4abca" Sep 11 03:53:33.917432 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:33.917401 2568 generic.go:356] "Generic (PLEG): container finished" podID="d6906bf8-7095-4f6a-8969-d6f28167548b" containerID="e26fe9ec1462384f8fa17b02151c9d2de6a0e4b7dc32f5bfd70923b4c2476d72" exitCode=0 Sep 11 03:53:33.917806 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:33.917478 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9r65j" event={"ID":"d6906bf8-7095-4f6a-8969-d6f28167548b","Type":"ContainerDied","Data":"e26fe9ec1462384f8fa17b02151c9d2de6a0e4b7dc32f5bfd70923b4c2476d72"} Sep 11 03:53:34.762277 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:34.762243 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5fb2" Sep 11 03:53:34.762446 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:34.762373 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5fb2" podUID="9fc48d3a-bf43-44f7-b142-cd4937cea66b" Sep 11 03:53:35.762527 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:35.762484 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vssnm" Sep 11 03:53:35.762945 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:35.762571 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vssnm" podUID="fb9f553c-03c7-4489-b43f-b62d630e7f50" Sep 11 03:53:35.762945 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:35.762574 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk8q7" Sep 11 03:53:35.762945 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:35.762649 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk8q7" podUID="fe1486cf-6276-41a8-a685-025064c4abca" Sep 11 03:53:36.761873 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:36.761840 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5fb2" Sep 11 03:53:36.762038 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:36.761963 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5fb2" podUID="9fc48d3a-bf43-44f7-b142-cd4937cea66b" Sep 11 03:53:37.762577 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:37.762539 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk8q7" Sep 11 03:53:37.763102 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:37.762678 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk8q7" podUID="fe1486cf-6276-41a8-a685-025064c4abca" Sep 11 03:53:37.763102 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:37.762539 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vssnm" Sep 11 03:53:37.763102 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:37.763094 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vssnm" podUID="fb9f553c-03c7-4489-b43f-b62d630e7f50" Sep 11 03:53:38.762832 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:38.762802 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5fb2" Sep 11 03:53:38.763231 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:38.762916 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t5fb2" podUID="9fc48d3a-bf43-44f7-b142-cd4937cea66b" Sep 11 03:53:39.431600 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:39.431516 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe1486cf-6276-41a8-a685-025064c4abca-metrics-certs\") pod \"network-metrics-daemon-tk8q7\" (UID: \"fe1486cf-6276-41a8-a685-025064c4abca\") " pod="openshift-multus/network-metrics-daemon-tk8q7" Sep 11 03:53:39.431861 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:39.431619 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 03:53:39.431861 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:39.431669 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe1486cf-6276-41a8-a685-025064c4abca-metrics-certs podName:fe1486cf-6276-41a8-a685-025064c4abca nodeName:}" failed. No retries permitted until 2026-09-11 03:54:11.43165491 +0000 UTC m=+66.185286290 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe1486cf-6276-41a8-a685-025064c4abca-metrics-certs") pod "network-metrics-daemon-tk8q7" (UID: "fe1486cf-6276-41a8-a685-025064c4abca") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 03:53:39.532338 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:39.532315 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnmts\" (UniqueName: \"kubernetes.io/projected/fb9f553c-03c7-4489-b43f-b62d630e7f50-kube-api-access-lnmts\") pod \"network-check-target-vssnm\" (UID: \"fb9f553c-03c7-4489-b43f-b62d630e7f50\") " pod="openshift-network-diagnostics/network-check-target-vssnm" Sep 11 03:53:39.532442 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:39.532363 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9fc48d3a-bf43-44f7-b142-cd4937cea66b-original-pull-secret\") pod \"global-pull-secret-syncer-t5fb2\" (UID: \"9fc48d3a-bf43-44f7-b142-cd4937cea66b\") " pod="kube-system/global-pull-secret-syncer-t5fb2" Sep 11 03:53:39.532477 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:39.532455 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 03:53:39.532509 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:39.532472 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 03:53:39.532509 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:39.532493 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 03:53:39.532509 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:39.532504 2568 projected.go:196] Error preparing data for projected volume kube-api-access-lnmts for pod openshift-network-diagnostics/network-check-target-vssnm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 03:53:39.532598 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:39.532515 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fc48d3a-bf43-44f7-b142-cd4937cea66b-original-pull-secret podName:9fc48d3a-bf43-44f7-b142-cd4937cea66b nodeName:}" failed. No retries permitted until 2026-09-11 03:54:11.532497699 +0000 UTC m=+66.286129083 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9fc48d3a-bf43-44f7-b142-cd4937cea66b-original-pull-secret") pod "global-pull-secret-syncer-t5fb2" (UID: "9fc48d3a-bf43-44f7-b142-cd4937cea66b") : object "kube-system"/"original-pull-secret" not registered Sep 11 03:53:39.532598 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:39.532538 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb9f553c-03c7-4489-b43f-b62d630e7f50-kube-api-access-lnmts podName:fb9f553c-03c7-4489-b43f-b62d630e7f50 nodeName:}" failed. No retries permitted until 2026-09-11 03:54:11.532529122 +0000 UTC m=+66.286160506 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-lnmts" (UniqueName: "kubernetes.io/projected/fb9f553c-03c7-4489-b43f-b62d630e7f50-kube-api-access-lnmts") pod "network-check-target-vssnm" (UID: "fb9f553c-03c7-4489-b43f-b62d630e7f50") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 03:53:39.762174 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:39.761948 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk8q7" Sep 11 03:53:39.762396 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:39.761959 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vssnm" Sep 11 03:53:39.762396 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:39.762235 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk8q7" podUID="fe1486cf-6276-41a8-a685-025064c4abca" Sep 11 03:53:39.762396 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:39.762272 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vssnm" podUID="fb9f553c-03c7-4489-b43f-b62d630e7f50" Sep 11 03:53:39.932675 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:39.932644 2568 generic.go:356] "Generic (PLEG): container finished" podID="d6906bf8-7095-4f6a-8969-d6f28167548b" containerID="75d0008063de8830a2cc061b8870e7f98e19ef063319d0bc63cd1b7a41081eb0" exitCode=0 Sep 11 03:53:39.932993 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:39.932713 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9r65j" event={"ID":"d6906bf8-7095-4f6a-8969-d6f28167548b","Type":"ContainerDied","Data":"75d0008063de8830a2cc061b8870e7f98e19ef063319d0bc63cd1b7a41081eb0"} Sep 11 03:53:40.026852 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.026825 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-30.ec2.internal" event="NodeReady" Sep 11 03:53:40.026974 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.026956 2568 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 11 03:53:40.061312 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.061286 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6c8dd7548c-jmns9"] Sep 11 03:53:40.078731 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.078705 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ssrwd"] Sep 11 03:53:40.078889 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.078837 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c8dd7548c-jmns9" Sep 11 03:53:40.081000 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.080981 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vm2xm\"" Sep 11 03:53:40.081247 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.081225 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 11 03:53:40.081685 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.081668 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 11 03:53:40.081776 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.081711 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 11 03:53:40.086919 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.086896 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 11 03:53:40.100289 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.100254 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-nq6qk"] Sep 11 03:53:40.100435 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.100409 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ssrwd" Sep 11 03:53:40.102887 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.102864 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 11 03:53:40.103031 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.102898 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 11 03:53:40.103127 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.103085 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h5zgd\"" Sep 11 03:53:40.124573 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.124542 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c8dd7548c-jmns9"] Sep 11 03:53:40.124573 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.124577 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nq6qk"] Sep 11 03:53:40.124762 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.124591 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ssrwd"] Sep 11 03:53:40.124762 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.124676 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nq6qk" Sep 11 03:53:40.127143 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.127119 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 11 03:53:40.127264 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.127119 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-n2mzh\"" Sep 11 03:53:40.127264 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.127196 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 11 03:53:40.127264 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.127121 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 11 03:53:40.222733 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.222700 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:53:40.237266 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.237243 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/52394690-486c-4ae0-b36b-603b438a944a-image-registry-private-configuration\") pod \"image-registry-6c8dd7548c-jmns9\" (UID: \"52394690-486c-4ae0-b36b-603b438a944a\") " pod="openshift-image-registry/image-registry-6c8dd7548c-jmns9" Sep 11 03:53:40.237353 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.237272 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/52394690-486c-4ae0-b36b-603b438a944a-registry-tls\") pod \"image-registry-6c8dd7548c-jmns9\" (UID: \"52394690-486c-4ae0-b36b-603b438a944a\") " pod="openshift-image-registry/image-registry-6c8dd7548c-jmns9" Sep 11 03:53:40.237353 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.237293 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/11fca7d2-d884-4012-9431-b3c085b8339c-metrics-tls\") pod \"dns-default-ssrwd\" (UID: \"11fca7d2-d884-4012-9431-b3c085b8339c\") " pod="openshift-dns/dns-default-ssrwd" Sep 11 03:53:40.237420 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.237360 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/52394690-486c-4ae0-b36b-603b438a944a-registry-certificates\") pod \"image-registry-6c8dd7548c-jmns9\" (UID: \"52394690-486c-4ae0-b36b-603b438a944a\") " pod="openshift-image-registry/image-registry-6c8dd7548c-jmns9" Sep 11 03:53:40.237420 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.237392 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wggx7\" (UniqueName: \"kubernetes.io/projected/52394690-486c-4ae0-b36b-603b438a944a-kube-api-access-wggx7\") pod \"image-registry-6c8dd7548c-jmns9\" (UID: \"52394690-486c-4ae0-b36b-603b438a944a\") " pod="openshift-image-registry/image-registry-6c8dd7548c-jmns9" Sep 11 03:53:40.237479 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.237421 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/52394690-486c-4ae0-b36b-603b438a944a-bound-sa-token\") pod \"image-registry-6c8dd7548c-jmns9\" (UID: \"52394690-486c-4ae0-b36b-603b438a944a\") " pod="openshift-image-registry/image-registry-6c8dd7548c-jmns9" Sep 11 03:53:40.237479 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.237440 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/52394690-486c-4ae0-b36b-603b438a944a-trusted-ca\") pod \"image-registry-6c8dd7548c-jmns9\" (UID: \"52394690-486c-4ae0-b36b-603b438a944a\") " pod="openshift-image-registry/image-registry-6c8dd7548c-jmns9" Sep 11 03:53:40.237479 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.237462 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/badea905-61e6-47e1-8f68-505d04824821-cert\") pod \"ingress-canary-nq6qk\" (UID: \"badea905-61e6-47e1-8f68-505d04824821\") " pod="openshift-ingress-canary/ingress-canary-nq6qk" Sep 11 03:53:40.237568 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.237494 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11fca7d2-d884-4012-9431-b3c085b8339c-tmp-dir\") pod \"dns-default-ssrwd\" (UID: \"11fca7d2-d884-4012-9431-b3c085b8339c\") " pod="openshift-dns/dns-default-ssrwd" Sep 11 03:53:40.237568 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.237536 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn7rf\" (UniqueName: \"kubernetes.io/projected/11fca7d2-d884-4012-9431-b3c085b8339c-kube-api-access-cn7rf\") pod \"dns-default-ssrwd\" (UID: \"11fca7d2-d884-4012-9431-b3c085b8339c\") " pod="openshift-dns/dns-default-ssrwd" Sep 11 03:53:40.237629 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.237578 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/52394690-486c-4ae0-b36b-603b438a944a-ca-trust-extracted\") pod \"image-registry-6c8dd7548c-jmns9\" (UID: \"52394690-486c-4ae0-b36b-603b438a944a\") " pod="openshift-image-registry/image-registry-6c8dd7548c-jmns9" Sep 11 03:53:40.237629 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.237595 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlh7z\" (UniqueName: \"kubernetes.io/projected/badea905-61e6-47e1-8f68-505d04824821-kube-api-access-mlh7z\") pod \"ingress-canary-nq6qk\" (UID: \"badea905-61e6-47e1-8f68-505d04824821\") " pod="openshift-ingress-canary/ingress-canary-nq6qk" Sep 11 03:53:40.237629 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.237613 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/52394690-486c-4ae0-b36b-603b438a944a-installation-pull-secrets\") pod \"image-registry-6c8dd7548c-jmns9\" (UID: \"52394690-486c-4ae0-b36b-603b438a944a\") " pod="openshift-image-registry/image-registry-6c8dd7548c-jmns9" Sep 11 03:53:40.237716 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.237628 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/11fca7d2-d884-4012-9431-b3c085b8339c-config-volume\") pod \"dns-default-ssrwd\" (UID: \"11fca7d2-d884-4012-9431-b3c085b8339c\") " pod="openshift-dns/dns-default-ssrwd" Sep 11 03:53:40.312622 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.312558 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:53:40.338602 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.338576 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/52394690-486c-4ae0-b36b-603b438a944a-image-registry-private-configuration\") pod \"image-registry-6c8dd7548c-jmns9\" (UID: \"52394690-486c-4ae0-b36b-603b438a944a\") " pod="openshift-image-registry/image-registry-6c8dd7548c-jmns9" Sep 11 03:53:40.338684 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.338611 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/52394690-486c-4ae0-b36b-603b438a944a-registry-tls\") pod \"image-registry-6c8dd7548c-jmns9\" (UID: \"52394690-486c-4ae0-b36b-603b438a944a\") " pod="openshift-image-registry/image-registry-6c8dd7548c-jmns9" Sep 11 03:53:40.338684 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.338628 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/11fca7d2-d884-4012-9431-b3c085b8339c-metrics-tls\") pod \"dns-default-ssrwd\" (UID: \"11fca7d2-d884-4012-9431-b3c085b8339c\") " pod="openshift-dns/dns-default-ssrwd" Sep 11 03:53:40.338749 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:40.338719 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 03:53:40.338749 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:40.338724 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 03:53:40.338749 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:40.338739 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c8dd7548c-jmns9: secret "image-registry-tls" not found Sep 11 03:53:40.338838 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:40.338760 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11fca7d2-d884-4012-9431-b3c085b8339c-metrics-tls podName:11fca7d2-d884-4012-9431-b3c085b8339c nodeName:}" failed. No retries permitted until 2026-09-11 03:53:40.838747907 +0000 UTC m=+35.592379300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/11fca7d2-d884-4012-9431-b3c085b8339c-metrics-tls") pod "dns-default-ssrwd" (UID: "11fca7d2-d884-4012-9431-b3c085b8339c") : secret "dns-default-metrics-tls" not found Sep 11 03:53:40.338838 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:40.338801 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52394690-486c-4ae0-b36b-603b438a944a-registry-tls podName:52394690-486c-4ae0-b36b-603b438a944a nodeName:}" failed. No retries permitted until 2026-09-11 03:53:40.83878381 +0000 UTC m=+35.592415189 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/52394690-486c-4ae0-b36b-603b438a944a-registry-tls") pod "image-registry-6c8dd7548c-jmns9" (UID: "52394690-486c-4ae0-b36b-603b438a944a") : secret "image-registry-tls" not found Sep 11 03:53:40.338911 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.338833 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/52394690-486c-4ae0-b36b-603b438a944a-registry-certificates\") pod \"image-registry-6c8dd7548c-jmns9\" (UID: \"52394690-486c-4ae0-b36b-603b438a944a\") " pod="openshift-image-registry/image-registry-6c8dd7548c-jmns9" Sep 11 03:53:40.338911 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.338859 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wggx7\" (UniqueName: \"kubernetes.io/projected/52394690-486c-4ae0-b36b-603b438a944a-kube-api-access-wggx7\") pod \"image-registry-6c8dd7548c-jmns9\" (UID: \"52394690-486c-4ae0-b36b-603b438a944a\") " pod="openshift-image-registry/image-registry-6c8dd7548c-jmns9" Sep 11 03:53:40.338911 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.338890 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/52394690-486c-4ae0-b36b-603b438a944a-bound-sa-token\") pod \"image-registry-6c8dd7548c-jmns9\" (UID: \"52394690-486c-4ae0-b36b-603b438a944a\") " pod="openshift-image-registry/image-registry-6c8dd7548c-jmns9" Sep 11 03:53:40.339005 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.338918 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/52394690-486c-4ae0-b36b-603b438a944a-trusted-ca\") pod \"image-registry-6c8dd7548c-jmns9\" (UID: \"52394690-486c-4ae0-b36b-603b438a944a\") " pod="openshift-image-registry/image-registry-6c8dd7548c-jmns9" Sep 11 03:53:40.339005 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.338952 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/badea905-61e6-47e1-8f68-505d04824821-cert\") pod \"ingress-canary-nq6qk\" (UID: \"badea905-61e6-47e1-8f68-505d04824821\") " pod="openshift-ingress-canary/ingress-canary-nq6qk" Sep 11 03:53:40.339005 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.338971 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11fca7d2-d884-4012-9431-b3c085b8339c-tmp-dir\") pod \"dns-default-ssrwd\" (UID: \"11fca7d2-d884-4012-9431-b3c085b8339c\") " pod="openshift-dns/dns-default-ssrwd" Sep 11 03:53:40.339140 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.339006 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn7rf\" (UniqueName: \"kubernetes.io/projected/11fca7d2-d884-4012-9431-b3c085b8339c-kube-api-access-cn7rf\") pod \"dns-default-ssrwd\" (UID: \"11fca7d2-d884-4012-9431-b3c085b8339c\") " pod="openshift-dns/dns-default-ssrwd" Sep 11 03:53:40.339140 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.339069 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/52394690-486c-4ae0-b36b-603b438a944a-ca-trust-extracted\") pod \"image-registry-6c8dd7548c-jmns9\" (UID: \"52394690-486c-4ae0-b36b-603b438a944a\") " pod="openshift-image-registry/image-registry-6c8dd7548c-jmns9" Sep 11 03:53:40.339140 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.339091 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlh7z\" (UniqueName: \"kubernetes.io/projected/badea905-61e6-47e1-8f68-505d04824821-kube-api-access-mlh7z\") pod \"ingress-canary-nq6qk\" (UID: \"badea905-61e6-47e1-8f68-505d04824821\") " pod="openshift-ingress-canary/ingress-canary-nq6qk" Sep 11 03:53:40.339140 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.339115 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/52394690-486c-4ae0-b36b-603b438a944a-installation-pull-secrets\") pod \"image-registry-6c8dd7548c-jmns9\" (UID: \"52394690-486c-4ae0-b36b-603b438a944a\") " pod="openshift-image-registry/image-registry-6c8dd7548c-jmns9" Sep 11 03:53:40.339140 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.339135 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/11fca7d2-d884-4012-9431-b3c085b8339c-config-volume\") pod \"dns-default-ssrwd\" (UID: \"11fca7d2-d884-4012-9431-b3c085b8339c\") " pod="openshift-dns/dns-default-ssrwd" Sep 11 03:53:40.339399 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.339377 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11fca7d2-d884-4012-9431-b3c085b8339c-tmp-dir\") pod \"dns-default-ssrwd\" (UID: \"11fca7d2-d884-4012-9431-b3c085b8339c\") " pod="openshift-dns/dns-default-ssrwd" Sep 11 03:53:40.339485 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.339460 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/52394690-486c-4ae0-b36b-603b438a944a-ca-trust-extracted\") pod \"image-registry-6c8dd7548c-jmns9\" (UID: \"52394690-486c-4ae0-b36b-603b438a944a\") " pod="openshift-image-registry/image-registry-6c8dd7548c-jmns9" Sep 11 03:53:40.339618 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.339584 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/52394690-486c-4ae0-b36b-603b438a944a-registry-certificates\") pod \"image-registry-6c8dd7548c-jmns9\" (UID: \"52394690-486c-4ae0-b36b-603b438a944a\") " pod="openshift-image-registry/image-registry-6c8dd7548c-jmns9" Sep 11 03:53:40.339714 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.339695 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/11fca7d2-d884-4012-9431-b3c085b8339c-config-volume\") pod \"dns-default-ssrwd\" (UID: \"11fca7d2-d884-4012-9431-b3c085b8339c\") " pod="openshift-dns/dns-default-ssrwd" Sep 11 03:53:40.340354 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.340335 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/52394690-486c-4ae0-b36b-603b438a944a-trusted-ca\") pod \"image-registry-6c8dd7548c-jmns9\" (UID: \"52394690-486c-4ae0-b36b-603b438a944a\") " pod="openshift-image-registry/image-registry-6c8dd7548c-jmns9" Sep 11 03:53:40.342976 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.342953 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/52394690-486c-4ae0-b36b-603b438a944a-installation-pull-secrets\") pod \"image-registry-6c8dd7548c-jmns9\" (UID: \"52394690-486c-4ae0-b36b-603b438a944a\") " pod="openshift-image-registry/image-registry-6c8dd7548c-jmns9" Sep 11 03:53:40.343079 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.342974 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/badea905-61e6-47e1-8f68-505d04824821-cert\") pod \"ingress-canary-nq6qk\" (UID: \"badea905-61e6-47e1-8f68-505d04824821\") " pod="openshift-ingress-canary/ingress-canary-nq6qk" Sep 11 03:53:40.343079 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.342958 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/52394690-486c-4ae0-b36b-603b438a944a-image-registry-private-configuration\") pod \"image-registry-6c8dd7548c-jmns9\" (UID: \"52394690-486c-4ae0-b36b-603b438a944a\") " pod="openshift-image-registry/image-registry-6c8dd7548c-jmns9" Sep 11 03:53:40.348863 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.348842 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/52394690-486c-4ae0-b36b-603b438a944a-bound-sa-token\") pod \"image-registry-6c8dd7548c-jmns9\" (UID: \"52394690-486c-4ae0-b36b-603b438a944a\") " pod="openshift-image-registry/image-registry-6c8dd7548c-jmns9" Sep 11 03:53:40.348974 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.348953 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn7rf\" (UniqueName: \"kubernetes.io/projected/11fca7d2-d884-4012-9431-b3c085b8339c-kube-api-access-cn7rf\") pod \"dns-default-ssrwd\" (UID: \"11fca7d2-d884-4012-9431-b3c085b8339c\") " pod="openshift-dns/dns-default-ssrwd" Sep 11 03:53:40.349779 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.349756 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wggx7\" (UniqueName: \"kubernetes.io/projected/52394690-486c-4ae0-b36b-603b438a944a-kube-api-access-wggx7\") pod \"image-registry-6c8dd7548c-jmns9\" (UID: \"52394690-486c-4ae0-b36b-603b438a944a\") " pod="openshift-image-registry/image-registry-6c8dd7548c-jmns9" Sep 11 03:53:40.349858 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.349822 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlh7z\" (UniqueName: \"kubernetes.io/projected/badea905-61e6-47e1-8f68-505d04824821-kube-api-access-mlh7z\") pod \"ingress-canary-nq6qk\" (UID: \"badea905-61e6-47e1-8f68-505d04824821\") " pod="openshift-ingress-canary/ingress-canary-nq6qk" Sep 11 03:53:40.433084 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.433060 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nq6qk" Sep 11 03:53:40.751265 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.751235 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nq6qk"] Sep 11 03:53:40.754461 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:53:40.754433 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbadea905_61e6_47e1_8f68_505d04824821.slice/crio-d031dcc9797453593991a1cb33a6ff5a04e40d2acd20c3e38445fe0d5be6371d WatchSource:0}: Error finding container d031dcc9797453593991a1cb33a6ff5a04e40d2acd20c3e38445fe0d5be6371d: Status 404 returned error can't find the container with id d031dcc9797453593991a1cb33a6ff5a04e40d2acd20c3e38445fe0d5be6371d Sep 11 03:53:40.762669 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.762648 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5fb2" Sep 11 03:53:40.768975 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.768625 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 03:53:40.844141 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.844119 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/52394690-486c-4ae0-b36b-603b438a944a-registry-tls\") pod \"image-registry-6c8dd7548c-jmns9\" (UID: \"52394690-486c-4ae0-b36b-603b438a944a\") " pod="openshift-image-registry/image-registry-6c8dd7548c-jmns9" Sep 11 03:53:40.844256 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.844147 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/11fca7d2-d884-4012-9431-b3c085b8339c-metrics-tls\") pod \"dns-default-ssrwd\" (UID: \"11fca7d2-d884-4012-9431-b3c085b8339c\") " pod="openshift-dns/dns-default-ssrwd" Sep 11 03:53:40.844295 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:40.844258 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 03:53:40.844295 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:40.844273 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 03:53:40.844295 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:40.844288 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c8dd7548c-jmns9: secret "image-registry-tls" not found Sep 11 03:53:40.844386 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:40.844312 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11fca7d2-d884-4012-9431-b3c085b8339c-metrics-tls podName:11fca7d2-d884-4012-9431-b3c085b8339c nodeName:}" failed. No retries permitted until 2026-09-11 03:53:41.844295747 +0000 UTC m=+36.597927128 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/11fca7d2-d884-4012-9431-b3c085b8339c-metrics-tls") pod "dns-default-ssrwd" (UID: "11fca7d2-d884-4012-9431-b3c085b8339c") : secret "dns-default-metrics-tls" not found Sep 11 03:53:40.844386 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:40.844330 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52394690-486c-4ae0-b36b-603b438a944a-registry-tls podName:52394690-486c-4ae0-b36b-603b438a944a nodeName:}" failed. No retries permitted until 2026-09-11 03:53:41.844317683 +0000 UTC m=+36.597949063 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/52394690-486c-4ae0-b36b-603b438a944a-registry-tls") pod "image-registry-6c8dd7548c-jmns9" (UID: "52394690-486c-4ae0-b36b-603b438a944a") : secret "image-registry-tls" not found Sep 11 03:53:40.940778 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.940749 2568 generic.go:356] "Generic (PLEG): container finished" podID="d6906bf8-7095-4f6a-8969-d6f28167548b" containerID="7b9b66c3b6a36b52a3a2b83ec128077cbd7d975b4e94b263a79a41fd61130e2a" exitCode=0 Sep 11 03:53:40.941585 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.940815 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9r65j" event={"ID":"d6906bf8-7095-4f6a-8969-d6f28167548b","Type":"ContainerDied","Data":"7b9b66c3b6a36b52a3a2b83ec128077cbd7d975b4e94b263a79a41fd61130e2a"} Sep 11 03:53:40.942041 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:40.942017 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nq6qk" event={"ID":"badea905-61e6-47e1-8f68-505d04824821","Type":"ContainerStarted","Data":"d031dcc9797453593991a1cb33a6ff5a04e40d2acd20c3e38445fe0d5be6371d"} Sep 11 03:53:41.762840 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:41.762649 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vssnm" Sep 11 03:53:41.763027 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:41.762680 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk8q7" Sep 11 03:53:41.766949 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:41.766481 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-j6dl8\"" Sep 11 03:53:41.766949 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:41.766529 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 11 03:53:41.766949 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:41.766547 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 03:53:41.766949 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:41.766685 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rtj7z\"" Sep 11 03:53:41.766949 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:41.766800 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 11 03:53:41.852223 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:41.852186 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/52394690-486c-4ae0-b36b-603b438a944a-registry-tls\") pod \"image-registry-6c8dd7548c-jmns9\" (UID: \"52394690-486c-4ae0-b36b-603b438a944a\") " pod="openshift-image-registry/image-registry-6c8dd7548c-jmns9" Sep 11 03:53:41.852223 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:41.852225 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/11fca7d2-d884-4012-9431-b3c085b8339c-metrics-tls\") pod \"dns-default-ssrwd\" (UID: \"11fca7d2-d884-4012-9431-b3c085b8339c\") " pod="openshift-dns/dns-default-ssrwd" Sep 11 03:53:41.852452 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:41.852378 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 03:53:41.852452 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:41.852427 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11fca7d2-d884-4012-9431-b3c085b8339c-metrics-tls podName:11fca7d2-d884-4012-9431-b3c085b8339c nodeName:}" failed. No retries permitted until 2026-09-11 03:53:43.852413892 +0000 UTC m=+38.606045286 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/11fca7d2-d884-4012-9431-b3c085b8339c-metrics-tls") pod "dns-default-ssrwd" (UID: "11fca7d2-d884-4012-9431-b3c085b8339c") : secret "dns-default-metrics-tls" not found Sep 11 03:53:41.852553 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:41.852379 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 03:53:41.852553 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:41.852471 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c8dd7548c-jmns9: secret "image-registry-tls" not found Sep 11 03:53:41.852553 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:41.852526 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52394690-486c-4ae0-b36b-603b438a944a-registry-tls podName:52394690-486c-4ae0-b36b-603b438a944a nodeName:}" failed. No retries permitted until 2026-09-11 03:53:43.852507676 +0000 UTC m=+38.606139076 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/52394690-486c-4ae0-b36b-603b438a944a-registry-tls") pod "image-registry-6c8dd7548c-jmns9" (UID: "52394690-486c-4ae0-b36b-603b438a944a") : secret "image-registry-tls" not found Sep 11 03:53:41.948519 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:41.948480 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9r65j" event={"ID":"d6906bf8-7095-4f6a-8969-d6f28167548b","Type":"ContainerStarted","Data":"0443fa11679e965990d3ea8ddd423973b9eb82f219fc5978b1453520e83644cc"} Sep 11 03:53:41.969463 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:41.969403 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9r65j" podStartSLOduration=8.175558089 podStartE2EDuration="36.969384653s" totalImagesPullingTime="28.793826564s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:53:05 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:53:41.969283404 +0000 UTC m=+36.722914805" watchObservedRunningTime="2026-09-11 03:53:41.969384653 +0000 UTC m=+36.723016055" Sep 11 03:53:42.090979 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:42.090891 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:53:42.145193 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:42.145157 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:53:43.868842 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:43.868806 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/52394690-486c-4ae0-b36b-603b438a944a-registry-tls\") pod \"image-registry-6c8dd7548c-jmns9\" (UID: \"52394690-486c-4ae0-b36b-603b438a944a\") " pod="openshift-image-registry/image-registry-6c8dd7548c-jmns9" Sep 11 03:53:43.868842 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:43.868846 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/11fca7d2-d884-4012-9431-b3c085b8339c-metrics-tls\") pod \"dns-default-ssrwd\" (UID: \"11fca7d2-d884-4012-9431-b3c085b8339c\") " pod="openshift-dns/dns-default-ssrwd" Sep 11 03:53:43.869307 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:43.868964 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 03:53:43.869307 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:43.868973 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 03:53:43.869307 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:43.868991 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c8dd7548c-jmns9: secret "image-registry-tls" not found Sep 11 03:53:43.869307 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:43.869017 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11fca7d2-d884-4012-9431-b3c085b8339c-metrics-tls podName:11fca7d2-d884-4012-9431-b3c085b8339c nodeName:}" failed. No retries permitted until 2026-09-11 03:53:47.869003228 +0000 UTC m=+42.622634608 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/11fca7d2-d884-4012-9431-b3c085b8339c-metrics-tls") pod "dns-default-ssrwd" (UID: "11fca7d2-d884-4012-9431-b3c085b8339c") : secret "dns-default-metrics-tls" not found Sep 11 03:53:43.869307 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:43.869035 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52394690-486c-4ae0-b36b-603b438a944a-registry-tls podName:52394690-486c-4ae0-b36b-603b438a944a nodeName:}" failed. No retries permitted until 2026-09-11 03:53:47.869022833 +0000 UTC m=+42.622654212 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/52394690-486c-4ae0-b36b-603b438a944a-registry-tls") pod "image-registry-6c8dd7548c-jmns9" (UID: "52394690-486c-4ae0-b36b-603b438a944a") : secret "image-registry-tls" not found Sep 11 03:53:43.958467 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:43.958424 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nq6qk" event={"ID":"badea905-61e6-47e1-8f68-505d04824821","Type":"ContainerStarted","Data":"d647b7fff8880ff4b2147a0d4f8744f8ceee082f8d08d1e61900a3743831fab8"} Sep 11 03:53:43.975523 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:43.975482 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-nq6qk" podStartSLOduration=1.342479071 podStartE2EDuration="3.975468464s" totalImagesPullingTime="2.632989393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:53:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:53:43.975146873 +0000 UTC m=+38.728778289" watchObservedRunningTime="2026-09-11 03:53:43.975468464 +0000 UTC m=+38.729099866" Sep 11 03:53:44.729117 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:44.729083 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:53:44.829330 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:44.829298 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:53:45.145979 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:45.145890 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:53:45.254061 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:45.254005 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:53:45.331062 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:45.331020 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:53:46.158284 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:46.158251 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:53:46.262668 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:46.262630 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:53:46.366231 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:46.366193 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:53:46.466811 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:46.466720 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:53:46.568603 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:46.568543 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:53:46.651066 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:46.651015 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:53:47.375617 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:47.375585 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:53:47.456034 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:47.455996 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:53:47.760672 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:47.760644 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:53:47.861163 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:47.861133 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:53:47.898297 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:47.898268 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/52394690-486c-4ae0-b36b-603b438a944a-registry-tls\") pod \"image-registry-6c8dd7548c-jmns9\" (UID: \"52394690-486c-4ae0-b36b-603b438a944a\") " pod="openshift-image-registry/image-registry-6c8dd7548c-jmns9" Sep 11 03:53:47.898429 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:47.898302 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/11fca7d2-d884-4012-9431-b3c085b8339c-metrics-tls\") pod \"dns-default-ssrwd\" (UID: \"11fca7d2-d884-4012-9431-b3c085b8339c\") " pod="openshift-dns/dns-default-ssrwd" Sep 11 03:53:47.898429 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:47.898412 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 03:53:47.898429 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:47.898423 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 03:53:47.898591 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:47.898486 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11fca7d2-d884-4012-9431-b3c085b8339c-metrics-tls podName:11fca7d2-d884-4012-9431-b3c085b8339c nodeName:}" failed. No retries permitted until 2026-09-11 03:53:55.898468801 +0000 UTC m=+50.652100189 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/11fca7d2-d884-4012-9431-b3c085b8339c-metrics-tls") pod "dns-default-ssrwd" (UID: "11fca7d2-d884-4012-9431-b3c085b8339c") : secret "dns-default-metrics-tls" not found Sep 11 03:53:47.898591 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:47.898428 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c8dd7548c-jmns9: secret "image-registry-tls" not found Sep 11 03:53:47.898591 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:47.898562 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52394690-486c-4ae0-b36b-603b438a944a-registry-tls podName:52394690-486c-4ae0-b36b-603b438a944a nodeName:}" failed. No retries permitted until 2026-09-11 03:53:55.898548854 +0000 UTC m=+50.652180238 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/52394690-486c-4ae0-b36b-603b438a944a-registry-tls") pod "image-registry-6c8dd7548c-jmns9" (UID: "52394690-486c-4ae0-b36b-603b438a944a") : secret "image-registry-tls" not found Sep 11 03:53:50.812514 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:50.812478 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:53:50.908775 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:50.908740 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:53:53.389781 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:53.389746 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:53:55.957629 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:55.957593 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/52394690-486c-4ae0-b36b-603b438a944a-registry-tls\") pod \"image-registry-6c8dd7548c-jmns9\" (UID: \"52394690-486c-4ae0-b36b-603b438a944a\") " pod="openshift-image-registry/image-registry-6c8dd7548c-jmns9" Sep 11 03:53:55.957629 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:55.957631 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/11fca7d2-d884-4012-9431-b3c085b8339c-metrics-tls\") pod \"dns-default-ssrwd\" (UID: \"11fca7d2-d884-4012-9431-b3c085b8339c\") " pod="openshift-dns/dns-default-ssrwd" Sep 11 03:53:55.958092 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:55.957739 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 03:53:55.958092 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:55.957766 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 03:53:55.958092 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:55.957785 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c8dd7548c-jmns9: secret "image-registry-tls" not found Sep 11 03:53:55.958092 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:55.957796 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11fca7d2-d884-4012-9431-b3c085b8339c-metrics-tls podName:11fca7d2-d884-4012-9431-b3c085b8339c nodeName:}" failed. No retries permitted until 2026-09-11 03:54:11.957782556 +0000 UTC m=+66.711413936 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/11fca7d2-d884-4012-9431-b3c085b8339c-metrics-tls") pod "dns-default-ssrwd" (UID: "11fca7d2-d884-4012-9431-b3c085b8339c") : secret "dns-default-metrics-tls" not found Sep 11 03:53:55.958092 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:53:55.957901 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52394690-486c-4ae0-b36b-603b438a944a-registry-tls podName:52394690-486c-4ae0-b36b-603b438a944a nodeName:}" failed. No retries permitted until 2026-09-11 03:54:11.957887052 +0000 UTC m=+66.711518431 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/52394690-486c-4ae0-b36b-603b438a944a-registry-tls") pod "image-registry-6c8dd7548c-jmns9" (UID: "52394690-486c-4ae0-b36b-603b438a944a") : secret "image-registry-tls" not found Sep 11 03:53:57.152978 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:53:57.152937 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:00.243439 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:00.243402 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:00.343291 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:00.343260 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:02.217812 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:02.217778 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:02.278764 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:02.278734 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:04.604656 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.604625 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-gqfwl"] Sep 11 03:54:04.607247 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.607225 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-5tpdc"] Sep 11 03:54:04.607363 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.607344 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-gqfwl" Sep 11 03:54:04.608810 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.608789 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-5tpdc" Sep 11 03:54:04.610223 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.610203 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-ttjms\"" Sep 11 03:54:04.610407 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.610393 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 11 03:54:04.610793 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.610781 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 11 03:54:04.612232 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.612218 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Sep 11 03:54:04.613702 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.613686 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-mcrgq\"" Sep 11 03:54:04.613775 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.613702 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Sep 11 03:54:04.613944 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.613928 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Sep 11 03:54:04.614260 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.614237 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Sep 11 03:54:04.615648 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.615630 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9112557d-d0ef-44c7-8b2a-f6ecd0a980d6-serving-cert\") pod \"console-operator-696884fb9c-5tpdc\" (UID: \"9112557d-d0ef-44c7-8b2a-f6ecd0a980d6\") " pod="openshift-console-operator/console-operator-696884fb9c-5tpdc" Sep 11 03:54:04.615752 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.615654 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9112557d-d0ef-44c7-8b2a-f6ecd0a980d6-trusted-ca\") pod \"console-operator-696884fb9c-5tpdc\" (UID: \"9112557d-d0ef-44c7-8b2a-f6ecd0a980d6\") " pod="openshift-console-operator/console-operator-696884fb9c-5tpdc" Sep 11 03:54:04.615752 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.615698 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vnjv\" (UniqueName: \"kubernetes.io/projected/9112557d-d0ef-44c7-8b2a-f6ecd0a980d6-kube-api-access-6vnjv\") pod \"console-operator-696884fb9c-5tpdc\" (UID: \"9112557d-d0ef-44c7-8b2a-f6ecd0a980d6\") " pod="openshift-console-operator/console-operator-696884fb9c-5tpdc" Sep 11 03:54:04.615884 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.615827 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9112557d-d0ef-44c7-8b2a-f6ecd0a980d6-config\") pod \"console-operator-696884fb9c-5tpdc\" (UID: \"9112557d-d0ef-44c7-8b2a-f6ecd0a980d6\") " pod="openshift-console-operator/console-operator-696884fb9c-5tpdc" Sep 11 03:54:04.615884 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.615872 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chpv5\" (UniqueName: \"kubernetes.io/projected/03b84729-9693-4100-b064-2cfb5e385273-kube-api-access-chpv5\") pod \"volume-data-source-validator-86695b7cc7-gqfwl\" (UID: \"03b84729-9693-4100-b064-2cfb5e385273\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-gqfwl" Sep 11 03:54:04.620095 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.620073 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-gqfwl"] Sep 11 03:54:04.621109 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.621087 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-5tpdc"] Sep 11 03:54:04.622124 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.622104 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Sep 11 03:54:04.677726 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.677691 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qdnbj"] Sep 11 03:54:04.679716 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.679701 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qdnbj" Sep 11 03:54:04.682479 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.682435 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 11 03:54:04.682670 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.682650 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-6pp4m\"" Sep 11 03:54:04.682868 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.682699 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 11 03:54:04.682978 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.682905 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 11 03:54:04.708499 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.708477 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qdnbj"] Sep 11 03:54:04.716113 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.716090 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d3a584c-2a1b-4c64-9fd6-89d7527d6c87-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-qdnbj\" (UID: \"7d3a584c-2a1b-4c64-9fd6-89d7527d6c87\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qdnbj" Sep 11 03:54:04.716197 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.716122 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vnjv\" (UniqueName: \"kubernetes.io/projected/9112557d-d0ef-44c7-8b2a-f6ecd0a980d6-kube-api-access-6vnjv\") pod \"console-operator-696884fb9c-5tpdc\" (UID: \"9112557d-d0ef-44c7-8b2a-f6ecd0a980d6\") " pod="openshift-console-operator/console-operator-696884fb9c-5tpdc" Sep 11 03:54:04.716239 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.716215 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9112557d-d0ef-44c7-8b2a-f6ecd0a980d6-config\") pod \"console-operator-696884fb9c-5tpdc\" (UID: \"9112557d-d0ef-44c7-8b2a-f6ecd0a980d6\") " pod="openshift-console-operator/console-operator-696884fb9c-5tpdc" Sep 11 03:54:04.716283 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.716266 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chpv5\" (UniqueName: \"kubernetes.io/projected/03b84729-9693-4100-b064-2cfb5e385273-kube-api-access-chpv5\") pod \"volume-data-source-validator-86695b7cc7-gqfwl\" (UID: \"03b84729-9693-4100-b064-2cfb5e385273\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-gqfwl" Sep 11 03:54:04.716321 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.716300 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9112557d-d0ef-44c7-8b2a-f6ecd0a980d6-serving-cert\") pod \"console-operator-696884fb9c-5tpdc\" (UID: \"9112557d-d0ef-44c7-8b2a-f6ecd0a980d6\") " pod="openshift-console-operator/console-operator-696884fb9c-5tpdc" Sep 11 03:54:04.716406 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.716325 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9112557d-d0ef-44c7-8b2a-f6ecd0a980d6-trusted-ca\") pod \"console-operator-696884fb9c-5tpdc\" (UID: \"9112557d-d0ef-44c7-8b2a-f6ecd0a980d6\") " pod="openshift-console-operator/console-operator-696884fb9c-5tpdc" Sep 11 03:54:04.716406 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.716366 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbw7f\" (UniqueName: \"kubernetes.io/projected/7d3a584c-2a1b-4c64-9fd6-89d7527d6c87-kube-api-access-gbw7f\") pod \"cluster-samples-operator-86499c89c4-qdnbj\" (UID: \"7d3a584c-2a1b-4c64-9fd6-89d7527d6c87\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qdnbj" Sep 11 03:54:04.716948 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.716925 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9112557d-d0ef-44c7-8b2a-f6ecd0a980d6-config\") pod \"console-operator-696884fb9c-5tpdc\" (UID: \"9112557d-d0ef-44c7-8b2a-f6ecd0a980d6\") " pod="openshift-console-operator/console-operator-696884fb9c-5tpdc" Sep 11 03:54:04.717180 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.717165 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9112557d-d0ef-44c7-8b2a-f6ecd0a980d6-trusted-ca\") pod \"console-operator-696884fb9c-5tpdc\" (UID: \"9112557d-d0ef-44c7-8b2a-f6ecd0a980d6\") " pod="openshift-console-operator/console-operator-696884fb9c-5tpdc" Sep 11 03:54:04.719486 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.719469 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9112557d-d0ef-44c7-8b2a-f6ecd0a980d6-serving-cert\") pod \"console-operator-696884fb9c-5tpdc\" (UID: \"9112557d-d0ef-44c7-8b2a-f6ecd0a980d6\") " pod="openshift-console-operator/console-operator-696884fb9c-5tpdc" Sep 11 03:54:04.731751 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.731730 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chpv5\" (UniqueName: \"kubernetes.io/projected/03b84729-9693-4100-b064-2cfb5e385273-kube-api-access-chpv5\") pod \"volume-data-source-validator-86695b7cc7-gqfwl\" (UID: \"03b84729-9693-4100-b064-2cfb5e385273\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-gqfwl" Sep 11 03:54:04.731870 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.731756 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vnjv\" (UniqueName: \"kubernetes.io/projected/9112557d-d0ef-44c7-8b2a-f6ecd0a980d6-kube-api-access-6vnjv\") pod \"console-operator-696884fb9c-5tpdc\" (UID: \"9112557d-d0ef-44c7-8b2a-f6ecd0a980d6\") " pod="openshift-console-operator/console-operator-696884fb9c-5tpdc" Sep 11 03:54:04.781956 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.781933 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5844b856db-4w9k5"] Sep 11 03:54:04.784219 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.784205 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-b67kh"] Sep 11 03:54:04.784335 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.784319 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-4w9k5" Sep 11 03:54:04.788327 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.787387 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-b67kh" Sep 11 03:54:04.788327 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.787489 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 11 03:54:04.788327 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.787682 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 11 03:54:04.788327 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.787828 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-g6nj4\"" Sep 11 03:54:04.788327 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.788167 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 11 03:54:04.788327 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.788238 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 11 03:54:04.790863 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.790840 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 11 03:54:04.791916 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.791897 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 11 03:54:04.792018 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.791947 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 11 03:54:04.792101 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.792026 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 11 03:54:04.792381 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.792363 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-pgcdx\"" Sep 11 03:54:04.794200 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.794063 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 11 03:54:04.805669 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.805650 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-4w9k5"] Sep 11 03:54:04.816838 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.816807 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbw7f\" (UniqueName: \"kubernetes.io/projected/7d3a584c-2a1b-4c64-9fd6-89d7527d6c87-kube-api-access-gbw7f\") pod \"cluster-samples-operator-86499c89c4-qdnbj\" (UID: \"7d3a584c-2a1b-4c64-9fd6-89d7527d6c87\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qdnbj" Sep 11 03:54:04.816939 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.816852 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7be9d187-475e-4a85-afc1-2f385d5a262e-service-ca-bundle\") pod \"insights-operator-5844b856db-4w9k5\" (UID: \"7be9d187-475e-4a85-afc1-2f385d5a262e\") " pod="openshift-insights/insights-operator-5844b856db-4w9k5" Sep 11 03:54:04.816939 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.816873 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb8gt\" (UniqueName: \"kubernetes.io/projected/320f0913-5ebe-45b4-a60d-8359a83bd24b-kube-api-access-jb8gt\") pod \"kube-storage-version-migrator-operator-6c8f84f897-b67kh\" (UID: \"320f0913-5ebe-45b4-a60d-8359a83bd24b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-b67kh" Sep 11 03:54:04.817025 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.816933 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/320f0913-5ebe-45b4-a60d-8359a83bd24b-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-b67kh\" (UID: \"320f0913-5ebe-45b4-a60d-8359a83bd24b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-b67kh" Sep 11 03:54:04.817025 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.816964 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7be9d187-475e-4a85-afc1-2f385d5a262e-trusted-ca-bundle\") pod \"insights-operator-5844b856db-4w9k5\" (UID: \"7be9d187-475e-4a85-afc1-2f385d5a262e\") " pod="openshift-insights/insights-operator-5844b856db-4w9k5" Sep 11 03:54:04.817025 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.816998 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/320f0913-5ebe-45b4-a60d-8359a83bd24b-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-b67kh\" (UID: \"320f0913-5ebe-45b4-a60d-8359a83bd24b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-b67kh" Sep 11 03:54:04.817156 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.817034 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7be9d187-475e-4a85-afc1-2f385d5a262e-tmp\") pod \"insights-operator-5844b856db-4w9k5\" (UID: \"7be9d187-475e-4a85-afc1-2f385d5a262e\") " pod="openshift-insights/insights-operator-5844b856db-4w9k5" Sep 11 03:54:04.817156 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.817062 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7be9d187-475e-4a85-afc1-2f385d5a262e-snapshots\") pod \"insights-operator-5844b856db-4w9k5\" (UID: \"7be9d187-475e-4a85-afc1-2f385d5a262e\") " pod="openshift-insights/insights-operator-5844b856db-4w9k5" Sep 11 03:54:04.817156 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.817089 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d3a584c-2a1b-4c64-9fd6-89d7527d6c87-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-qdnbj\" (UID: \"7d3a584c-2a1b-4c64-9fd6-89d7527d6c87\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qdnbj" Sep 11 03:54:04.817284 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:54:04.817160 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 03:54:04.817284 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.817164 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8q9f\" (UniqueName: \"kubernetes.io/projected/7be9d187-475e-4a85-afc1-2f385d5a262e-kube-api-access-c8q9f\") pod \"insights-operator-5844b856db-4w9k5\" (UID: \"7be9d187-475e-4a85-afc1-2f385d5a262e\") " pod="openshift-insights/insights-operator-5844b856db-4w9k5" Sep 11 03:54:04.817284 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:54:04.817225 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d3a584c-2a1b-4c64-9fd6-89d7527d6c87-samples-operator-tls podName:7d3a584c-2a1b-4c64-9fd6-89d7527d6c87 nodeName:}" failed. No retries permitted until 2026-09-11 03:54:05.31721181 +0000 UTC m=+60.070843194 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7d3a584c-2a1b-4c64-9fd6-89d7527d6c87-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-qdnbj" (UID: "7d3a584c-2a1b-4c64-9fd6-89d7527d6c87") : secret "samples-operator-tls" not found Sep 11 03:54:04.817284 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.817265 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7be9d187-475e-4a85-afc1-2f385d5a262e-serving-cert\") pod \"insights-operator-5844b856db-4w9k5\" (UID: \"7be9d187-475e-4a85-afc1-2f385d5a262e\") " pod="openshift-insights/insights-operator-5844b856db-4w9k5" Sep 11 03:54:04.821227 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.821206 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-b67kh"] Sep 11 03:54:04.827412 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.827393 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbw7f\" (UniqueName: \"kubernetes.io/projected/7d3a584c-2a1b-4c64-9fd6-89d7527d6c87-kube-api-access-gbw7f\") pod \"cluster-samples-operator-86499c89c4-qdnbj\" (UID: \"7d3a584c-2a1b-4c64-9fd6-89d7527d6c87\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qdnbj" Sep 11 03:54:04.917405 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.917368 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-gqfwl" Sep 11 03:54:04.917671 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.917647 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/320f0913-5ebe-45b4-a60d-8359a83bd24b-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-b67kh\" (UID: \"320f0913-5ebe-45b4-a60d-8359a83bd24b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-b67kh" Sep 11 03:54:04.917735 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.917690 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7be9d187-475e-4a85-afc1-2f385d5a262e-trusted-ca-bundle\") pod \"insights-operator-5844b856db-4w9k5\" (UID: \"7be9d187-475e-4a85-afc1-2f385d5a262e\") " pod="openshift-insights/insights-operator-5844b856db-4w9k5" Sep 11 03:54:04.917735 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.917714 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/320f0913-5ebe-45b4-a60d-8359a83bd24b-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-b67kh\" (UID: \"320f0913-5ebe-45b4-a60d-8359a83bd24b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-b67kh" Sep 11 03:54:04.917926 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.917909 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7be9d187-475e-4a85-afc1-2f385d5a262e-tmp\") pod \"insights-operator-5844b856db-4w9k5\" (UID: \"7be9d187-475e-4a85-afc1-2f385d5a262e\") " pod="openshift-insights/insights-operator-5844b856db-4w9k5" Sep 11 03:54:04.918010 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.917931 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7be9d187-475e-4a85-afc1-2f385d5a262e-snapshots\") pod \"insights-operator-5844b856db-4w9k5\" (UID: \"7be9d187-475e-4a85-afc1-2f385d5a262e\") " pod="openshift-insights/insights-operator-5844b856db-4w9k5" Sep 11 03:54:04.918010 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.917955 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8q9f\" (UniqueName: \"kubernetes.io/projected/7be9d187-475e-4a85-afc1-2f385d5a262e-kube-api-access-c8q9f\") pod \"insights-operator-5844b856db-4w9k5\" (UID: \"7be9d187-475e-4a85-afc1-2f385d5a262e\") " pod="openshift-insights/insights-operator-5844b856db-4w9k5" Sep 11 03:54:04.918010 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.917970 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7be9d187-475e-4a85-afc1-2f385d5a262e-serving-cert\") pod \"insights-operator-5844b856db-4w9k5\" (UID: \"7be9d187-475e-4a85-afc1-2f385d5a262e\") " pod="openshift-insights/insights-operator-5844b856db-4w9k5" Sep 11 03:54:04.918203 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.918019 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7be9d187-475e-4a85-afc1-2f385d5a262e-service-ca-bundle\") pod \"insights-operator-5844b856db-4w9k5\" (UID: \"7be9d187-475e-4a85-afc1-2f385d5a262e\") " pod="openshift-insights/insights-operator-5844b856db-4w9k5" Sep 11 03:54:04.918203 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.918064 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb8gt\" (UniqueName: \"kubernetes.io/projected/320f0913-5ebe-45b4-a60d-8359a83bd24b-kube-api-access-jb8gt\") pod \"kube-storage-version-migrator-operator-6c8f84f897-b67kh\" (UID: \"320f0913-5ebe-45b4-a60d-8359a83bd24b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-b67kh" Sep 11 03:54:04.918418 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.918364 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/320f0913-5ebe-45b4-a60d-8359a83bd24b-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-b67kh\" (UID: \"320f0913-5ebe-45b4-a60d-8359a83bd24b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-b67kh" Sep 11 03:54:04.918683 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.918657 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7be9d187-475e-4a85-afc1-2f385d5a262e-tmp\") pod \"insights-operator-5844b856db-4w9k5\" (UID: \"7be9d187-475e-4a85-afc1-2f385d5a262e\") " pod="openshift-insights/insights-operator-5844b856db-4w9k5" Sep 11 03:54:04.918773 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.918691 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/7be9d187-475e-4a85-afc1-2f385d5a262e-snapshots\") pod \"insights-operator-5844b856db-4w9k5\" (UID: \"7be9d187-475e-4a85-afc1-2f385d5a262e\") " pod="openshift-insights/insights-operator-5844b856db-4w9k5" Sep 11 03:54:04.918832 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.918782 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7be9d187-475e-4a85-afc1-2f385d5a262e-service-ca-bundle\") pod \"insights-operator-5844b856db-4w9k5\" (UID: \"7be9d187-475e-4a85-afc1-2f385d5a262e\") " pod="openshift-insights/insights-operator-5844b856db-4w9k5" Sep 11 03:54:04.918885 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.918858 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7be9d187-475e-4a85-afc1-2f385d5a262e-trusted-ca-bundle\") pod \"insights-operator-5844b856db-4w9k5\" (UID: \"7be9d187-475e-4a85-afc1-2f385d5a262e\") " pod="openshift-insights/insights-operator-5844b856db-4w9k5" Sep 11 03:54:04.920246 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.920216 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/320f0913-5ebe-45b4-a60d-8359a83bd24b-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-b67kh\" (UID: \"320f0913-5ebe-45b4-a60d-8359a83bd24b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-b67kh" Sep 11 03:54:04.920929 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.920450 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7be9d187-475e-4a85-afc1-2f385d5a262e-serving-cert\") pod \"insights-operator-5844b856db-4w9k5\" (UID: \"7be9d187-475e-4a85-afc1-2f385d5a262e\") " pod="openshift-insights/insights-operator-5844b856db-4w9k5" Sep 11 03:54:04.922624 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.922605 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-5tpdc" Sep 11 03:54:04.926779 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.926728 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb8gt\" (UniqueName: \"kubernetes.io/projected/320f0913-5ebe-45b4-a60d-8359a83bd24b-kube-api-access-jb8gt\") pod \"kube-storage-version-migrator-operator-6c8f84f897-b67kh\" (UID: \"320f0913-5ebe-45b4-a60d-8359a83bd24b\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-b67kh" Sep 11 03:54:04.927788 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.927693 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8q9f\" (UniqueName: \"kubernetes.io/projected/7be9d187-475e-4a85-afc1-2f385d5a262e-kube-api-access-c8q9f\") pod \"insights-operator-5844b856db-4w9k5\" (UID: \"7be9d187-475e-4a85-afc1-2f385d5a262e\") " pod="openshift-insights/insights-operator-5844b856db-4w9k5" Sep 11 03:54:04.928324 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.928299 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:04.932203 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:04.932184 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-htjph" Sep 11 03:54:05.030678 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:05.030642 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:05.097376 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:05.097344 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5844b856db-4w9k5" Sep 11 03:54:05.101634 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:05.101616 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-b67kh" Sep 11 03:54:05.134335 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:05.134273 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:05.191119 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:05.191012 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-gqfwl"] Sep 11 03:54:05.194307 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:54:05.194280 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b84729_9693_4100_b064_2cfb5e385273.slice/crio-a69dfddf3b583b317718c30c2d9213d7203f7b1befe47e203933794e7b0a4014 WatchSource:0}: Error finding container a69dfddf3b583b317718c30c2d9213d7203f7b1befe47e203933794e7b0a4014: Status 404 returned error can't find the container with id a69dfddf3b583b317718c30c2d9213d7203f7b1befe47e203933794e7b0a4014 Sep 11 03:54:05.196485 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:05.196362 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-5tpdc"] Sep 11 03:54:05.199632 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:54:05.199612 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9112557d_d0ef_44c7_8b2a_f6ecd0a980d6.slice/crio-d39567dab42184c2e7a14a2867b4eb7db7779c14481faf1347d9ac6d7ee92a81 WatchSource:0}: Error finding container d39567dab42184c2e7a14a2867b4eb7db7779c14481faf1347d9ac6d7ee92a81: Status 404 returned error can't find the container with id d39567dab42184c2e7a14a2867b4eb7db7779c14481faf1347d9ac6d7ee92a81 Sep 11 03:54:05.321727 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:05.321690 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d3a584c-2a1b-4c64-9fd6-89d7527d6c87-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-qdnbj\" (UID: \"7d3a584c-2a1b-4c64-9fd6-89d7527d6c87\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qdnbj" Sep 11 03:54:05.321898 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:54:05.321820 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 03:54:05.321898 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:54:05.321876 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d3a584c-2a1b-4c64-9fd6-89d7527d6c87-samples-operator-tls podName:7d3a584c-2a1b-4c64-9fd6-89d7527d6c87 nodeName:}" failed. No retries permitted until 2026-09-11 03:54:06.321860852 +0000 UTC m=+61.075492232 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7d3a584c-2a1b-4c64-9fd6-89d7527d6c87-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-qdnbj" (UID: "7d3a584c-2a1b-4c64-9fd6-89d7527d6c87") : secret "samples-operator-tls" not found Sep 11 03:54:05.367007 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:05.366986 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5844b856db-4w9k5"] Sep 11 03:54:05.370171 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:05.370133 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-b67kh"] Sep 11 03:54:05.370171 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:54:05.370157 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7be9d187_475e_4a85_afc1_2f385d5a262e.slice/crio-3a6d03dbe1c4ceaeb32a8927202edf203609a3ce7c25edfc1c90b3bb5d222e68 WatchSource:0}: Error finding container 3a6d03dbe1c4ceaeb32a8927202edf203609a3ce7c25edfc1c90b3bb5d222e68: Status 404 returned error can't find the container with id 3a6d03dbe1c4ceaeb32a8927202edf203609a3ce7c25edfc1c90b3bb5d222e68 Sep 11 03:54:05.374108 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:54:05.374082 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod320f0913_5ebe_45b4_a60d_8359a83bd24b.slice/crio-ecd8c6d730722573f4b95f1af5ada25265e780bd466893a993f74b9efc51b5b9 WatchSource:0}: Error finding container ecd8c6d730722573f4b95f1af5ada25265e780bd466893a993f74b9efc51b5b9: Status 404 returned error can't find the container with id ecd8c6d730722573f4b95f1af5ada25265e780bd466893a993f74b9efc51b5b9 Sep 11 03:54:05.655928 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:05.655888 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:05.712997 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:05.712962 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:06.015254 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:06.015126 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-4w9k5" event={"ID":"7be9d187-475e-4a85-afc1-2f385d5a262e","Type":"ContainerStarted","Data":"3a6d03dbe1c4ceaeb32a8927202edf203609a3ce7c25edfc1c90b3bb5d222e68"} Sep 11 03:54:06.019246 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:06.019216 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-b67kh" event={"ID":"320f0913-5ebe-45b4-a60d-8359a83bd24b","Type":"ContainerStarted","Data":"ecd8c6d730722573f4b95f1af5ada25265e780bd466893a993f74b9efc51b5b9"} Sep 11 03:54:06.021828 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:06.021798 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-5tpdc" event={"ID":"9112557d-d0ef-44c7-8b2a-f6ecd0a980d6","Type":"ContainerStarted","Data":"d39567dab42184c2e7a14a2867b4eb7db7779c14481faf1347d9ac6d7ee92a81"} Sep 11 03:54:06.023880 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:06.023855 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-gqfwl" event={"ID":"03b84729-9693-4100-b064-2cfb5e385273","Type":"ContainerStarted","Data":"a69dfddf3b583b317718c30c2d9213d7203f7b1befe47e203933794e7b0a4014"} Sep 11 03:54:06.071533 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:06.071496 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:06.130658 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:06.130625 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:06.331479 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:06.331387 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d3a584c-2a1b-4c64-9fd6-89d7527d6c87-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-qdnbj\" (UID: \"7d3a584c-2a1b-4c64-9fd6-89d7527d6c87\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qdnbj" Sep 11 03:54:06.331684 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:54:06.331664 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 03:54:06.331794 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:54:06.331740 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d3a584c-2a1b-4c64-9fd6-89d7527d6c87-samples-operator-tls podName:7d3a584c-2a1b-4c64-9fd6-89d7527d6c87 nodeName:}" failed. No retries permitted until 2026-09-11 03:54:08.331720773 +0000 UTC m=+63.085352152 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7d3a584c-2a1b-4c64-9fd6-89d7527d6c87-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-qdnbj" (UID: "7d3a584c-2a1b-4c64-9fd6-89d7527d6c87") : secret "samples-operator-tls" not found Sep 11 03:54:07.028322 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:07.028218 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-gqfwl" event={"ID":"03b84729-9693-4100-b064-2cfb5e385273","Type":"ContainerStarted","Data":"545278962c18ffeec5053818c7acbecc3396772f43247cbf2ecedd99d7740000"} Sep 11 03:54:07.042827 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:07.042774 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-gqfwl" podStartSLOduration=1.503166944 podStartE2EDuration="3.042757818s" totalImagesPullingTime="1.539590874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:54:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:07.042336973 +0000 UTC m=+61.795968394" watchObservedRunningTime="2026-09-11 03:54:07.042757818 +0000 UTC m=+61.796389220" Sep 11 03:54:07.148298 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:07.148186 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:07.250027 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:07.249987 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:07.537529 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:07.537480 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:07.636169 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:07.636138 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:07.694886 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:07.694849 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:08.347397 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:08.347347 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d3a584c-2a1b-4c64-9fd6-89d7527d6c87-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-qdnbj\" (UID: \"7d3a584c-2a1b-4c64-9fd6-89d7527d6c87\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qdnbj" Sep 11 03:54:08.347772 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:54:08.347535 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 03:54:08.347772 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:54:08.347605 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d3a584c-2a1b-4c64-9fd6-89d7527d6c87-samples-operator-tls podName:7d3a584c-2a1b-4c64-9fd6-89d7527d6c87 nodeName:}" failed. No retries permitted until 2026-09-11 03:54:12.347588702 +0000 UTC m=+67.101220082 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7d3a584c-2a1b-4c64-9fd6-89d7527d6c87-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-qdnbj" (UID: "7d3a584c-2a1b-4c64-9fd6-89d7527d6c87") : secret "samples-operator-tls" not found Sep 11 03:54:08.553808 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:08.553768 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:08.617957 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:08.617872 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:09.039352 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:09.039313 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-b67kh" event={"ID":"320f0913-5ebe-45b4-a60d-8359a83bd24b","Type":"ContainerStarted","Data":"7bb112cde210b5ca8e4295c44716b4faa283644f63e724b54346e6e82b01fb01"} Sep 11 03:54:09.041028 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:09.041004 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-5tpdc_9112557d-d0ef-44c7-8b2a-f6ecd0a980d6/console-operator/0.log" Sep 11 03:54:09.041176 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:09.041063 2568 generic.go:356] "Generic (PLEG): container finished" podID="9112557d-d0ef-44c7-8b2a-f6ecd0a980d6" containerID="a674c1dcbe1675a763b71664c40c40662fd6941799ca4168e0c48c5c64470b56" exitCode=255 Sep 11 03:54:09.041176 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:09.041144 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-5tpdc" event={"ID":"9112557d-d0ef-44c7-8b2a-f6ecd0a980d6","Type":"ContainerDied","Data":"a674c1dcbe1675a763b71664c40c40662fd6941799ca4168e0c48c5c64470b56"} Sep 11 03:54:09.041340 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:09.041306 2568 scope.go:117] "RemoveContainer" containerID="a674c1dcbe1675a763b71664c40c40662fd6941799ca4168e0c48c5c64470b56" Sep 11 03:54:09.042809 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:09.042785 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-4w9k5" event={"ID":"7be9d187-475e-4a85-afc1-2f385d5a262e","Type":"ContainerStarted","Data":"d01bdb1e50838341b79260bc870b9f5ee37f042aa4f7d53ed2f5fd5252fd5226"} Sep 11 03:54:09.055307 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:09.055260 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-b67kh" podStartSLOduration=2.002383081 podStartE2EDuration="5.055242502s" totalImagesPullingTime="3.052859421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:54:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:09.054465633 +0000 UTC m=+63.808097038" watchObservedRunningTime="2026-09-11 03:54:09.055242502 +0000 UTC m=+63.808873899" Sep 11 03:54:09.085634 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:09.085577 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5844b856db-4w9k5" podStartSLOduration=2.033076066 podStartE2EDuration="5.085560512s" totalImagesPullingTime="3.052484446s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:54:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:09.085520107 +0000 UTC m=+63.839151512" watchObservedRunningTime="2026-09-11 03:54:09.085560512 +0000 UTC m=+63.839191917" Sep 11 03:54:09.239781 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:09.239743 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-phn2f"] Sep 11 03:54:09.242877 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:09.242855 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-phn2f" Sep 11 03:54:09.245193 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:09.245166 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 11 03:54:09.246365 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:09.246119 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xkdhm\"" Sep 11 03:54:09.246365 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:09.246227 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 11 03:54:09.252439 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:09.252414 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-phn2f"] Sep 11 03:54:09.255588 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:09.255567 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f-crio-socket\") pod \"insights-runtime-extractor-phn2f\" (UID: \"c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f\") " pod="openshift-insights/insights-runtime-extractor-phn2f" Sep 11 03:54:09.255669 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:09.255595 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x487p\" (UniqueName: \"kubernetes.io/projected/c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f-kube-api-access-x487p\") pod \"insights-runtime-extractor-phn2f\" (UID: \"c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f\") " pod="openshift-insights/insights-runtime-extractor-phn2f" Sep 11 03:54:09.255708 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:09.255670 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f-data-volume\") pod \"insights-runtime-extractor-phn2f\" (UID: \"c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f\") " pod="openshift-insights/insights-runtime-extractor-phn2f" Sep 11 03:54:09.255744 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:09.255710 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-phn2f\" (UID: \"c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f\") " pod="openshift-insights/insights-runtime-extractor-phn2f" Sep 11 03:54:09.255744 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:09.255738 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-phn2f\" (UID: \"c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f\") " pod="openshift-insights/insights-runtime-extractor-phn2f" Sep 11 03:54:09.257598 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:09.257571 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:09.313921 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:09.313854 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:09.356975 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:09.356948 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f-data-volume\") pod \"insights-runtime-extractor-phn2f\" (UID: \"c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f\") " pod="openshift-insights/insights-runtime-extractor-phn2f" Sep 11 03:54:09.357373 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:09.356993 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-phn2f\" (UID: \"c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f\") " pod="openshift-insights/insights-runtime-extractor-phn2f" Sep 11 03:54:09.357373 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:09.357023 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-phn2f\" (UID: \"c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f\") " pod="openshift-insights/insights-runtime-extractor-phn2f" Sep 11 03:54:09.357373 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:09.357076 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f-crio-socket\") pod \"insights-runtime-extractor-phn2f\" (UID: \"c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f\") " pod="openshift-insights/insights-runtime-extractor-phn2f" Sep 11 03:54:09.357373 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:09.357092 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x487p\" (UniqueName: \"kubernetes.io/projected/c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f-kube-api-access-x487p\") pod \"insights-runtime-extractor-phn2f\" (UID: \"c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f\") " pod="openshift-insights/insights-runtime-extractor-phn2f" Sep 11 03:54:09.357373 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:54:09.357188 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 03:54:09.357373 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:09.357200 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f-crio-socket\") pod \"insights-runtime-extractor-phn2f\" (UID: \"c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f\") " pod="openshift-insights/insights-runtime-extractor-phn2f" Sep 11 03:54:09.357373 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:54:09.357271 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f-kube-rbac-proxy-cm podName:c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f nodeName:}" failed. No retries permitted until 2026-09-11 03:54:09.857252393 +0000 UTC m=+64.610883773 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-phn2f" (UID: "c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f") : configmap "kube-rbac-proxy" not found Sep 11 03:54:09.357373 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:54:09.357189 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 03:54:09.357373 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:09.357334 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f-data-volume\") pod \"insights-runtime-extractor-phn2f\" (UID: \"c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f\") " pod="openshift-insights/insights-runtime-extractor-phn2f" Sep 11 03:54:09.357373 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:54:09.357340 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f-insights-runtime-extractor-tls podName:c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f nodeName:}" failed. No retries permitted until 2026-09-11 03:54:09.857323223 +0000 UTC m=+64.610954616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-phn2f" (UID: "c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f") : secret "insights-runtime-extractor-tls" not found Sep 11 03:54:09.365943 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:09.365914 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x487p\" (UniqueName: \"kubernetes.io/projected/c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f-kube-api-access-x487p\") pod \"insights-runtime-extractor-phn2f\" (UID: \"c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f\") " pod="openshift-insights/insights-runtime-extractor-phn2f" Sep 11 03:54:09.861129 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:09.861095 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-phn2f\" (UID: \"c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f\") " pod="openshift-insights/insights-runtime-extractor-phn2f" Sep 11 03:54:09.861328 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:09.861140 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-phn2f\" (UID: \"c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f\") " pod="openshift-insights/insights-runtime-extractor-phn2f" Sep 11 03:54:09.861328 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:54:09.861245 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 03:54:09.861328 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:54:09.861288 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 03:54:09.861457 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:54:09.861298 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f-kube-rbac-proxy-cm podName:c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f nodeName:}" failed. No retries permitted until 2026-09-11 03:54:10.861285117 +0000 UTC m=+65.614916496 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-phn2f" (UID: "c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f") : configmap "kube-rbac-proxy" not found Sep 11 03:54:09.861457 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:54:09.861364 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f-insights-runtime-extractor-tls podName:c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f nodeName:}" failed. No retries permitted until 2026-09-11 03:54:10.861345738 +0000 UTC m=+65.614977132 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-phn2f" (UID: "c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f") : secret "insights-runtime-extractor-tls" not found Sep 11 03:54:10.047224 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:10.047198 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-5tpdc_9112557d-d0ef-44c7-8b2a-f6ecd0a980d6/console-operator/1.log" Sep 11 03:54:10.047628 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:10.047611 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-5tpdc_9112557d-d0ef-44c7-8b2a-f6ecd0a980d6/console-operator/0.log" Sep 11 03:54:10.047719 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:10.047648 2568 generic.go:356] "Generic (PLEG): container finished" podID="9112557d-d0ef-44c7-8b2a-f6ecd0a980d6" containerID="7b39b5dd7addd53859683437468a35defbd26e16329598cb0d2933823f501103" exitCode=255 Sep 11 03:54:10.047772 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:10.047741 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-5tpdc" event={"ID":"9112557d-d0ef-44c7-8b2a-f6ecd0a980d6","Type":"ContainerDied","Data":"7b39b5dd7addd53859683437468a35defbd26e16329598cb0d2933823f501103"} Sep 11 03:54:10.047823 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:10.047814 2568 scope.go:117] "RemoveContainer" containerID="a674c1dcbe1675a763b71664c40c40662fd6941799ca4168e0c48c5c64470b56" Sep 11 03:54:10.048024 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:10.048002 2568 scope.go:117] "RemoveContainer" containerID="7b39b5dd7addd53859683437468a35defbd26e16329598cb0d2933823f501103" Sep 11 03:54:10.048242 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:54:10.048207 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-5tpdc_openshift-console-operator(9112557d-d0ef-44c7-8b2a-f6ecd0a980d6)\"" pod="openshift-console-operator/console-operator-696884fb9c-5tpdc" podUID="9112557d-d0ef-44c7-8b2a-f6ecd0a980d6" Sep 11 03:54:10.151308 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:10.151274 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:10.167121 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:10.167077 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-z28cw"] Sep 11 03:54:10.201406 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:10.201374 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-z28cw"] Sep 11 03:54:10.201548 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:10.201490 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-z28cw" Sep 11 03:54:10.203777 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:10.203755 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-99gkn\"" Sep 11 03:54:10.203910 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:10.203756 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 11 03:54:10.203910 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:10.203756 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 11 03:54:10.225512 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:10.225482 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:10.268594 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:10.268565 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqnsm\" (UniqueName: \"kubernetes.io/projected/83e53968-6fe7-4c74-a859-61fd72e01de4-kube-api-access-hqnsm\") pod \"migrator-8b4d697c5-z28cw\" (UID: \"83e53968-6fe7-4c74-a859-61fd72e01de4\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-z28cw" Sep 11 03:54:10.287450 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:10.287422 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:10.369722 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:10.369692 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqnsm\" (UniqueName: \"kubernetes.io/projected/83e53968-6fe7-4c74-a859-61fd72e01de4-kube-api-access-hqnsm\") pod \"migrator-8b4d697c5-z28cw\" (UID: \"83e53968-6fe7-4c74-a859-61fd72e01de4\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-z28cw" Sep 11 03:54:10.378298 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:10.378271 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqnsm\" (UniqueName: \"kubernetes.io/projected/83e53968-6fe7-4c74-a859-61fd72e01de4-kube-api-access-hqnsm\") pod \"migrator-8b4d697c5-z28cw\" (UID: \"83e53968-6fe7-4c74-a859-61fd72e01de4\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-z28cw" Sep 11 03:54:10.393320 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:10.393291 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:10.462792 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:10.462730 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:10.511301 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:10.511278 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-z28cw" Sep 11 03:54:10.800576 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:10.800413 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-z28cw"] Sep 11 03:54:10.802866 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:54:10.802839 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83e53968_6fe7_4c74_a859_61fd72e01de4.slice/crio-b93c6e61b6731a920022829f37aaea43e94915cca334e1b575c5056d380b4dc0 WatchSource:0}: Error finding container b93c6e61b6731a920022829f37aaea43e94915cca334e1b575c5056d380b4dc0: Status 404 returned error can't find the container with id b93c6e61b6731a920022829f37aaea43e94915cca334e1b575c5056d380b4dc0 Sep 11 03:54:10.830904 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:10.830858 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:10.874032 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:10.874008 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-phn2f\" (UID: \"c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f\") " pod="openshift-insights/insights-runtime-extractor-phn2f" Sep 11 03:54:10.874128 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:10.874061 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-phn2f\" (UID: \"c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f\") " pod="openshift-insights/insights-runtime-extractor-phn2f" Sep 11 03:54:10.874172 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:54:10.874147 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 03:54:10.874207 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:54:10.874191 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f-kube-rbac-proxy-cm podName:c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f nodeName:}" failed. No retries permitted until 2026-09-11 03:54:12.874177721 +0000 UTC m=+67.627809100 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-phn2f" (UID: "c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f") : configmap "kube-rbac-proxy" not found Sep 11 03:54:10.874258 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:54:10.874148 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 03:54:10.874290 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:54:10.874275 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f-insights-runtime-extractor-tls podName:c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f nodeName:}" failed. No retries permitted until 2026-09-11 03:54:12.87426297 +0000 UTC m=+67.627894354 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-phn2f" (UID: "c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f") : secret "insights-runtime-extractor-tls" not found Sep 11 03:54:10.936677 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:10.936648 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:11.001300 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:11.001277 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7f284_a48e2404-e37c-43d5-8902-b139cb2e07d9/dns-node-resolver/0.log" Sep 11 03:54:11.051232 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:11.051158 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-z28cw" event={"ID":"83e53968-6fe7-4c74-a859-61fd72e01de4","Type":"ContainerStarted","Data":"b93c6e61b6731a920022829f37aaea43e94915cca334e1b575c5056d380b4dc0"} Sep 11 03:54:11.052613 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:11.052594 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-5tpdc_9112557d-d0ef-44c7-8b2a-f6ecd0a980d6/console-operator/1.log" Sep 11 03:54:11.052907 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:11.052884 2568 scope.go:117] "RemoveContainer" containerID="7b39b5dd7addd53859683437468a35defbd26e16329598cb0d2933823f501103" Sep 11 03:54:11.053064 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:54:11.053027 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-5tpdc_openshift-console-operator(9112557d-d0ef-44c7-8b2a-f6ecd0a980d6)\"" pod="openshift-console-operator/console-operator-696884fb9c-5tpdc" podUID="9112557d-d0ef-44c7-8b2a-f6ecd0a980d6" Sep 11 03:54:11.103988 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:11.103962 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:11.159966 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:11.159941 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:11.480235 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:11.480037 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe1486cf-6276-41a8-a685-025064c4abca-metrics-certs\") pod \"network-metrics-daemon-tk8q7\" (UID: \"fe1486cf-6276-41a8-a685-025064c4abca\") " pod="openshift-multus/network-metrics-daemon-tk8q7" Sep 11 03:54:11.482341 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:11.482312 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 03:54:11.490268 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:54:11.490248 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 11 03:54:11.490368 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:54:11.490311 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe1486cf-6276-41a8-a685-025064c4abca-metrics-certs podName:fe1486cf-6276-41a8-a685-025064c4abca nodeName:}" failed. No retries permitted until 2026-09-11 03:55:15.490292705 +0000 UTC m=+130.243924085 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fe1486cf-6276-41a8-a685-025064c4abca-metrics-certs") pod "network-metrics-daemon-tk8q7" (UID: "fe1486cf-6276-41a8-a685-025064c4abca") : secret "metrics-daemon-secret" not found Sep 11 03:54:11.581244 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:11.581203 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnmts\" (UniqueName: \"kubernetes.io/projected/fb9f553c-03c7-4489-b43f-b62d630e7f50-kube-api-access-lnmts\") pod \"network-check-target-vssnm\" (UID: \"fb9f553c-03c7-4489-b43f-b62d630e7f50\") " pod="openshift-network-diagnostics/network-check-target-vssnm" Sep 11 03:54:11.581406 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:11.581271 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9fc48d3a-bf43-44f7-b142-cd4937cea66b-original-pull-secret\") pod \"global-pull-secret-syncer-t5fb2\" (UID: \"9fc48d3a-bf43-44f7-b142-cd4937cea66b\") " pod="kube-system/global-pull-secret-syncer-t5fb2" Sep 11 03:54:11.583661 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:11.583635 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 11 03:54:11.583784 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:11.583667 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 03:54:11.594007 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:11.593979 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 11 03:54:11.594440 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:11.594411 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9fc48d3a-bf43-44f7-b142-cd4937cea66b-original-pull-secret\") pod \"global-pull-secret-syncer-t5fb2\" (UID: \"9fc48d3a-bf43-44f7-b142-cd4937cea66b\") " pod="kube-system/global-pull-secret-syncer-t5fb2" Sep 11 03:54:11.605616 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:11.605542 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnmts\" (UniqueName: \"kubernetes.io/projected/fb9f553c-03c7-4489-b43f-b62d630e7f50-kube-api-access-lnmts\") pod \"network-check-target-vssnm\" (UID: \"fb9f553c-03c7-4489-b43f-b62d630e7f50\") " pod="openshift-network-diagnostics/network-check-target-vssnm" Sep 11 03:54:11.676265 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:11.676232 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t5fb2" Sep 11 03:54:11.777030 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:11.776998 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-j6dl8\"" Sep 11 03:54:11.785287 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:11.785252 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vssnm" Sep 11 03:54:11.802539 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:11.802519 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gmdt4_db235f27-7ffe-4c12-9d60-53fb8b810d97/node-ca/0.log" Sep 11 03:54:11.951091 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:11.951066 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-t5fb2"] Sep 11 03:54:11.954457 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:54:11.954430 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fc48d3a_bf43_44f7_b142_cd4937cea66b.slice/crio-31a7df22937733b25363067b6a9810399c2783a37095145ac4047549b7fc0088 WatchSource:0}: Error finding container 31a7df22937733b25363067b6a9810399c2783a37095145ac4047549b7fc0088: Status 404 returned error can't find the container with id 31a7df22937733b25363067b6a9810399c2783a37095145ac4047549b7fc0088 Sep 11 03:54:11.984539 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:11.984472 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/52394690-486c-4ae0-b36b-603b438a944a-registry-tls\") pod \"image-registry-6c8dd7548c-jmns9\" (UID: \"52394690-486c-4ae0-b36b-603b438a944a\") " pod="openshift-image-registry/image-registry-6c8dd7548c-jmns9" Sep 11 03:54:11.984539 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:11.984518 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/11fca7d2-d884-4012-9431-b3c085b8339c-metrics-tls\") pod \"dns-default-ssrwd\" (UID: \"11fca7d2-d884-4012-9431-b3c085b8339c\") " pod="openshift-dns/dns-default-ssrwd" Sep 11 03:54:11.984716 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:54:11.984665 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 03:54:11.984716 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:54:11.984689 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c8dd7548c-jmns9: secret "image-registry-tls" not found Sep 11 03:54:11.984840 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:54:11.984741 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 03:54:11.984840 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:54:11.984771 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/52394690-486c-4ae0-b36b-603b438a944a-registry-tls podName:52394690-486c-4ae0-b36b-603b438a944a nodeName:}" failed. No retries permitted until 2026-09-11 03:54:43.984749392 +0000 UTC m=+98.738380787 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/52394690-486c-4ae0-b36b-603b438a944a-registry-tls") pod "image-registry-6c8dd7548c-jmns9" (UID: "52394690-486c-4ae0-b36b-603b438a944a") : secret "image-registry-tls" not found Sep 11 03:54:11.984840 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:54:11.984793 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/11fca7d2-d884-4012-9431-b3c085b8339c-metrics-tls podName:11fca7d2-d884-4012-9431-b3c085b8339c nodeName:}" failed. No retries permitted until 2026-09-11 03:54:43.98477847 +0000 UTC m=+98.738409855 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/11fca7d2-d884-4012-9431-b3c085b8339c-metrics-tls") pod "dns-default-ssrwd" (UID: "11fca7d2-d884-4012-9431-b3c085b8339c") : secret "dns-default-metrics-tls" not found Sep 11 03:54:12.058007 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:12.057975 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-t5fb2" event={"ID":"9fc48d3a-bf43-44f7-b142-cd4937cea66b","Type":"ContainerStarted","Data":"31a7df22937733b25363067b6a9810399c2783a37095145ac4047549b7fc0088"} Sep 11 03:54:12.059430 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:12.059408 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-z28cw" event={"ID":"83e53968-6fe7-4c74-a859-61fd72e01de4","Type":"ContainerStarted","Data":"65d2cb4527ec9439f5a8f47d0d462473e202033a3b662b2c4418202890c436b9"} Sep 11 03:54:12.211551 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:12.211522 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vssnm"] Sep 11 03:54:12.214217 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:54:12.214191 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb9f553c_03c7_4489_b43f_b62d630e7f50.slice/crio-ab52b7f795ab640f54333ed76487604035b1597dec4a58c30678dbb9574618ef WatchSource:0}: Error finding container ab52b7f795ab640f54333ed76487604035b1597dec4a58c30678dbb9574618ef: Status 404 returned error can't find the container with id ab52b7f795ab640f54333ed76487604035b1597dec4a58c30678dbb9574618ef Sep 11 03:54:12.388134 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:12.388062 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d3a584c-2a1b-4c64-9fd6-89d7527d6c87-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-qdnbj\" (UID: \"7d3a584c-2a1b-4c64-9fd6-89d7527d6c87\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qdnbj" Sep 11 03:54:12.388243 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:54:12.388209 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 03:54:12.388278 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:54:12.388264 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d3a584c-2a1b-4c64-9fd6-89d7527d6c87-samples-operator-tls podName:7d3a584c-2a1b-4c64-9fd6-89d7527d6c87 nodeName:}" failed. No retries permitted until 2026-09-11 03:54:20.388248074 +0000 UTC m=+75.141879458 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7d3a584c-2a1b-4c64-9fd6-89d7527d6c87-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-qdnbj" (UID: "7d3a584c-2a1b-4c64-9fd6-89d7527d6c87") : secret "samples-operator-tls" not found Sep 11 03:54:12.401338 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:12.401315 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nq6qk_badea905-61e6-47e1-8f68-505d04824821/serve-healthcheck-canary/0.log" Sep 11 03:54:12.696318 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:12.696281 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-2wllz"] Sep 11 03:54:12.699553 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:12.699530 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-2wllz" Sep 11 03:54:12.701954 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:12.701930 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Sep 11 03:54:12.702832 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:12.702813 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Sep 11 03:54:12.702832 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:12.702827 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-54s54\"" Sep 11 03:54:12.703278 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:12.703099 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Sep 11 03:54:12.703278 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:12.703111 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Sep 11 03:54:12.707919 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:12.707864 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-2wllz"] Sep 11 03:54:12.725855 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:12.725824 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:12.786479 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:12.786437 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:12.791401 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:12.791370 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4291adc4-b84a-4678-ba35-cd426bfb8892-signing-key\") pod \"service-ca-568869fdc5-2wllz\" (UID: \"4291adc4-b84a-4678-ba35-cd426bfb8892\") " pod="openshift-service-ca/service-ca-568869fdc5-2wllz" Sep 11 03:54:12.791536 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:12.791514 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4291adc4-b84a-4678-ba35-cd426bfb8892-signing-cabundle\") pod \"service-ca-568869fdc5-2wllz\" (UID: \"4291adc4-b84a-4678-ba35-cd426bfb8892\") " pod="openshift-service-ca/service-ca-568869fdc5-2wllz" Sep 11 03:54:12.791590 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:12.791556 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jmxc\" (UniqueName: \"kubernetes.io/projected/4291adc4-b84a-4678-ba35-cd426bfb8892-kube-api-access-6jmxc\") pod \"service-ca-568869fdc5-2wllz\" (UID: \"4291adc4-b84a-4678-ba35-cd426bfb8892\") " pod="openshift-service-ca/service-ca-568869fdc5-2wllz" Sep 11 03:54:12.892038 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:12.892003 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4291adc4-b84a-4678-ba35-cd426bfb8892-signing-cabundle\") pod \"service-ca-568869fdc5-2wllz\" (UID: \"4291adc4-b84a-4678-ba35-cd426bfb8892\") " pod="openshift-service-ca/service-ca-568869fdc5-2wllz" Sep 11 03:54:12.892218 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:12.892063 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jmxc\" (UniqueName: \"kubernetes.io/projected/4291adc4-b84a-4678-ba35-cd426bfb8892-kube-api-access-6jmxc\") pod \"service-ca-568869fdc5-2wllz\" (UID: \"4291adc4-b84a-4678-ba35-cd426bfb8892\") " pod="openshift-service-ca/service-ca-568869fdc5-2wllz" Sep 11 03:54:12.892218 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:12.892144 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-phn2f\" (UID: \"c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f\") " pod="openshift-insights/insights-runtime-extractor-phn2f" Sep 11 03:54:12.892218 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:12.892178 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4291adc4-b84a-4678-ba35-cd426bfb8892-signing-key\") pod \"service-ca-568869fdc5-2wllz\" (UID: \"4291adc4-b84a-4678-ba35-cd426bfb8892\") " pod="openshift-service-ca/service-ca-568869fdc5-2wllz" Sep 11 03:54:12.892218 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:12.892200 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-phn2f\" (UID: \"c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f\") " pod="openshift-insights/insights-runtime-extractor-phn2f" Sep 11 03:54:12.892433 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:54:12.892313 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 03:54:12.892433 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:54:12.892374 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f-kube-rbac-proxy-cm podName:c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f nodeName:}" failed. No retries permitted until 2026-09-11 03:54:16.89235306 +0000 UTC m=+71.645984442 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-phn2f" (UID: "c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f") : configmap "kube-rbac-proxy" not found Sep 11 03:54:12.893156 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:12.893131 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4291adc4-b84a-4678-ba35-cd426bfb8892-signing-cabundle\") pod \"service-ca-568869fdc5-2wllz\" (UID: \"4291adc4-b84a-4678-ba35-cd426bfb8892\") " pod="openshift-service-ca/service-ca-568869fdc5-2wllz" Sep 11 03:54:12.893255 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:54:12.893163 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 03:54:12.893255 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:54:12.893227 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f-insights-runtime-extractor-tls podName:c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f nodeName:}" failed. No retries permitted until 2026-09-11 03:54:16.893208965 +0000 UTC m=+71.646840363 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-phn2f" (UID: "c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f") : secret "insights-runtime-extractor-tls" not found Sep 11 03:54:12.896637 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:12.896613 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4291adc4-b84a-4678-ba35-cd426bfb8892-signing-key\") pod \"service-ca-568869fdc5-2wllz\" (UID: \"4291adc4-b84a-4678-ba35-cd426bfb8892\") " pod="openshift-service-ca/service-ca-568869fdc5-2wllz" Sep 11 03:54:12.906415 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:12.906388 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jmxc\" (UniqueName: \"kubernetes.io/projected/4291adc4-b84a-4678-ba35-cd426bfb8892-kube-api-access-6jmxc\") pod \"service-ca-568869fdc5-2wllz\" (UID: \"4291adc4-b84a-4678-ba35-cd426bfb8892\") " pod="openshift-service-ca/service-ca-568869fdc5-2wllz" Sep 11 03:54:13.013428 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:13.013345 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-2wllz" Sep 11 03:54:13.067580 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:13.067498 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-z28cw" event={"ID":"83e53968-6fe7-4c74-a859-61fd72e01de4","Type":"ContainerStarted","Data":"b49a938191280e4c452ac338b2d34aa6f67358015ea01acd9417f0b12e18d559"} Sep 11 03:54:13.069090 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:13.069038 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vssnm" event={"ID":"fb9f553c-03c7-4489-b43f-b62d630e7f50","Type":"ContainerStarted","Data":"ab52b7f795ab640f54333ed76487604035b1597dec4a58c30678dbb9574618ef"} Sep 11 03:54:13.084007 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:13.083951 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-z28cw" podStartSLOduration=1.89089252 podStartE2EDuration="3.083930042s" totalImagesPullingTime="1.193037522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:54:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:13.0834367 +0000 UTC m=+67.837068114" watchObservedRunningTime="2026-09-11 03:54:13.083930042 +0000 UTC m=+67.837561445" Sep 11 03:54:13.163401 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:13.163352 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:13.264309 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:13.264222 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:13.298064 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:13.297872 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-2wllz"] Sep 11 03:54:13.301018 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:54:13.300990 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4291adc4_b84a_4678_ba35_cd426bfb8892.slice/crio-47659bbf1cda3453882a428ceec7dcc8850b002a6559f3de93581b94394cac22 WatchSource:0}: Error finding container 47659bbf1cda3453882a428ceec7dcc8850b002a6559f3de93581b94394cac22: Status 404 returned error can't find the container with id 47659bbf1cda3453882a428ceec7dcc8850b002a6559f3de93581b94394cac22 Sep 11 03:54:13.659417 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:13.659195 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:14.075279 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:14.075234 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-2wllz" event={"ID":"4291adc4-b84a-4678-ba35-cd426bfb8892","Type":"ContainerStarted","Data":"47659bbf1cda3453882a428ceec7dcc8850b002a6559f3de93581b94394cac22"} Sep 11 03:54:14.922832 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:14.922787 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-5tpdc" Sep 11 03:54:14.923013 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:14.922848 2568 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-5tpdc" Sep 11 03:54:14.923238 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:14.923222 2568 scope.go:117] "RemoveContainer" containerID="7b39b5dd7addd53859683437468a35defbd26e16329598cb0d2933823f501103" Sep 11 03:54:14.923402 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:54:14.923381 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-5tpdc_openshift-console-operator(9112557d-d0ef-44c7-8b2a-f6ecd0a980d6)\"" pod="openshift-console-operator/console-operator-696884fb9c-5tpdc" podUID="9112557d-d0ef-44c7-8b2a-f6ecd0a980d6" Sep 11 03:54:16.154619 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:16.154569 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:16.930167 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:16.930128 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-phn2f\" (UID: \"c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f\") " pod="openshift-insights/insights-runtime-extractor-phn2f" Sep 11 03:54:16.930330 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:54:16.930242 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 03:54:16.930330 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:16.930252 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-phn2f\" (UID: \"c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f\") " pod="openshift-insights/insights-runtime-extractor-phn2f" Sep 11 03:54:16.930330 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:54:16.930302 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f-kube-rbac-proxy-cm podName:c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f nodeName:}" failed. No retries permitted until 2026-09-11 03:54:24.930287008 +0000 UTC m=+79.683918387 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-phn2f" (UID: "c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f") : configmap "kube-rbac-proxy" not found Sep 11 03:54:16.930468 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:54:16.930367 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 03:54:16.930468 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:54:16.930424 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f-insights-runtime-extractor-tls podName:c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f nodeName:}" failed. No retries permitted until 2026-09-11 03:54:24.930409456 +0000 UTC m=+79.684040837 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-phn2f" (UID: "c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f") : secret "insights-runtime-extractor-tls" not found Sep 11 03:54:18.091408 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:18.091372 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-t5fb2" event={"ID":"9fc48d3a-bf43-44f7-b142-cd4937cea66b","Type":"ContainerStarted","Data":"e21de8582d2242683174f497a70b42a97d691df8741f67367f5df41118d16a7c"} Sep 11 03:54:18.092928 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:18.092903 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vssnm" event={"ID":"fb9f553c-03c7-4489-b43f-b62d630e7f50","Type":"ContainerStarted","Data":"d999c7fc117a568b95d4741511ee05ecaba51fd929b0598a5a6f4fc2cfe15b3a"} Sep 11 03:54:18.093055 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:18.092973 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vssnm" Sep 11 03:54:18.094420 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:18.094398 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-2wllz" event={"ID":"4291adc4-b84a-4678-ba35-cd426bfb8892","Type":"ContainerStarted","Data":"db04eb73c02d39c29e8ef3550ff8717770373de51cf01dff5efb0ab50372f141"} Sep 11 03:54:18.105713 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:18.105661 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-t5fb2" podStartSLOduration=66.723669607 podStartE2EDuration="1m12.105644382s" totalImagesPullingTime="5.381974775s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:53:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:18.104462068 +0000 UTC m=+72.858093466" watchObservedRunningTime="2026-09-11 03:54:18.105644382 +0000 UTC m=+72.859275779" Sep 11 03:54:18.120505 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:18.120465 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vssnm" podStartSLOduration=67.01091395 podStartE2EDuration="1m12.12045258s" totalImagesPullingTime="5.10953863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:53:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:18.119527656 +0000 UTC m=+72.873159052" watchObservedRunningTime="2026-09-11 03:54:18.12045258 +0000 UTC m=+72.874084007" Sep 11 03:54:18.133452 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:18.133401 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-2wllz" podStartSLOduration=2.110613001 podStartE2EDuration="6.133383886s" totalImagesPullingTime="4.022770885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:54:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:18.132859883 +0000 UTC m=+72.886491291" watchObservedRunningTime="2026-09-11 03:54:18.133383886 +0000 UTC m=+72.887015291" Sep 11 03:54:18.322844 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:18.322809 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:18.464489 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:18.464440 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:18.545307 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:18.545269 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:18.630360 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:18.630325 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:18.731850 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:18.731755 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:19.420619 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:19.420580 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:19.797103 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:19.797009 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:19.854272 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:19.854234 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:20.223322 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:20.223288 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:20.460626 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:20.460590 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d3a584c-2a1b-4c64-9fd6-89d7527d6c87-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-qdnbj\" (UID: \"7d3a584c-2a1b-4c64-9fd6-89d7527d6c87\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qdnbj" Sep 11 03:54:20.463160 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:20.463136 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d3a584c-2a1b-4c64-9fd6-89d7527d6c87-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-qdnbj\" (UID: \"7d3a584c-2a1b-4c64-9fd6-89d7527d6c87\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qdnbj" Sep 11 03:54:20.591012 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:20.590934 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-6pp4m\"" Sep 11 03:54:20.598909 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:20.598890 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qdnbj" Sep 11 03:54:20.623828 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:20.623783 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:20.869489 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:20.869412 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qdnbj"] Sep 11 03:54:21.019812 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:21.019777 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:21.106957 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:21.106924 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qdnbj" event={"ID":"7d3a584c-2a1b-4c64-9fd6-89d7527d6c87","Type":"ContainerStarted","Data":"98e649dc31ee56dca0cdc94f3e7d6e20530f12871c81bbc4381e3c1c8f4013a5"} Sep 11 03:54:21.418589 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:21.418537 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:21.615848 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:21.615812 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:22.015138 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:22.015105 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:22.072820 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:22.072741 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:22.366155 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:22.366068 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:22.474207 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:22.474170 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:22.846248 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:22.846215 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:23.031255 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:23.031219 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:23.115890 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:23.115852 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qdnbj" event={"ID":"7d3a584c-2a1b-4c64-9fd6-89d7527d6c87","Type":"ContainerStarted","Data":"2957dfb3c3627800ca5b2ef3014e91bf238f37f70d529aa38b95424e84bcc56e"} Sep 11 03:54:23.116086 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:23.115907 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qdnbj" event={"ID":"7d3a584c-2a1b-4c64-9fd6-89d7527d6c87","Type":"ContainerStarted","Data":"7d272d9ee742537937f24d0f0b665eea1082faae756baa76bac4dd75836a89ff"} Sep 11 03:54:23.132299 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:23.132265 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:23.135863 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:23.135817 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-qdnbj" podStartSLOduration=17.475691951 podStartE2EDuration="19.135775743s" totalImagesPullingTime="1.660083792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:54:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:23.134987989 +0000 UTC m=+77.888619394" watchObservedRunningTime="2026-09-11 03:54:23.135775743 +0000 UTC m=+77.889407144" Sep 11 03:54:23.422453 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:23.422425 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:23.622822 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:23.622775 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:24.027037 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:24.027001 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:24.526012 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:24.525973 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:24.626902 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:24.626872 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:24.750496 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:24.750461 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:24.851372 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:24.851282 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:24.996471 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:24.996431 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-phn2f\" (UID: \"c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f\") " pod="openshift-insights/insights-runtime-extractor-phn2f" Sep 11 03:54:24.996650 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:24.996489 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-phn2f\" (UID: \"c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f\") " pod="openshift-insights/insights-runtime-extractor-phn2f" Sep 11 03:54:24.996650 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:54:24.996575 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 03:54:24.996753 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:54:24.996652 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f-insights-runtime-extractor-tls podName:c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f nodeName:}" failed. No retries permitted until 2026-09-11 03:54:40.996632591 +0000 UTC m=+95.750263970 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-phn2f" (UID: "c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f") : secret "insights-runtime-extractor-tls" not found Sep 11 03:54:24.997082 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:24.997066 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-phn2f\" (UID: \"c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f\") " pod="openshift-insights/insights-runtime-extractor-phn2f" Sep 11 03:54:25.036145 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:25.036102 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:25.427861 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:25.427813 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:25.624790 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:25.624733 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:26.217561 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:26.217509 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:26.832531 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:26.832489 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:27.021659 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:27.021613 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:27.221801 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:27.221765 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:27.416562 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:27.416531 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:28.447033 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:28.446910 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:28.617475 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:28.617427 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:28.762662 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:28.762594 2568 scope.go:117] "RemoveContainer" containerID="7b39b5dd7addd53859683437468a35defbd26e16329598cb0d2933823f501103" Sep 11 03:54:28.820238 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:28.820208 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:29.021133 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:29.021025 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:29.137512 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:29.137482 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-5tpdc_9112557d-d0ef-44c7-8b2a-f6ecd0a980d6/console-operator/1.log" Sep 11 03:54:29.137687 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:29.137542 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-5tpdc" event={"ID":"9112557d-d0ef-44c7-8b2a-f6ecd0a980d6","Type":"ContainerStarted","Data":"42e92b88098e4262d4d46fdd347c5450695ff14c30a015c29f836d1a476662d4"} Sep 11 03:54:29.137867 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:29.137845 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-5tpdc" Sep 11 03:54:29.155377 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:29.155333 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-5tpdc" podStartSLOduration=21.934783937 podStartE2EDuration="25.155319034s" totalImagesPullingTime="3.220535097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:54:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:29.154660586 +0000 UTC m=+83.908291989" watchObservedRunningTime="2026-09-11 03:54:29.155319034 +0000 UTC m=+83.908950436" Sep 11 03:54:29.262656 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:29.262627 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-5tpdc" Sep 11 03:54:29.439770 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:29.439729 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:29.539464 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:29.539434 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:29.738845 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:29.738762 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:30.115620 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:30.115540 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:30.216094 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:30.216042 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:30.320846 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:30.320815 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:30.423358 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:30.423331 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:30.726916 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:30.726831 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:30.827988 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:30.827964 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:31.326990 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:31.326953 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:31.530289 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:31.530260 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:31.663511 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:31.663479 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:31.791807 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:31.791773 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:31.865219 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:31.865163 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:34.434870 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:34.434838 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768947bcf7-z4kzv"] Sep 11 03:54:34.437889 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:34.437874 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768947bcf7-z4kzv" Sep 11 03:54:34.442215 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:34.442188 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 11 03:54:34.442760 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:34.442738 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Sep 11 03:54:34.443214 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:34.443199 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Sep 11 03:54:34.443317 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:34.443213 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Sep 11 03:54:34.443317 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:34.443271 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Sep 11 03:54:34.443438 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:34.443312 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 11 03:54:34.443604 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:34.443589 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 11 03:54:34.457765 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:34.457740 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768947bcf7-z4kzv"] Sep 11 03:54:34.575640 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:34.575605 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/04516d79-a3ee-43c4-8d90-54682169a99d-ca\") pod \"cluster-proxy-proxy-agent-768947bcf7-z4kzv\" (UID: \"04516d79-a3ee-43c4-8d90-54682169a99d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768947bcf7-z4kzv" Sep 11 03:54:34.575640 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:34.575642 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/04516d79-a3ee-43c4-8d90-54682169a99d-hub\") pod \"cluster-proxy-proxy-agent-768947bcf7-z4kzv\" (UID: \"04516d79-a3ee-43c4-8d90-54682169a99d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768947bcf7-z4kzv" Sep 11 03:54:34.575863 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:34.575741 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/04516d79-a3ee-43c4-8d90-54682169a99d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-768947bcf7-z4kzv\" (UID: \"04516d79-a3ee-43c4-8d90-54682169a99d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768947bcf7-z4kzv" Sep 11 03:54:34.575863 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:34.575764 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qwcc\" (UniqueName: \"kubernetes.io/projected/04516d79-a3ee-43c4-8d90-54682169a99d-kube-api-access-8qwcc\") pod \"cluster-proxy-proxy-agent-768947bcf7-z4kzv\" (UID: \"04516d79-a3ee-43c4-8d90-54682169a99d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768947bcf7-z4kzv" Sep 11 03:54:34.575863 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:34.575791 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/04516d79-a3ee-43c4-8d90-54682169a99d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-768947bcf7-z4kzv\" (UID: \"04516d79-a3ee-43c4-8d90-54682169a99d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768947bcf7-z4kzv" Sep 11 03:54:34.575863 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:34.575835 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/04516d79-a3ee-43c4-8d90-54682169a99d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-768947bcf7-z4kzv\" (UID: \"04516d79-a3ee-43c4-8d90-54682169a99d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768947bcf7-z4kzv" Sep 11 03:54:34.578797 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:34.578760 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:34.673974 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:34.673935 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:34.676940 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:34.676921 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/04516d79-a3ee-43c4-8d90-54682169a99d-ca\") pod \"cluster-proxy-proxy-agent-768947bcf7-z4kzv\" (UID: \"04516d79-a3ee-43c4-8d90-54682169a99d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768947bcf7-z4kzv" Sep 11 03:54:34.677023 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:34.676952 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/04516d79-a3ee-43c4-8d90-54682169a99d-hub\") pod \"cluster-proxy-proxy-agent-768947bcf7-z4kzv\" (UID: \"04516d79-a3ee-43c4-8d90-54682169a99d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768947bcf7-z4kzv" Sep 11 03:54:34.677023 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:34.677008 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/04516d79-a3ee-43c4-8d90-54682169a99d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-768947bcf7-z4kzv\" (UID: \"04516d79-a3ee-43c4-8d90-54682169a99d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768947bcf7-z4kzv" Sep 11 03:54:34.677118 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:34.677026 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qwcc\" (UniqueName: \"kubernetes.io/projected/04516d79-a3ee-43c4-8d90-54682169a99d-kube-api-access-8qwcc\") pod \"cluster-proxy-proxy-agent-768947bcf7-z4kzv\" (UID: \"04516d79-a3ee-43c4-8d90-54682169a99d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768947bcf7-z4kzv" Sep 11 03:54:34.677118 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:34.677091 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/04516d79-a3ee-43c4-8d90-54682169a99d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-768947bcf7-z4kzv\" (UID: \"04516d79-a3ee-43c4-8d90-54682169a99d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768947bcf7-z4kzv" Sep 11 03:54:34.677190 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:34.677149 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/04516d79-a3ee-43c4-8d90-54682169a99d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-768947bcf7-z4kzv\" (UID: \"04516d79-a3ee-43c4-8d90-54682169a99d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768947bcf7-z4kzv" Sep 11 03:54:34.677900 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:34.677868 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/04516d79-a3ee-43c4-8d90-54682169a99d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-768947bcf7-z4kzv\" (UID: \"04516d79-a3ee-43c4-8d90-54682169a99d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768947bcf7-z4kzv" Sep 11 03:54:34.679593 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:34.679572 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/04516d79-a3ee-43c4-8d90-54682169a99d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-768947bcf7-z4kzv\" (UID: \"04516d79-a3ee-43c4-8d90-54682169a99d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768947bcf7-z4kzv" Sep 11 03:54:34.679593 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:34.679584 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/04516d79-a3ee-43c4-8d90-54682169a99d-hub\") pod \"cluster-proxy-proxy-agent-768947bcf7-z4kzv\" (UID: \"04516d79-a3ee-43c4-8d90-54682169a99d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768947bcf7-z4kzv" Sep 11 03:54:34.679696 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:34.679641 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/04516d79-a3ee-43c4-8d90-54682169a99d-ca\") pod \"cluster-proxy-proxy-agent-768947bcf7-z4kzv\" (UID: \"04516d79-a3ee-43c4-8d90-54682169a99d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768947bcf7-z4kzv" Sep 11 03:54:34.679696 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:34.679641 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/04516d79-a3ee-43c4-8d90-54682169a99d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-768947bcf7-z4kzv\" (UID: \"04516d79-a3ee-43c4-8d90-54682169a99d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768947bcf7-z4kzv" Sep 11 03:54:34.685945 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:34.685892 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qwcc\" (UniqueName: \"kubernetes.io/projected/04516d79-a3ee-43c4-8d90-54682169a99d-kube-api-access-8qwcc\") pod \"cluster-proxy-proxy-agent-768947bcf7-z4kzv\" (UID: \"04516d79-a3ee-43c4-8d90-54682169a99d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768947bcf7-z4kzv" Sep 11 03:54:34.765063 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:34.765012 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768947bcf7-z4kzv" Sep 11 03:54:34.774752 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:34.774728 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:34.881525 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:34.881490 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:34.977109 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:34.976996 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:35.038593 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:35.038552 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768947bcf7-z4kzv"] Sep 11 03:54:35.041540 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:54:35.041513 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04516d79_a3ee_43c4_8d90_54682169a99d.slice/crio-97a1919659b5c3cb9f6ba7932d272d0325fa4d05546c4fbb5e8247e4c8c95d6c WatchSource:0}: Error finding container 97a1919659b5c3cb9f6ba7932d272d0325fa4d05546c4fbb5e8247e4c8c95d6c: Status 404 returned error can't find the container with id 97a1919659b5c3cb9f6ba7932d272d0325fa4d05546c4fbb5e8247e4c8c95d6c Sep 11 03:54:35.159131 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:35.159099 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768947bcf7-z4kzv" event={"ID":"04516d79-a3ee-43c4-8d90-54682169a99d","Type":"ContainerStarted","Data":"97a1919659b5c3cb9f6ba7932d272d0325fa4d05546c4fbb5e8247e4c8c95d6c"} Sep 11 03:54:35.826852 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:35.826820 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:35.980534 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:35.980496 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:36.785627 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:36.785590 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:36.956148 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:36.956075 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:39.177927 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:39.177889 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768947bcf7-z4kzv" event={"ID":"04516d79-a3ee-43c4-8d90-54682169a99d","Type":"ContainerStarted","Data":"2d79a38827d80193161f7c83134b23515e731af8a3c92066a59b7629bf2d9eac"} Sep 11 03:54:40.183214 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:40.183182 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768947bcf7-z4kzv" event={"ID":"04516d79-a3ee-43c4-8d90-54682169a99d","Type":"ContainerStarted","Data":"27815f5d53bfb9bf8a86bab6b899a08163105320bf27ce6a31dcc93ea53ffc96"} Sep 11 03:54:40.183214 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:40.183222 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768947bcf7-z4kzv" event={"ID":"04516d79-a3ee-43c4-8d90-54682169a99d","Type":"ContainerStarted","Data":"16c3f3df136e837ff549331b5c82c2a8a0ee0fa7623f5e51045840d7bc143681"} Sep 11 03:54:40.201822 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:40.201767 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-768947bcf7-z4kzv" podStartSLOduration=2.18407187 podStartE2EDuration="6.201751725s" totalImagesPullingTime="4.017679855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:54:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:40.199974668 +0000 UTC m=+94.953606068" watchObservedRunningTime="2026-09-11 03:54:40.201751725 +0000 UTC m=+94.955383128" Sep 11 03:54:41.025413 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:41.025364 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-phn2f\" (UID: \"c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f\") " pod="openshift-insights/insights-runtime-extractor-phn2f" Sep 11 03:54:41.027851 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:41.027824 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-phn2f\" (UID: \"c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f\") " pod="openshift-insights/insights-runtime-extractor-phn2f" Sep 11 03:54:41.051923 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:41.051886 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-phn2f" Sep 11 03:54:41.321571 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:41.321413 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-phn2f"] Sep 11 03:54:41.324124 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:54:41.324097 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6f6c1ac_998f_40f9_a13a_8da25c5e6c0f.slice/crio-da5b114272d99726668c34fcd5e6085a16295c3740a5d3039cfe56287bc82853 WatchSource:0}: Error finding container da5b114272d99726668c34fcd5e6085a16295c3740a5d3039cfe56287bc82853: Status 404 returned error can't find the container with id da5b114272d99726668c34fcd5e6085a16295c3740a5d3039cfe56287bc82853 Sep 11 03:54:41.756699 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:41.756666 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:42.191041 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:42.191002 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-phn2f" event={"ID":"c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f","Type":"ContainerStarted","Data":"78eb9e5045e36fda52056015b63bec8a684c387681e34fe734e92bb0fbfbeec8"} Sep 11 03:54:42.191280 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:42.191094 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-phn2f" event={"ID":"c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f","Type":"ContainerStarted","Data":"13699f316a598f57a1331134f90c01233f1f9d3ce33c2cc225d4a415d0e25e99"} Sep 11 03:54:42.191280 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:42.191122 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-phn2f" event={"ID":"c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f","Type":"ContainerStarted","Data":"da5b114272d99726668c34fcd5e6085a16295c3740a5d3039cfe56287bc82853"} Sep 11 03:54:42.289415 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:42.289386 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:42.390039 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:42.390002 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:42.457339 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:42.457204 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:42.661994 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:42.661955 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:43.415763 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:43.415733 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:43.522536 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:43.522507 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:44.049850 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:44.049816 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/52394690-486c-4ae0-b36b-603b438a944a-registry-tls\") pod \"image-registry-6c8dd7548c-jmns9\" (UID: \"52394690-486c-4ae0-b36b-603b438a944a\") " pod="openshift-image-registry/image-registry-6c8dd7548c-jmns9" Sep 11 03:54:44.049850 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:44.049854 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/11fca7d2-d884-4012-9431-b3c085b8339c-metrics-tls\") pod \"dns-default-ssrwd\" (UID: \"11fca7d2-d884-4012-9431-b3c085b8339c\") " pod="openshift-dns/dns-default-ssrwd" Sep 11 03:54:44.052535 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:44.052509 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/11fca7d2-d884-4012-9431-b3c085b8339c-metrics-tls\") pod \"dns-default-ssrwd\" (UID: \"11fca7d2-d884-4012-9431-b3c085b8339c\") " pod="openshift-dns/dns-default-ssrwd" Sep 11 03:54:44.052788 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:44.052768 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/52394690-486c-4ae0-b36b-603b438a944a-registry-tls\") pod \"image-registry-6c8dd7548c-jmns9\" (UID: \"52394690-486c-4ae0-b36b-603b438a944a\") " pod="openshift-image-registry/image-registry-6c8dd7548c-jmns9" Sep 11 03:54:44.200032 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:44.199992 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-phn2f" event={"ID":"c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f","Type":"ContainerStarted","Data":"57b9273f3ad12583184c4494778346c25b4021e32435438da2901f7b3df6fcee"} Sep 11 03:54:44.243972 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:44.243923 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-phn2f" podStartSLOduration=33.371244135 podStartE2EDuration="35.243907513s" totalImagesPullingTime="1.872663378s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 03:54:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:44.241564491 +0000 UTC m=+98.995195894" watchObservedRunningTime="2026-09-11 03:54:44.243907513 +0000 UTC m=+98.997538915" Sep 11 03:54:44.297303 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:44.297272 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vm2xm\"" Sep 11 03:54:44.300632 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:44.300576 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c8dd7548c-jmns9" Sep 11 03:54:44.314441 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:44.314416 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h5zgd\"" Sep 11 03:54:44.320970 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:44.320949 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ssrwd" Sep 11 03:54:44.586716 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:44.586636 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c8dd7548c-jmns9"] Sep 11 03:54:44.589903 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:54:44.589877 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52394690_486c_4ae0_b36b_603b438a944a.slice/crio-076c0369aa41514e2a388d19f77809b73e1e07be75be7da77e044e9491b0125e WatchSource:0}: Error finding container 076c0369aa41514e2a388d19f77809b73e1e07be75be7da77e044e9491b0125e: Status 404 returned error can't find the container with id 076c0369aa41514e2a388d19f77809b73e1e07be75be7da77e044e9491b0125e Sep 11 03:54:44.606288 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:44.606264 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ssrwd"] Sep 11 03:54:44.609493 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:54:44.609462 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11fca7d2_d884_4012_9431_b3c085b8339c.slice/crio-95d6607ac853e893d239eaf4bd23256928ad66bc0fb30652db3c94c31355c9f4 WatchSource:0}: Error finding container 95d6607ac853e893d239eaf4bd23256928ad66bc0fb30652db3c94c31355c9f4: Status 404 returned error can't find the container with id 95d6607ac853e893d239eaf4bd23256928ad66bc0fb30652db3c94c31355c9f4 Sep 11 03:54:44.998407 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:44.998371 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:45.080965 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:45.080916 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:45.206145 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:45.206106 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c8dd7548c-jmns9" event={"ID":"52394690-486c-4ae0-b36b-603b438a944a","Type":"ContainerStarted","Data":"9dc520ca64ec9550eb3e6371e7482920d3de8721aa06fda44e735ff8eaef6b6e"} Sep 11 03:54:45.206330 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:45.206264 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6c8dd7548c-jmns9" Sep 11 03:54:45.206330 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:45.206304 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c8dd7548c-jmns9" event={"ID":"52394690-486c-4ae0-b36b-603b438a944a","Type":"ContainerStarted","Data":"076c0369aa41514e2a388d19f77809b73e1e07be75be7da77e044e9491b0125e"} Sep 11 03:54:45.207393 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:45.207353 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ssrwd" event={"ID":"11fca7d2-d884-4012-9431-b3c085b8339c","Type":"ContainerStarted","Data":"95d6607ac853e893d239eaf4bd23256928ad66bc0fb30652db3c94c31355c9f4"} Sep 11 03:54:45.229754 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:45.229706 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6c8dd7548c-jmns9" podStartSLOduration=99.229690352 podStartE2EDuration="1m39.229690352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:53:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:45.228721232 +0000 UTC m=+99.982352647" watchObservedRunningTime="2026-09-11 03:54:45.229690352 +0000 UTC m=+99.983321753" Sep 11 03:54:45.544638 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:45.544601 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:45.659491 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:45.659458 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:45.757068 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:45.756999 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:45.825029 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:45.824922 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:47.215325 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:47.215287 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ssrwd" event={"ID":"11fca7d2-d884-4012-9431-b3c085b8339c","Type":"ContainerStarted","Data":"7b7139d064f4eed24940ed4628e20089cedcc4f00694c45475ad8326b5ea237e"} Sep 11 03:54:47.215712 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:47.215340 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ssrwd" event={"ID":"11fca7d2-d884-4012-9431-b3c085b8339c","Type":"ContainerStarted","Data":"72e88ae8a18d1dd519cd64af9a7675180c49b1e00e1725203d1beea681eee23e"} Sep 11 03:54:47.215712 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:47.215429 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ssrwd" Sep 11 03:54:47.232991 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:47.232941 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-ssrwd" podStartSLOduration=65.327700482 podStartE2EDuration="1m7.232926938s" totalImagesPullingTime="1.905226456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:53:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:47.232020671 +0000 UTC m=+101.985652084" watchObservedRunningTime="2026-09-11 03:54:47.232926938 +0000 UTC m=+101.986558354" Sep 11 03:54:47.929323 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:47.929284 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:48.144394 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:48.144351 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:49.103690 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:49.103660 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vssnm" Sep 11 03:54:50.729598 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:50.729548 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:50.824934 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:50.824903 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:50.939796 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:50.939757 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:50.996507 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:50.996414 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kdmjj"] Sep 11 03:54:51.000659 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:51.000637 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kdmjj" Sep 11 03:54:51.002697 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:51.002675 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 11 03:54:51.002938 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:51.002918 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 11 03:54:51.003064 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:51.003026 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 11 03:54:51.003713 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:51.003694 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 11 03:54:51.003815 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:51.003732 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zn7tg\"" Sep 11 03:54:51.003815 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:51.003694 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 11 03:54:51.003931 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:51.003879 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 11 03:54:51.036379 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:51.036344 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:51.110492 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:51.110450 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/86ba6125-3e14-4a45-ba63-dc428059e6ea-node-exporter-accelerators-collector-config\") pod \"node-exporter-kdmjj\" (UID: \"86ba6125-3e14-4a45-ba63-dc428059e6ea\") " pod="openshift-monitoring/node-exporter-kdmjj" Sep 11 03:54:51.110706 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:51.110502 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/86ba6125-3e14-4a45-ba63-dc428059e6ea-node-exporter-textfile\") pod \"node-exporter-kdmjj\" (UID: \"86ba6125-3e14-4a45-ba63-dc428059e6ea\") " pod="openshift-monitoring/node-exporter-kdmjj" Sep 11 03:54:51.110706 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:51.110575 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/86ba6125-3e14-4a45-ba63-dc428059e6ea-root\") pod \"node-exporter-kdmjj\" (UID: \"86ba6125-3e14-4a45-ba63-dc428059e6ea\") " pod="openshift-monitoring/node-exporter-kdmjj" Sep 11 03:54:51.110706 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:51.110598 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/86ba6125-3e14-4a45-ba63-dc428059e6ea-node-exporter-tls\") pod \"node-exporter-kdmjj\" (UID: \"86ba6125-3e14-4a45-ba63-dc428059e6ea\") " pod="openshift-monitoring/node-exporter-kdmjj" Sep 11 03:54:51.110706 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:51.110643 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/86ba6125-3e14-4a45-ba63-dc428059e6ea-sys\") pod \"node-exporter-kdmjj\" (UID: \"86ba6125-3e14-4a45-ba63-dc428059e6ea\") " pod="openshift-monitoring/node-exporter-kdmjj" Sep 11 03:54:51.110706 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:51.110671 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/86ba6125-3e14-4a45-ba63-dc428059e6ea-node-exporter-wtmp\") pod \"node-exporter-kdmjj\" (UID: \"86ba6125-3e14-4a45-ba63-dc428059e6ea\") " pod="openshift-monitoring/node-exporter-kdmjj" Sep 11 03:54:51.110706 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:51.110696 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86ba6125-3e14-4a45-ba63-dc428059e6ea-metrics-client-ca\") pod \"node-exporter-kdmjj\" (UID: \"86ba6125-3e14-4a45-ba63-dc428059e6ea\") " pod="openshift-monitoring/node-exporter-kdmjj" Sep 11 03:54:51.110927 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:51.110745 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw7t8\" (UniqueName: \"kubernetes.io/projected/86ba6125-3e14-4a45-ba63-dc428059e6ea-kube-api-access-bw7t8\") pod \"node-exporter-kdmjj\" (UID: \"86ba6125-3e14-4a45-ba63-dc428059e6ea\") " pod="openshift-monitoring/node-exporter-kdmjj" Sep 11 03:54:51.110927 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:51.110771 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/86ba6125-3e14-4a45-ba63-dc428059e6ea-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kdmjj\" (UID: \"86ba6125-3e14-4a45-ba63-dc428059e6ea\") " pod="openshift-monitoring/node-exporter-kdmjj" Sep 11 03:54:51.197597 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:51.197550 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:51.211503 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:51.211478 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw7t8\" (UniqueName: \"kubernetes.io/projected/86ba6125-3e14-4a45-ba63-dc428059e6ea-kube-api-access-bw7t8\") pod \"node-exporter-kdmjj\" (UID: \"86ba6125-3e14-4a45-ba63-dc428059e6ea\") " pod="openshift-monitoring/node-exporter-kdmjj" Sep 11 03:54:51.211656 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:51.211515 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/86ba6125-3e14-4a45-ba63-dc428059e6ea-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kdmjj\" (UID: \"86ba6125-3e14-4a45-ba63-dc428059e6ea\") " pod="openshift-monitoring/node-exporter-kdmjj" Sep 11 03:54:51.211656 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:51.211586 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/86ba6125-3e14-4a45-ba63-dc428059e6ea-node-exporter-accelerators-collector-config\") pod \"node-exporter-kdmjj\" (UID: \"86ba6125-3e14-4a45-ba63-dc428059e6ea\") " pod="openshift-monitoring/node-exporter-kdmjj" Sep 11 03:54:51.211656 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:51.211616 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/86ba6125-3e14-4a45-ba63-dc428059e6ea-node-exporter-textfile\") pod \"node-exporter-kdmjj\" (UID: \"86ba6125-3e14-4a45-ba63-dc428059e6ea\") " pod="openshift-monitoring/node-exporter-kdmjj" Sep 11 03:54:51.211820 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:51.211656 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/86ba6125-3e14-4a45-ba63-dc428059e6ea-root\") pod \"node-exporter-kdmjj\" (UID: \"86ba6125-3e14-4a45-ba63-dc428059e6ea\") " pod="openshift-monitoring/node-exporter-kdmjj" Sep 11 03:54:51.211820 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:51.211678 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/86ba6125-3e14-4a45-ba63-dc428059e6ea-node-exporter-tls\") pod \"node-exporter-kdmjj\" (UID: \"86ba6125-3e14-4a45-ba63-dc428059e6ea\") " pod="openshift-monitoring/node-exporter-kdmjj" Sep 11 03:54:51.211820 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:51.211711 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/86ba6125-3e14-4a45-ba63-dc428059e6ea-sys\") pod \"node-exporter-kdmjj\" (UID: \"86ba6125-3e14-4a45-ba63-dc428059e6ea\") " pod="openshift-monitoring/node-exporter-kdmjj" Sep 11 03:54:51.211820 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:51.211773 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/86ba6125-3e14-4a45-ba63-dc428059e6ea-sys\") pod \"node-exporter-kdmjj\" (UID: \"86ba6125-3e14-4a45-ba63-dc428059e6ea\") " pod="openshift-monitoring/node-exporter-kdmjj" Sep 11 03:54:51.211820 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:51.211772 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/86ba6125-3e14-4a45-ba63-dc428059e6ea-root\") pod \"node-exporter-kdmjj\" (UID: \"86ba6125-3e14-4a45-ba63-dc428059e6ea\") " pod="openshift-monitoring/node-exporter-kdmjj" Sep 11 03:54:51.212070 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:51.211849 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/86ba6125-3e14-4a45-ba63-dc428059e6ea-node-exporter-wtmp\") pod \"node-exporter-kdmjj\" (UID: \"86ba6125-3e14-4a45-ba63-dc428059e6ea\") " pod="openshift-monitoring/node-exporter-kdmjj" Sep 11 03:54:51.212070 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:51.211884 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86ba6125-3e14-4a45-ba63-dc428059e6ea-metrics-client-ca\") pod \"node-exporter-kdmjj\" (UID: \"86ba6125-3e14-4a45-ba63-dc428059e6ea\") " pod="openshift-monitoring/node-exporter-kdmjj" Sep 11 03:54:51.212070 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:51.211978 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/86ba6125-3e14-4a45-ba63-dc428059e6ea-node-exporter-wtmp\") pod \"node-exporter-kdmjj\" (UID: \"86ba6125-3e14-4a45-ba63-dc428059e6ea\") " pod="openshift-monitoring/node-exporter-kdmjj" Sep 11 03:54:51.212286 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:51.212267 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/86ba6125-3e14-4a45-ba63-dc428059e6ea-node-exporter-textfile\") pod \"node-exporter-kdmjj\" (UID: \"86ba6125-3e14-4a45-ba63-dc428059e6ea\") " pod="openshift-monitoring/node-exporter-kdmjj" Sep 11 03:54:51.212349 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:51.212321 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/86ba6125-3e14-4a45-ba63-dc428059e6ea-node-exporter-accelerators-collector-config\") pod \"node-exporter-kdmjj\" (UID: \"86ba6125-3e14-4a45-ba63-dc428059e6ea\") " pod="openshift-monitoring/node-exporter-kdmjj" Sep 11 03:54:51.212447 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:51.212427 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/86ba6125-3e14-4a45-ba63-dc428059e6ea-metrics-client-ca\") pod \"node-exporter-kdmjj\" (UID: \"86ba6125-3e14-4a45-ba63-dc428059e6ea\") " pod="openshift-monitoring/node-exporter-kdmjj" Sep 11 03:54:51.214530 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:51.214508 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/86ba6125-3e14-4a45-ba63-dc428059e6ea-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kdmjj\" (UID: \"86ba6125-3e14-4a45-ba63-dc428059e6ea\") " pod="openshift-monitoring/node-exporter-kdmjj" Sep 11 03:54:51.214599 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:51.214543 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/86ba6125-3e14-4a45-ba63-dc428059e6ea-node-exporter-tls\") pod \"node-exporter-kdmjj\" (UID: \"86ba6125-3e14-4a45-ba63-dc428059e6ea\") " pod="openshift-monitoring/node-exporter-kdmjj" Sep 11 03:54:51.224732 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:51.224706 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw7t8\" (UniqueName: \"kubernetes.io/projected/86ba6125-3e14-4a45-ba63-dc428059e6ea-kube-api-access-bw7t8\") pod \"node-exporter-kdmjj\" (UID: \"86ba6125-3e14-4a45-ba63-dc428059e6ea\") " pod="openshift-monitoring/node-exporter-kdmjj" Sep 11 03:54:51.298340 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:51.298264 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:51.312061 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:51.312031 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kdmjj" Sep 11 03:54:51.325068 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:54:51.325030 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86ba6125_3e14_4a45_ba63_dc428059e6ea.slice/crio-d21402436bac97e5abac0e9e2ed42e688d97623e404e3c8ba1fba35abc416813 WatchSource:0}: Error finding container d21402436bac97e5abac0e9e2ed42e688d97623e404e3c8ba1fba35abc416813: Status 404 returned error can't find the container with id d21402436bac97e5abac0e9e2ed42e688d97623e404e3c8ba1fba35abc416813 Sep 11 03:54:51.415863 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:51.415827 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:51.617383 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:51.617304 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:51.944377 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:51.944321 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:52.029103 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:52.029069 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:52.152202 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:52.152162 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:52.214817 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:52.214721 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:52.233623 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:52.233589 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kdmjj" event={"ID":"86ba6125-3e14-4a45-ba63-dc428059e6ea","Type":"ContainerStarted","Data":"d21402436bac97e5abac0e9e2ed42e688d97623e404e3c8ba1fba35abc416813"} Sep 11 03:54:52.411957 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:52.411932 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:52.512942 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:52.512854 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:53.238688 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:53.238653 2568 generic.go:356] "Generic (PLEG): container finished" podID="86ba6125-3e14-4a45-ba63-dc428059e6ea" containerID="1c1723eb6ed0113fe038b40cb3b6031ff454dd56091834f6b496539ba1b4ffd5" exitCode=0 Sep 11 03:54:53.239279 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:53.238718 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kdmjj" event={"ID":"86ba6125-3e14-4a45-ba63-dc428059e6ea","Type":"ContainerDied","Data":"1c1723eb6ed0113fe038b40cb3b6031ff454dd56091834f6b496539ba1b4ffd5"} Sep 11 03:54:53.379518 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:53.379490 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:53.480736 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:53.480693 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:53.908889 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:53.908855 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:53.988769 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:53.988731 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:54.083670 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:54.083639 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:54.184902 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:54.184827 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:54.243869 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:54.243834 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kdmjj" event={"ID":"86ba6125-3e14-4a45-ba63-dc428059e6ea","Type":"ContainerStarted","Data":"bc35efa283ca5cc5343247b10db939c45f1fe5994ba0e55ba555b754358c368d"} Sep 11 03:54:54.244329 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:54.243886 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kdmjj" event={"ID":"86ba6125-3e14-4a45-ba63-dc428059e6ea","Type":"ContainerStarted","Data":"9e083144fae0c63166eb871854375a8bd70d4050c32a781846368a8fbd37886a"} Sep 11 03:54:54.264642 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:54.264597 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kdmjj" podStartSLOduration=3.130148138 podStartE2EDuration="4.264584577s" totalImagesPullingTime="1.074109514s" totalInitContainerRuntime="60.326925ms" isStatefulPod=true podCreationTimestamp="2026-09-11 03:54:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:54:54.262964772 +0000 UTC m=+109.016596173" watchObservedRunningTime="2026-09-11 03:54:54.264584577 +0000 UTC m=+109.018215978" Sep 11 03:54:55.797512 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:55.797469 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:55.899554 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:55.899516 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:55.967720 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:55.967685 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:56.034692 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:56.034659 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:57.059548 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:57.059512 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:57.159184 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:57.159150 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:57.220491 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:57.220460 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ssrwd" Sep 11 03:54:57.221339 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:57.221296 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:57.356021 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:57.355943 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6c8dd7548c-jmns9"] Sep 11 03:54:57.484705 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:57.484671 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:57.583823 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:57.583781 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:57.731249 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:57.731212 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:54:57.831554 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:54:57.831516 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:55:01.349241 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:01.349202 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:55:01.449770 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:01.449731 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:55:05.944997 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:05.944957 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:55:06.007623 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:06.007585 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:55:07.361779 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:07.361748 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6c8dd7548c-jmns9" Sep 11 03:55:07.945490 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:07.945457 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:55:08.054750 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:08.054707 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:55:08.151331 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:08.151298 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:55:10.401725 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:10.401689 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:55:10.510411 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:10.510371 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:55:13.174015 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:13.173975 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:55:13.269770 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:13.269733 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:55:15.034631 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:15.034588 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:55:15.136238 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:15.136203 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:55:15.505803 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:15.505774 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe1486cf-6276-41a8-a685-025064c4abca-metrics-certs\") pod \"network-metrics-daemon-tk8q7\" (UID: \"fe1486cf-6276-41a8-a685-025064c4abca\") " pod="openshift-multus/network-metrics-daemon-tk8q7" Sep 11 03:55:15.508002 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:15.507974 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fe1486cf-6276-41a8-a685-025064c4abca-metrics-certs\") pod \"network-metrics-daemon-tk8q7\" (UID: \"fe1486cf-6276-41a8-a685-025064c4abca\") " pod="openshift-multus/network-metrics-daemon-tk8q7" Sep 11 03:55:15.682573 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:15.682545 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rtj7z\"" Sep 11 03:55:15.690706 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:15.690682 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk8q7" Sep 11 03:55:15.970321 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:15.970292 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tk8q7"] Sep 11 03:55:15.973518 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:55:15.973492 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe1486cf_6276_41a8_a685_025064c4abca.slice/crio-cda11397f193259c9dfa1435da2f4a25601c021cb804115af3e9a0013c0ec377 WatchSource:0}: Error finding container cda11397f193259c9dfa1435da2f4a25601c021cb804115af3e9a0013c0ec377: Status 404 returned error can't find the container with id cda11397f193259c9dfa1435da2f4a25601c021cb804115af3e9a0013c0ec377 Sep 11 03:55:16.320194 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:16.320109 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tk8q7" event={"ID":"fe1486cf-6276-41a8-a685-025064c4abca","Type":"ContainerStarted","Data":"cda11397f193259c9dfa1435da2f4a25601c021cb804115af3e9a0013c0ec377"} Sep 11 03:55:17.325836 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:17.325755 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tk8q7" event={"ID":"fe1486cf-6276-41a8-a685-025064c4abca","Type":"ContainerStarted","Data":"0ffb9821b8d7e08a5f09bc1481ebaf2f96cb342daa1a6f3270c190bd42dc19ae"} Sep 11 03:55:17.325836 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:17.325812 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tk8q7" event={"ID":"fe1486cf-6276-41a8-a685-025064c4abca","Type":"ContainerStarted","Data":"47890c70c471e884740c90f6f60241abd2f9db34943b68a9f26c2655ee7004cb"} Sep 11 03:55:17.472490 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:17.472462 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:55:17.571138 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:17.571102 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:55:17.651958 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:17.651930 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:55:20.813788 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:20.813748 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:55:20.874979 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:20.874945 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:55:22.374497 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:22.374440 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6c8dd7548c-jmns9" podUID="52394690-486c-4ae0-b36b-603b438a944a" containerName="registry" containerID="cri-o://9dc520ca64ec9550eb3e6371e7482920d3de8721aa06fda44e735ff8eaef6b6e" gracePeriod=30 Sep 11 03:55:22.609466 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:22.609436 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c8dd7548c-jmns9" Sep 11 03:55:22.632218 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:22.632128 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-tk8q7" podStartSLOduration=136.631099971 podStartE2EDuration="2m17.632111049s" totalImagesPullingTime="1.001011078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:53:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:55:17.345552134 +0000 UTC m=+132.099183536" watchObservedRunningTime="2026-09-11 03:55:22.632111049 +0000 UTC m=+137.385742454" Sep 11 03:55:22.660167 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:22.660136 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/52394690-486c-4ae0-b36b-603b438a944a-ca-trust-extracted\") pod \"52394690-486c-4ae0-b36b-603b438a944a\" (UID: \"52394690-486c-4ae0-b36b-603b438a944a\") " Sep 11 03:55:22.660319 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:22.660192 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/52394690-486c-4ae0-b36b-603b438a944a-bound-sa-token\") pod \"52394690-486c-4ae0-b36b-603b438a944a\" (UID: \"52394690-486c-4ae0-b36b-603b438a944a\") " Sep 11 03:55:22.660319 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:22.660212 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/52394690-486c-4ae0-b36b-603b438a944a-trusted-ca\") pod \"52394690-486c-4ae0-b36b-603b438a944a\" (UID: \"52394690-486c-4ae0-b36b-603b438a944a\") " Sep 11 03:55:22.660319 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:22.660234 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wggx7\" (UniqueName: \"kubernetes.io/projected/52394690-486c-4ae0-b36b-603b438a944a-kube-api-access-wggx7\") pod \"52394690-486c-4ae0-b36b-603b438a944a\" (UID: \"52394690-486c-4ae0-b36b-603b438a944a\") " Sep 11 03:55:22.660319 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:22.660279 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/52394690-486c-4ae0-b36b-603b438a944a-registry-certificates\") pod \"52394690-486c-4ae0-b36b-603b438a944a\" (UID: \"52394690-486c-4ae0-b36b-603b438a944a\") " Sep 11 03:55:22.660319 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:22.660312 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/52394690-486c-4ae0-b36b-603b438a944a-registry-tls\") pod \"52394690-486c-4ae0-b36b-603b438a944a\" (UID: \"52394690-486c-4ae0-b36b-603b438a944a\") " Sep 11 03:55:22.660542 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:22.660360 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/52394690-486c-4ae0-b36b-603b438a944a-image-registry-private-configuration\") pod \"52394690-486c-4ae0-b36b-603b438a944a\" (UID: \"52394690-486c-4ae0-b36b-603b438a944a\") " Sep 11 03:55:22.660542 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:22.660387 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/52394690-486c-4ae0-b36b-603b438a944a-installation-pull-secrets\") pod \"52394690-486c-4ae0-b36b-603b438a944a\" (UID: \"52394690-486c-4ae0-b36b-603b438a944a\") " Sep 11 03:55:22.660814 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:22.660755 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52394690-486c-4ae0-b36b-603b438a944a-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "52394690-486c-4ae0-b36b-603b438a944a" (UID: "52394690-486c-4ae0-b36b-603b438a944a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 03:55:22.660939 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:22.660817 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52394690-486c-4ae0-b36b-603b438a944a-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "52394690-486c-4ae0-b36b-603b438a944a" (UID: "52394690-486c-4ae0-b36b-603b438a944a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 03:55:22.662971 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:22.662930 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52394690-486c-4ae0-b36b-603b438a944a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "52394690-486c-4ae0-b36b-603b438a944a" (UID: "52394690-486c-4ae0-b36b-603b438a944a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 03:55:22.662971 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:22.662950 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52394690-486c-4ae0-b36b-603b438a944a-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "52394690-486c-4ae0-b36b-603b438a944a" (UID: "52394690-486c-4ae0-b36b-603b438a944a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 03:55:22.663215 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:22.663191 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52394690-486c-4ae0-b36b-603b438a944a-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "52394690-486c-4ae0-b36b-603b438a944a" (UID: "52394690-486c-4ae0-b36b-603b438a944a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 03:55:22.663272 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:22.663207 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52394690-486c-4ae0-b36b-603b438a944a-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "52394690-486c-4ae0-b36b-603b438a944a" (UID: "52394690-486c-4ae0-b36b-603b438a944a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 03:55:22.663308 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:22.663280 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52394690-486c-4ae0-b36b-603b438a944a-kube-api-access-wggx7" (OuterVolumeSpecName: "kube-api-access-wggx7") pod "52394690-486c-4ae0-b36b-603b438a944a" (UID: "52394690-486c-4ae0-b36b-603b438a944a"). InnerVolumeSpecName "kube-api-access-wggx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 03:55:22.668783 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:22.668758 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52394690-486c-4ae0-b36b-603b438a944a-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "52394690-486c-4ae0-b36b-603b438a944a" (UID: "52394690-486c-4ae0-b36b-603b438a944a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 03:55:22.761632 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:22.761598 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wggx7\" (UniqueName: \"kubernetes.io/projected/52394690-486c-4ae0-b36b-603b438a944a-kube-api-access-wggx7\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 03:55:22.761632 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:22.761628 2568 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/52394690-486c-4ae0-b36b-603b438a944a-registry-certificates\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 03:55:22.761632 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:22.761638 2568 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/52394690-486c-4ae0-b36b-603b438a944a-registry-tls\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 03:55:22.761840 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:22.761648 2568 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/52394690-486c-4ae0-b36b-603b438a944a-image-registry-private-configuration\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 03:55:22.761840 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:22.761659 2568 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/52394690-486c-4ae0-b36b-603b438a944a-installation-pull-secrets\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 03:55:22.761840 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:22.761669 2568 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/52394690-486c-4ae0-b36b-603b438a944a-ca-trust-extracted\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 03:55:22.761840 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:22.761677 2568 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/52394690-486c-4ae0-b36b-603b438a944a-bound-sa-token\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 03:55:22.761840 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:22.761684 2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/52394690-486c-4ae0-b36b-603b438a944a-trusted-ca\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 03:55:23.346828 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:23.346794 2568 generic.go:356] "Generic (PLEG): container finished" podID="52394690-486c-4ae0-b36b-603b438a944a" containerID="9dc520ca64ec9550eb3e6371e7482920d3de8721aa06fda44e735ff8eaef6b6e" exitCode=0 Sep 11 03:55:23.347020 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:23.346845 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c8dd7548c-jmns9" Sep 11 03:55:23.347020 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:23.346878 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c8dd7548c-jmns9" event={"ID":"52394690-486c-4ae0-b36b-603b438a944a","Type":"ContainerDied","Data":"9dc520ca64ec9550eb3e6371e7482920d3de8721aa06fda44e735ff8eaef6b6e"} Sep 11 03:55:23.347020 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:23.346932 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c8dd7548c-jmns9" event={"ID":"52394690-486c-4ae0-b36b-603b438a944a","Type":"ContainerDied","Data":"076c0369aa41514e2a388d19f77809b73e1e07be75be7da77e044e9491b0125e"} Sep 11 03:55:23.347020 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:23.346954 2568 scope.go:117] "RemoveContainer" containerID="9dc520ca64ec9550eb3e6371e7482920d3de8721aa06fda44e735ff8eaef6b6e" Sep 11 03:55:23.355364 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:23.355351 2568 scope.go:117] "RemoveContainer" containerID="9dc520ca64ec9550eb3e6371e7482920d3de8721aa06fda44e735ff8eaef6b6e" Sep 11 03:55:23.355617 ip-10-0-140-30 kubenswrapper[2568]: E0911 03:55:23.355596 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dc520ca64ec9550eb3e6371e7482920d3de8721aa06fda44e735ff8eaef6b6e\": container with ID starting with 9dc520ca64ec9550eb3e6371e7482920d3de8721aa06fda44e735ff8eaef6b6e not found: ID does not exist" containerID="9dc520ca64ec9550eb3e6371e7482920d3de8721aa06fda44e735ff8eaef6b6e" Sep 11 03:55:23.355661 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:23.355626 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dc520ca64ec9550eb3e6371e7482920d3de8721aa06fda44e735ff8eaef6b6e"} err="failed to get container status \"9dc520ca64ec9550eb3e6371e7482920d3de8721aa06fda44e735ff8eaef6b6e\": rpc error: code = NotFound desc = could not find container \"9dc520ca64ec9550eb3e6371e7482920d3de8721aa06fda44e735ff8eaef6b6e\": container with ID starting with 9dc520ca64ec9550eb3e6371e7482920d3de8721aa06fda44e735ff8eaef6b6e not found: ID does not exist" Sep 11 03:55:23.379474 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:23.379447 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6c8dd7548c-jmns9"] Sep 11 03:55:23.386247 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:23.386227 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6c8dd7548c-jmns9"] Sep 11 03:55:23.767256 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:23.767216 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52394690-486c-4ae0-b36b-603b438a944a" path="/var/lib/kubelet/pods/52394690-486c-4ae0-b36b-603b438a944a/volumes" Sep 11 03:55:25.357341 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:25.357308 2568 generic.go:356] "Generic (PLEG): container finished" podID="320f0913-5ebe-45b4-a60d-8359a83bd24b" containerID="7bb112cde210b5ca8e4295c44716b4faa283644f63e724b54346e6e82b01fb01" exitCode=0 Sep 11 03:55:25.357712 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:25.357346 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-b67kh" event={"ID":"320f0913-5ebe-45b4-a60d-8359a83bd24b","Type":"ContainerDied","Data":"7bb112cde210b5ca8e4295c44716b4faa283644f63e724b54346e6e82b01fb01"} Sep 11 03:55:25.357712 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:25.357677 2568 scope.go:117] "RemoveContainer" containerID="7bb112cde210b5ca8e4295c44716b4faa283644f63e724b54346e6e82b01fb01" Sep 11 03:55:26.362578 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:26.362543 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-b67kh" event={"ID":"320f0913-5ebe-45b4-a60d-8359a83bd24b","Type":"ContainerStarted","Data":"867f099c31abf40a7e61238edc2c2fed18dab532533515cf41e503bbf0c82136"} Sep 11 03:55:35.815347 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:35.815311 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:55:35.873149 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:35.873114 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:55:39.408746 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:39.408710 2568 generic.go:356] "Generic (PLEG): container finished" podID="7be9d187-475e-4a85-afc1-2f385d5a262e" containerID="d01bdb1e50838341b79260bc870b9f5ee37f042aa4f7d53ed2f5fd5252fd5226" exitCode=0 Sep 11 03:55:39.409133 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:39.408789 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-4w9k5" event={"ID":"7be9d187-475e-4a85-afc1-2f385d5a262e","Type":"ContainerDied","Data":"d01bdb1e50838341b79260bc870b9f5ee37f042aa4f7d53ed2f5fd5252fd5226"} Sep 11 03:55:39.409133 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:39.409124 2568 scope.go:117] "RemoveContainer" containerID="d01bdb1e50838341b79260bc870b9f5ee37f042aa4f7d53ed2f5fd5252fd5226" Sep 11 03:55:39.953175 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:39.953146 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nq6qk_badea905-61e6-47e1-8f68-505d04824821/serve-healthcheck-canary/0.log" Sep 11 03:55:40.416127 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:40.416080 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5844b856db-4w9k5" event={"ID":"7be9d187-475e-4a85-afc1-2f385d5a262e","Type":"ContainerStarted","Data":"4972ee8a6671fbc15b1c1a97485086b8ab1334778c8b8baa58420e9928afe3dc"} Sep 11 03:55:43.743086 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:43.743028 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:55:43.843040 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:43.843004 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:55:43.943887 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:55:43.943851 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:56:10.194335 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:56:10.194295 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:56:10.329448 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:56:10.329411 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:56:18.229606 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:56:18.229567 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:56:18.330601 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:56:18.330559 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:56:29.259028 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:56:29.258987 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:56:29.311535 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:56:29.311501 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:56:29.369131 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:56:29.369095 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:56:42.033664 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:56:42.033628 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:57:10.040079 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:57:10.040025 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:57:10.095675 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:57:10.095651 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:57:25.502467 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:57:25.502429 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:57:25.586821 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:57:25.586791 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:57:28.748590 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:57:28.748505 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:57:28.843173 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:57:28.843137 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:57:28.901200 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:57:28.901165 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:57:29.462800 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:57:29.462768 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:57:29.564755 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:57:29.564721 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:57:31.959325 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:57:31.959291 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:57:37.330793 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:57:37.330758 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:57:37.430834 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:57:37.430798 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:57:39.577577 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:57:39.577546 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:57:39.639266 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:57:39.639238 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:57:39.704765 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:57:39.704726 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:57:50.830619 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:57:50.830581 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:57:50.929788 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:57:50.929734 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:58:05.668278 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:58:05.668250 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-5tpdc_9112557d-d0ef-44c7-8b2a-f6ecd0a980d6/console-operator/1.log" Sep 11 03:58:05.677866 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:58:05.677842 2568 kubelet.go:1648] "Image garbage collection succeeded" Sep 11 03:58:05.678253 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:58:05.678238 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-5tpdc_9112557d-d0ef-44c7-8b2a-f6ecd0a980d6/console-operator/1.log" Sep 11 03:58:14.026735 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:58:14.026689 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:58:15.254001 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:58:15.253971 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:58:36.987397 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:58:36.987358 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:58:38.326622 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:58:38.326572 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:59:03.746591 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:03.746543 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:59:04.541712 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:04.541673 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:59:07.364776 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:07.364740 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:59:07.464529 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:07.464494 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:59:08.818744 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:08.818706 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:59:08.937476 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:08.937442 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:59:09.046474 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:09.046417 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:59:09.142330 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:09.142230 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:59:09.147790 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:09.147766 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-7pg67"] Sep 11 03:59:09.148053 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:09.148031 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52394690-486c-4ae0-b36b-603b438a944a" containerName="registry" Sep 11 03:59:09.148102 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:09.148060 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="52394690-486c-4ae0-b36b-603b438a944a" containerName="registry" Sep 11 03:59:09.148137 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:09.148104 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="52394690-486c-4ae0-b36b-603b438a944a" containerName="registry" Sep 11 03:59:09.150756 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:09.150739 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-7pg67" Sep 11 03:59:09.153111 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:09.153092 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 11 03:59:09.153111 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:09.153105 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 11 03:59:09.160166 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:09.160136 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-7pg67"] Sep 11 03:59:09.304216 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:09.300779 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdqw9\" (UniqueName: \"kubernetes.io/projected/8cad5b9f-0f4d-4442-ab97-3ac7e878d45c-kube-api-access-kdqw9\") pod \"cert-manager-webhook-6486ffd997-7pg67\" (UID: \"8cad5b9f-0f4d-4442-ab97-3ac7e878d45c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7pg67" Sep 11 03:59:09.304216 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:09.300839 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8cad5b9f-0f4d-4442-ab97-3ac7e878d45c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-7pg67\" (UID: \"8cad5b9f-0f4d-4442-ab97-3ac7e878d45c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7pg67" Sep 11 03:59:09.401881 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:09.401841 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdqw9\" (UniqueName: \"kubernetes.io/projected/8cad5b9f-0f4d-4442-ab97-3ac7e878d45c-kube-api-access-kdqw9\") pod \"cert-manager-webhook-6486ffd997-7pg67\" (UID: \"8cad5b9f-0f4d-4442-ab97-3ac7e878d45c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7pg67" Sep 11 03:59:09.401881 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:09.401884 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8cad5b9f-0f4d-4442-ab97-3ac7e878d45c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-7pg67\" (UID: \"8cad5b9f-0f4d-4442-ab97-3ac7e878d45c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7pg67" Sep 11 03:59:09.411153 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:09.411122 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8cad5b9f-0f4d-4442-ab97-3ac7e878d45c-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-7pg67\" (UID: \"8cad5b9f-0f4d-4442-ab97-3ac7e878d45c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7pg67" Sep 11 03:59:09.411283 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:09.411252 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdqw9\" (UniqueName: \"kubernetes.io/projected/8cad5b9f-0f4d-4442-ab97-3ac7e878d45c-kube-api-access-kdqw9\") pod \"cert-manager-webhook-6486ffd997-7pg67\" (UID: \"8cad5b9f-0f4d-4442-ab97-3ac7e878d45c\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7pg67" Sep 11 03:59:09.459660 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:09.459632 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-7pg67" Sep 11 03:59:09.726900 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:09.726823 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-7pg67"] Sep 11 03:59:09.729799 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:59:09.729768 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8cad5b9f_0f4d_4442_ab97_3ac7e878d45c.slice/crio-575e37cab7c91e02b787fa1df2aeb7f27b17c16b98edee4f9f4731fdb6249111 WatchSource:0}: Error finding container 575e37cab7c91e02b787fa1df2aeb7f27b17c16b98edee4f9f4731fdb6249111: Status 404 returned error can't find the container with id 575e37cab7c91e02b787fa1df2aeb7f27b17c16b98edee4f9f4731fdb6249111 Sep 11 03:59:09.731664 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:09.731643 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 03:59:10.136029 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:10.135949 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-7pg67" event={"ID":"8cad5b9f-0f4d-4442-ab97-3ac7e878d45c","Type":"ContainerStarted","Data":"575e37cab7c91e02b787fa1df2aeb7f27b17c16b98edee4f9f4731fdb6249111"} Sep 11 03:59:11.219944 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:11.219908 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:59:11.608329 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:11.608234 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:59:13.150556 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:13.150468 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-7pg67" event={"ID":"8cad5b9f-0f4d-4442-ab97-3ac7e878d45c","Type":"ContainerStarted","Data":"274a2a120ec814ce8df9bfdf321577a30c825d0b1f01aac8cde284a449adc1f3"} Sep 11 03:59:13.150931 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:13.150581 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-7pg67" Sep 11 03:59:13.174239 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:13.174190 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-7pg67" podStartSLOduration=1.154828366 podStartE2EDuration="4.174175004s" totalImagesPullingTime="3.019346638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:59:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:59:13.173584898 +0000 UTC m=+367.927216301" watchObservedRunningTime="2026-09-11 03:59:13.174175004 +0000 UTC m=+367.927806405" Sep 11 03:59:14.160305 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:14.160270 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:59:14.260227 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:14.260193 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:59:19.155530 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:19.155500 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-7pg67" Sep 11 03:59:19.219028 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:19.218999 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:59:20.163204 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:20.163155 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:59:20.263356 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:20.263315 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:59:21.165094 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:21.165061 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:59:21.220991 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:21.220958 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:59:27.695153 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:27.695093 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:59:27.791236 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:27.791191 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:59:30.008398 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:30.008351 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:59:42.314143 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:42.314106 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:59:42.410499 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:42.410464 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:59:42.907924 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:42.907891 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:59:43.144408 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:43.144379 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:59:43.253853 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:43.253754 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:59:43.299198 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:43.299162 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-65b869c898-d5nnh"] Sep 11 03:59:43.304010 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:43.303991 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-65b869c898-d5nnh" Sep 11 03:59:43.307060 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:43.307019 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 11 03:59:43.307060 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:43.307041 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-nmbvm\"" Sep 11 03:59:43.321924 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:43.321901 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 11 03:59:43.322323 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:43.322281 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 11 03:59:43.322396 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:43.322372 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 11 03:59:43.339247 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:43.339208 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw8kj\" (UniqueName: \"kubernetes.io/projected/786a4185-ee93-4ab3-9b70-a64bfd995e1b-kube-api-access-xw8kj\") pod \"opendatahub-operator-controller-manager-65b869c898-d5nnh\" (UID: \"786a4185-ee93-4ab3-9b70-a64bfd995e1b\") " pod="opendatahub/opendatahub-operator-controller-manager-65b869c898-d5nnh" Sep 11 03:59:43.339403 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:43.339311 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/786a4185-ee93-4ab3-9b70-a64bfd995e1b-manifests\") pod \"opendatahub-operator-controller-manager-65b869c898-d5nnh\" (UID: \"786a4185-ee93-4ab3-9b70-a64bfd995e1b\") " pod="opendatahub/opendatahub-operator-controller-manager-65b869c898-d5nnh" Sep 11 03:59:43.339482 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:43.339404 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/786a4185-ee93-4ab3-9b70-a64bfd995e1b-webhook-cert\") pod \"opendatahub-operator-controller-manager-65b869c898-d5nnh\" (UID: \"786a4185-ee93-4ab3-9b70-a64bfd995e1b\") " pod="opendatahub/opendatahub-operator-controller-manager-65b869c898-d5nnh" Sep 11 03:59:43.339482 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:43.339452 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/786a4185-ee93-4ab3-9b70-a64bfd995e1b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-65b869c898-d5nnh\" (UID: \"786a4185-ee93-4ab3-9b70-a64bfd995e1b\") " pod="opendatahub/opendatahub-operator-controller-manager-65b869c898-d5nnh" Sep 11 03:59:43.339597 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:43.339525 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/786a4185-ee93-4ab3-9b70-a64bfd995e1b-tmp\") pod \"opendatahub-operator-controller-manager-65b869c898-d5nnh\" (UID: \"786a4185-ee93-4ab3-9b70-a64bfd995e1b\") " pod="opendatahub/opendatahub-operator-controller-manager-65b869c898-d5nnh" Sep 11 03:59:43.341067 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:43.341025 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-65b869c898-d5nnh"] Sep 11 03:59:43.440553 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:43.440515 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/786a4185-ee93-4ab3-9b70-a64bfd995e1b-manifests\") pod \"opendatahub-operator-controller-manager-65b869c898-d5nnh\" (UID: \"786a4185-ee93-4ab3-9b70-a64bfd995e1b\") " pod="opendatahub/opendatahub-operator-controller-manager-65b869c898-d5nnh" Sep 11 03:59:43.440553 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:43.440554 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/786a4185-ee93-4ab3-9b70-a64bfd995e1b-webhook-cert\") pod \"opendatahub-operator-controller-manager-65b869c898-d5nnh\" (UID: \"786a4185-ee93-4ab3-9b70-a64bfd995e1b\") " pod="opendatahub/opendatahub-operator-controller-manager-65b869c898-d5nnh" Sep 11 03:59:43.440764 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:43.440579 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/786a4185-ee93-4ab3-9b70-a64bfd995e1b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-65b869c898-d5nnh\" (UID: \"786a4185-ee93-4ab3-9b70-a64bfd995e1b\") " pod="opendatahub/opendatahub-operator-controller-manager-65b869c898-d5nnh" Sep 11 03:59:43.440764 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:43.440606 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/786a4185-ee93-4ab3-9b70-a64bfd995e1b-tmp\") pod \"opendatahub-operator-controller-manager-65b869c898-d5nnh\" (UID: \"786a4185-ee93-4ab3-9b70-a64bfd995e1b\") " pod="opendatahub/opendatahub-operator-controller-manager-65b869c898-d5nnh" Sep 11 03:59:43.440764 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:43.440645 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw8kj\" (UniqueName: \"kubernetes.io/projected/786a4185-ee93-4ab3-9b70-a64bfd995e1b-kube-api-access-xw8kj\") pod \"opendatahub-operator-controller-manager-65b869c898-d5nnh\" (UID: \"786a4185-ee93-4ab3-9b70-a64bfd995e1b\") " pod="opendatahub/opendatahub-operator-controller-manager-65b869c898-d5nnh" Sep 11 03:59:43.440997 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:43.440974 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/786a4185-ee93-4ab3-9b70-a64bfd995e1b-manifests\") pod \"opendatahub-operator-controller-manager-65b869c898-d5nnh\" (UID: \"786a4185-ee93-4ab3-9b70-a64bfd995e1b\") " pod="opendatahub/opendatahub-operator-controller-manager-65b869c898-d5nnh" Sep 11 03:59:43.442846 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:43.442823 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/786a4185-ee93-4ab3-9b70-a64bfd995e1b-tmp\") pod \"opendatahub-operator-controller-manager-65b869c898-d5nnh\" (UID: \"786a4185-ee93-4ab3-9b70-a64bfd995e1b\") " pod="opendatahub/opendatahub-operator-controller-manager-65b869c898-d5nnh" Sep 11 03:59:43.443194 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:43.443173 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/786a4185-ee93-4ab3-9b70-a64bfd995e1b-webhook-cert\") pod \"opendatahub-operator-controller-manager-65b869c898-d5nnh\" (UID: \"786a4185-ee93-4ab3-9b70-a64bfd995e1b\") " pod="opendatahub/opendatahub-operator-controller-manager-65b869c898-d5nnh" Sep 11 03:59:43.443279 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:43.443249 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/786a4185-ee93-4ab3-9b70-a64bfd995e1b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-65b869c898-d5nnh\" (UID: \"786a4185-ee93-4ab3-9b70-a64bfd995e1b\") " pod="opendatahub/opendatahub-operator-controller-manager-65b869c898-d5nnh" Sep 11 03:59:43.455407 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:43.455384 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw8kj\" (UniqueName: \"kubernetes.io/projected/786a4185-ee93-4ab3-9b70-a64bfd995e1b-kube-api-access-xw8kj\") pod \"opendatahub-operator-controller-manager-65b869c898-d5nnh\" (UID: \"786a4185-ee93-4ab3-9b70-a64bfd995e1b\") " pod="opendatahub/opendatahub-operator-controller-manager-65b869c898-d5nnh" Sep 11 03:59:43.499138 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:43.499112 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:59:43.598397 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:43.598319 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:59:43.614242 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:43.614220 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-65b869c898-d5nnh" Sep 11 03:59:43.699411 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:43.699375 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:59:43.799444 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:43.799413 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:59:43.891661 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:43.891614 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-65b869c898-d5nnh"] Sep 11 03:59:43.894445 ip-10-0-140-30 kubenswrapper[2568]: W0911 03:59:43.894409 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod786a4185_ee93_4ab3_9b70_a64bfd995e1b.slice/crio-dee33b4c8225728feb03619d00e76408e7180a4b6e579178fd9c7b7f947cda85 WatchSource:0}: Error finding container dee33b4c8225728feb03619d00e76408e7180a4b6e579178fd9c7b7f947cda85: Status 404 returned error can't find the container with id dee33b4c8225728feb03619d00e76408e7180a4b6e579178fd9c7b7f947cda85 Sep 11 03:59:43.896267 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:43.896238 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:59:43.963740 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:43.963714 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:59:44.019816 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:44.019786 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:59:44.161904 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:44.161875 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:59:44.243295 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:44.243260 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:59:44.262887 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:44.262854 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-65b869c898-d5nnh" event={"ID":"786a4185-ee93-4ab3-9b70-a64bfd995e1b","Type":"ContainerStarted","Data":"dee33b4c8225728feb03619d00e76408e7180a4b6e579178fd9c7b7f947cda85"} Sep 11 03:59:44.347117 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:44.347090 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:59:44.455000 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:44.454921 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:59:47.275180 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:47.275141 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-65b869c898-d5nnh" event={"ID":"786a4185-ee93-4ab3-9b70-a64bfd995e1b","Type":"ContainerStarted","Data":"b0fa1f9fca005e0f0352af0cdb855cc62843aaeec6795033914216b454ffcada"} Sep 11 03:59:48.280300 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:48.280213 2568 generic.go:356] "Generic (PLEG): container finished" podID="786a4185-ee93-4ab3-9b70-a64bfd995e1b" containerID="b0fa1f9fca005e0f0352af0cdb855cc62843aaeec6795033914216b454ffcada" exitCode=0 Sep 11 03:59:48.280300 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:48.280283 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-65b869c898-d5nnh" event={"ID":"786a4185-ee93-4ab3-9b70-a64bfd995e1b","Type":"ContainerDied","Data":"b0fa1f9fca005e0f0352af0cdb855cc62843aaeec6795033914216b454ffcada"} Sep 11 03:59:49.287015 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:49.286975 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-65b869c898-d5nnh" event={"ID":"786a4185-ee93-4ab3-9b70-a64bfd995e1b","Type":"ContainerStarted","Data":"6011c34f9918312502e53751ad23007ae3e3b80fc886b05464a8c1e8fa877995"} Sep 11 03:59:49.287510 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:49.287200 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-65b869c898-d5nnh" Sep 11 03:59:49.311108 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:49.311042 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-65b869c898-d5nnh" podStartSLOduration=1.953624049 podStartE2EDuration="6.311026973s" totalImagesPullingTime="3.28666377s" totalInitContainerRuntime="1.070739154s" isStatefulPod=false podCreationTimestamp="2026-09-11 03:59:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 03:59:49.309002622 +0000 UTC m=+404.062634037" watchObservedRunningTime="2026-09-11 03:59:49.311026973 +0000 UTC m=+404.064658374" Sep 11 03:59:52.624962 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:52.624929 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:59:52.726845 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:52.726813 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:59:52.830830 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:52.830754 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:59:53.344815 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:53.344759 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:59:53.446600 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:53.446568 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:59:53.956731 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:53.956700 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:59:54.126816 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:54.126786 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:59:55.277398 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:55.277361 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:59:55.393426 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:55.393396 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:59:55.720171 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:55.720132 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 03:59:55.817962 ip-10-0-140-30 kubenswrapper[2568]: I0911 03:59:55.817917 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:00:00.293352 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:00.293324 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-65b869c898-d5nnh" Sep 11 04:00:02.818677 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:02.818645 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:00:03.017137 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:03.017098 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:00:03.167475 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:03.167446 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5f8f45d65c-cn2gs"] Sep 11 04:00:03.175365 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:03.175344 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5f8f45d65c-cn2gs" Sep 11 04:00:03.188171 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:03.188150 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 11 04:00:03.188456 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:03.188441 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 11 04:00:03.188692 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:03.188673 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 11 04:00:03.189277 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:03.189257 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 11 04:00:03.189388 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:03.189327 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-t7hw4\"" Sep 11 04:00:03.192034 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:03.192014 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 11 04:00:03.192191 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:03.192170 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5f8f45d65c-cn2gs"] Sep 11 04:00:03.289036 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:03.288996 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b0f71181-cd15-4756-bd67-94d3334dc883-cert\") pod \"lws-controller-manager-5f8f45d65c-cn2gs\" (UID: \"b0f71181-cd15-4756-bd67-94d3334dc883\") " pod="openshift-lws-operator/lws-controller-manager-5f8f45d65c-cn2gs" Sep 11 04:00:03.289238 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:03.289116 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvqd4\" (UniqueName: \"kubernetes.io/projected/b0f71181-cd15-4756-bd67-94d3334dc883-kube-api-access-lvqd4\") pod \"lws-controller-manager-5f8f45d65c-cn2gs\" (UID: \"b0f71181-cd15-4756-bd67-94d3334dc883\") " pod="openshift-lws-operator/lws-controller-manager-5f8f45d65c-cn2gs" Sep 11 04:00:03.289238 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:03.289168 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b0f71181-cd15-4756-bd67-94d3334dc883-manager-config\") pod \"lws-controller-manager-5f8f45d65c-cn2gs\" (UID: \"b0f71181-cd15-4756-bd67-94d3334dc883\") " pod="openshift-lws-operator/lws-controller-manager-5f8f45d65c-cn2gs" Sep 11 04:00:03.289238 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:03.289190 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b0f71181-cd15-4756-bd67-94d3334dc883-metrics-cert\") pod \"lws-controller-manager-5f8f45d65c-cn2gs\" (UID: \"b0f71181-cd15-4756-bd67-94d3334dc883\") " pod="openshift-lws-operator/lws-controller-manager-5f8f45d65c-cn2gs" Sep 11 04:00:03.390322 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:03.390282 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvqd4\" (UniqueName: \"kubernetes.io/projected/b0f71181-cd15-4756-bd67-94d3334dc883-kube-api-access-lvqd4\") pod \"lws-controller-manager-5f8f45d65c-cn2gs\" (UID: \"b0f71181-cd15-4756-bd67-94d3334dc883\") " pod="openshift-lws-operator/lws-controller-manager-5f8f45d65c-cn2gs" Sep 11 04:00:03.390505 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:03.390339 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b0f71181-cd15-4756-bd67-94d3334dc883-manager-config\") pod \"lws-controller-manager-5f8f45d65c-cn2gs\" (UID: \"b0f71181-cd15-4756-bd67-94d3334dc883\") " pod="openshift-lws-operator/lws-controller-manager-5f8f45d65c-cn2gs" Sep 11 04:00:03.390505 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:03.390370 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b0f71181-cd15-4756-bd67-94d3334dc883-metrics-cert\") pod \"lws-controller-manager-5f8f45d65c-cn2gs\" (UID: \"b0f71181-cd15-4756-bd67-94d3334dc883\") " pod="openshift-lws-operator/lws-controller-manager-5f8f45d65c-cn2gs" Sep 11 04:00:03.390505 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:03.390425 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b0f71181-cd15-4756-bd67-94d3334dc883-cert\") pod \"lws-controller-manager-5f8f45d65c-cn2gs\" (UID: \"b0f71181-cd15-4756-bd67-94d3334dc883\") " pod="openshift-lws-operator/lws-controller-manager-5f8f45d65c-cn2gs" Sep 11 04:00:03.391011 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:03.390986 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b0f71181-cd15-4756-bd67-94d3334dc883-manager-config\") pod \"lws-controller-manager-5f8f45d65c-cn2gs\" (UID: \"b0f71181-cd15-4756-bd67-94d3334dc883\") " pod="openshift-lws-operator/lws-controller-manager-5f8f45d65c-cn2gs" Sep 11 04:00:03.392869 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:03.392845 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b0f71181-cd15-4756-bd67-94d3334dc883-cert\") pod \"lws-controller-manager-5f8f45d65c-cn2gs\" (UID: \"b0f71181-cd15-4756-bd67-94d3334dc883\") " pod="openshift-lws-operator/lws-controller-manager-5f8f45d65c-cn2gs" Sep 11 04:00:03.392989 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:03.392970 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b0f71181-cd15-4756-bd67-94d3334dc883-metrics-cert\") pod \"lws-controller-manager-5f8f45d65c-cn2gs\" (UID: \"b0f71181-cd15-4756-bd67-94d3334dc883\") " pod="openshift-lws-operator/lws-controller-manager-5f8f45d65c-cn2gs" Sep 11 04:00:03.427893 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:03.427797 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:00:03.429358 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:03.429336 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvqd4\" (UniqueName: \"kubernetes.io/projected/b0f71181-cd15-4756-bd67-94d3334dc883-kube-api-access-lvqd4\") pod \"lws-controller-manager-5f8f45d65c-cn2gs\" (UID: \"b0f71181-cd15-4756-bd67-94d3334dc883\") " pod="openshift-lws-operator/lws-controller-manager-5f8f45d65c-cn2gs" Sep 11 04:00:03.484011 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:03.483983 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5f8f45d65c-cn2gs" Sep 11 04:00:03.754848 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:03.754770 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5f8f45d65c-cn2gs"] Sep 11 04:00:03.758814 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:00:03.758782 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0f71181_cd15_4756_bd67_94d3334dc883.slice/crio-22bd6e09c4251e2fd4a39fabbcd57106bdf44a431345495baafd5ce36d55ae38 WatchSource:0}: Error finding container 22bd6e09c4251e2fd4a39fabbcd57106bdf44a431345495baafd5ce36d55ae38: Status 404 returned error can't find the container with id 22bd6e09c4251e2fd4a39fabbcd57106bdf44a431345495baafd5ce36d55ae38 Sep 11 04:00:04.012747 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:04.012666 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:00:04.090718 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:04.090683 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:00:04.344742 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:04.344658 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5f8f45d65c-cn2gs" event={"ID":"b0f71181-cd15-4756-bd67-94d3334dc883","Type":"ContainerStarted","Data":"22bd6e09c4251e2fd4a39fabbcd57106bdf44a431345495baafd5ce36d55ae38"} Sep 11 04:00:04.898407 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:04.898370 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:00:04.991988 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:04.991949 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:00:05.963465 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:05.963432 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:00:09.365347 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:09.365312 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5f8f45d65c-cn2gs" event={"ID":"b0f71181-cd15-4756-bd67-94d3334dc883","Type":"ContainerStarted","Data":"7d659528cc2518cc40196bcbbb0d130ae05213119ef4a3cfbdfad86921f2a8f9"} Sep 11 04:00:09.365811 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:09.365421 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5f8f45d65c-cn2gs" Sep 11 04:00:09.382973 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:09.382925 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5f8f45d65c-cn2gs" podStartSLOduration=1.533786565 podStartE2EDuration="6.38291161s" totalImagesPullingTime="4.849125045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:00:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:00:09.3819118 +0000 UTC m=+424.135543215" watchObservedRunningTime="2026-09-11 04:00:09.38291161 +0000 UTC m=+424.136543013" Sep 11 04:00:10.479787 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:10.479751 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:00:10.580599 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:10.580567 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:00:10.726197 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:10.726163 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc28v55"] Sep 11 04:00:10.729721 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:10.729700 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc28v55" Sep 11 04:00:10.732396 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:10.732342 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 11 04:00:10.732396 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:10.732362 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-22dtx\"" Sep 11 04:00:10.732555 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:10.732428 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 11 04:00:10.732555 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:10.732475 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 11 04:00:10.744316 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:10.744294 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc28v55"] Sep 11 04:00:10.756769 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:10.756740 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:00:10.845450 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:10.845417 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brgxs\" (UniqueName: \"kubernetes.io/projected/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-kube-api-access-brgxs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc28v55\" (UID: \"ff80e193-9c38-4f74-a9db-ab69a8bd5ded\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc28v55" Sep 11 04:00:10.845631 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:10.845468 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc28v55\" (UID: \"ff80e193-9c38-4f74-a9db-ab69a8bd5ded\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc28v55" Sep 11 04:00:10.845631 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:10.845496 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc28v55\" (UID: \"ff80e193-9c38-4f74-a9db-ab69a8bd5ded\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc28v55" Sep 11 04:00:10.845631 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:10.845562 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc28v55\" (UID: \"ff80e193-9c38-4f74-a9db-ab69a8bd5ded\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc28v55" Sep 11 04:00:10.845631 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:10.845590 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc28v55\" (UID: \"ff80e193-9c38-4f74-a9db-ab69a8bd5ded\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc28v55" Sep 11 04:00:10.845833 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:10.845633 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc28v55\" (UID: \"ff80e193-9c38-4f74-a9db-ab69a8bd5ded\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc28v55" Sep 11 04:00:10.845833 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:10.845657 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc28v55\" (UID: \"ff80e193-9c38-4f74-a9db-ab69a8bd5ded\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc28v55" Sep 11 04:00:10.845833 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:10.845689 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc28v55\" (UID: \"ff80e193-9c38-4f74-a9db-ab69a8bd5ded\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc28v55" Sep 11 04:00:10.845833 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:10.845785 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc28v55\" (UID: \"ff80e193-9c38-4f74-a9db-ab69a8bd5ded\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc28v55" Sep 11 04:00:10.946731 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:10.946690 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc28v55\" (UID: \"ff80e193-9c38-4f74-a9db-ab69a8bd5ded\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc28v55" Sep 11 04:00:10.946731 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:10.946730 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc28v55\" (UID: \"ff80e193-9c38-4f74-a9db-ab69a8bd5ded\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc28v55" Sep 11 04:00:10.946969 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:10.946756 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc28v55\" (UID: \"ff80e193-9c38-4f74-a9db-ab69a8bd5ded\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc28v55" Sep 11 04:00:10.946969 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:10.946782 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc28v55\" (UID: \"ff80e193-9c38-4f74-a9db-ab69a8bd5ded\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc28v55" Sep 11 04:00:10.946969 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:10.946805 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc28v55\" (UID: \"ff80e193-9c38-4f74-a9db-ab69a8bd5ded\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc28v55" Sep 11 04:00:10.946969 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:10.946840 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc28v55\" (UID: \"ff80e193-9c38-4f74-a9db-ab69a8bd5ded\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc28v55" Sep 11 04:00:10.946969 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:10.946877 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brgxs\" (UniqueName: \"kubernetes.io/projected/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-kube-api-access-brgxs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc28v55\" (UID: \"ff80e193-9c38-4f74-a9db-ab69a8bd5ded\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc28v55" Sep 11 04:00:10.946969 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:10.946916 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc28v55\" (UID: \"ff80e193-9c38-4f74-a9db-ab69a8bd5ded\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc28v55" Sep 11 04:00:10.946969 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:10.946937 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc28v55\" (UID: \"ff80e193-9c38-4f74-a9db-ab69a8bd5ded\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc28v55" Sep 11 04:00:10.947344 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:10.947184 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc28v55\" (UID: \"ff80e193-9c38-4f74-a9db-ab69a8bd5ded\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc28v55" Sep 11 04:00:10.947976 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:10.947947 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc28v55\" (UID: \"ff80e193-9c38-4f74-a9db-ab69a8bd5ded\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc28v55" Sep 11 04:00:10.947976 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:10.947958 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc28v55\" (UID: \"ff80e193-9c38-4f74-a9db-ab69a8bd5ded\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc28v55" Sep 11 04:00:10.948169 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:10.948041 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc28v55\" (UID: \"ff80e193-9c38-4f74-a9db-ab69a8bd5ded\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc28v55" Sep 11 04:00:10.948265 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:10.948242 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc28v55\" (UID: \"ff80e193-9c38-4f74-a9db-ab69a8bd5ded\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc28v55" Sep 11 04:00:10.949950 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:10.949922 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc28v55\" (UID: \"ff80e193-9c38-4f74-a9db-ab69a8bd5ded\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc28v55" Sep 11 04:00:10.955634 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:10.955609 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc28v55\" (UID: \"ff80e193-9c38-4f74-a9db-ab69a8bd5ded\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc28v55" Sep 11 04:00:10.957069 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:10.957020 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc28v55\" (UID: \"ff80e193-9c38-4f74-a9db-ab69a8bd5ded\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc28v55" Sep 11 04:00:10.957260 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:10.957239 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brgxs\" (UniqueName: \"kubernetes.io/projected/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-kube-api-access-brgxs\") pod \"data-science-gateway-data-science-gateway-class-6745bb66bc28v55\" (UID: \"ff80e193-9c38-4f74-a9db-ab69a8bd5ded\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc28v55" Sep 11 04:00:10.989499 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:10.989434 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:00:11.039377 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:11.039355 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc28v55" Sep 11 04:00:11.054632 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:11.054592 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:00:11.318657 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:11.318628 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc28v55"] Sep 11 04:00:11.320731 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:00:11.320703 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff80e193_9c38_4f74_a9db_ab69a8bd5ded.slice/crio-75ba04999b01b8cace80dd07a7595cd306be71a25e0f1beb62088b8dd15319ae WatchSource:0}: Error finding container 75ba04999b01b8cace80dd07a7595cd306be71a25e0f1beb62088b8dd15319ae: Status 404 returned error can't find the container with id 75ba04999b01b8cace80dd07a7595cd306be71a25e0f1beb62088b8dd15319ae Sep 11 04:00:11.373544 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:11.373510 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc28v55" event={"ID":"ff80e193-9c38-4f74-a9db-ab69a8bd5ded","Type":"ContainerStarted","Data":"75ba04999b01b8cace80dd07a7595cd306be71a25e0f1beb62088b8dd15319ae"} Sep 11 04:00:13.923869 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:13.923828 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-f6fj4"] Sep 11 04:00:13.930795 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:13.930765 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-f6fj4" Sep 11 04:00:13.933302 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:13.933274 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 11 04:00:13.934110 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:13.933954 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-7v7xt\"" Sep 11 04:00:13.935040 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:13.935013 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-f6fj4"] Sep 11 04:00:13.992200 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:13.992118 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:00:14.073524 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:14.073485 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be71696e-0b7d-4584-9761-4d5a7cf3bd5f-cert\") pod \"odh-model-controller-7998c67dc4-f6fj4\" (UID: \"be71696e-0b7d-4584-9761-4d5a7cf3bd5f\") " pod="opendatahub/odh-model-controller-7998c67dc4-f6fj4" Sep 11 04:00:14.073729 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:14.073574 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w996s\" (UniqueName: \"kubernetes.io/projected/be71696e-0b7d-4584-9761-4d5a7cf3bd5f-kube-api-access-w996s\") pod \"odh-model-controller-7998c67dc4-f6fj4\" (UID: \"be71696e-0b7d-4584-9761-4d5a7cf3bd5f\") " pod="opendatahub/odh-model-controller-7998c67dc4-f6fj4" Sep 11 04:00:14.092735 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:14.092693 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:00:14.175110 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:14.175005 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be71696e-0b7d-4584-9761-4d5a7cf3bd5f-cert\") pod \"odh-model-controller-7998c67dc4-f6fj4\" (UID: \"be71696e-0b7d-4584-9761-4d5a7cf3bd5f\") " pod="opendatahub/odh-model-controller-7998c67dc4-f6fj4" Sep 11 04:00:14.175281 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:14.175113 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w996s\" (UniqueName: \"kubernetes.io/projected/be71696e-0b7d-4584-9761-4d5a7cf3bd5f-kube-api-access-w996s\") pod \"odh-model-controller-7998c67dc4-f6fj4\" (UID: \"be71696e-0b7d-4584-9761-4d5a7cf3bd5f\") " pod="opendatahub/odh-model-controller-7998c67dc4-f6fj4" Sep 11 04:00:14.175281 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:00:14.175178 2568 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 11 04:00:14.175397 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:00:14.175272 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be71696e-0b7d-4584-9761-4d5a7cf3bd5f-cert podName:be71696e-0b7d-4584-9761-4d5a7cf3bd5f nodeName:}" failed. No retries permitted until 2026-09-11 04:00:14.675250234 +0000 UTC m=+429.428881634 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/be71696e-0b7d-4584-9761-4d5a7cf3bd5f-cert") pod "odh-model-controller-7998c67dc4-f6fj4" (UID: "be71696e-0b7d-4584-9761-4d5a7cf3bd5f") : secret "odh-model-controller-webhook-cert" not found Sep 11 04:00:14.186927 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:14.186886 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w996s\" (UniqueName: \"kubernetes.io/projected/be71696e-0b7d-4584-9761-4d5a7cf3bd5f-kube-api-access-w996s\") pod \"odh-model-controller-7998c67dc4-f6fj4\" (UID: \"be71696e-0b7d-4584-9761-4d5a7cf3bd5f\") " pod="opendatahub/odh-model-controller-7998c67dc4-f6fj4" Sep 11 04:00:14.527721 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:14.527648 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-v24qv"] Sep 11 04:00:14.533320 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:14.533286 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:00:14.536402 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:14.536380 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-v24qv" Sep 11 04:00:14.538948 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:14.538755 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 11 04:00:14.538948 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:14.538823 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-26bvc\"" Sep 11 04:00:14.541953 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:14.541894 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-v24qv"] Sep 11 04:00:14.595593 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:14.595557 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:00:14.666011 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:14.665966 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 04:00:14.666165 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:14.666073 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 04:00:14.666165 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:14.666120 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 04:00:14.679647 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:14.679619 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be71696e-0b7d-4584-9761-4d5a7cf3bd5f-cert\") pod \"odh-model-controller-7998c67dc4-f6fj4\" (UID: \"be71696e-0b7d-4584-9761-4d5a7cf3bd5f\") " pod="opendatahub/odh-model-controller-7998c67dc4-f6fj4" Sep 11 04:00:14.679761 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:14.679656 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0f7ac31-8a89-46b9-b805-4c19ecc6eb8f-tls-certs\") pod \"model-serving-api-54d89bd5fc-v24qv\" (UID: \"b0f7ac31-8a89-46b9-b805-4c19ecc6eb8f\") " pod="opendatahub/model-serving-api-54d89bd5fc-v24qv" Sep 11 04:00:14.679761 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:14.679741 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzq8m\" (UniqueName: \"kubernetes.io/projected/b0f7ac31-8a89-46b9-b805-4c19ecc6eb8f-kube-api-access-rzq8m\") pod \"model-serving-api-54d89bd5fc-v24qv\" (UID: \"b0f7ac31-8a89-46b9-b805-4c19ecc6eb8f\") " pod="opendatahub/model-serving-api-54d89bd5fc-v24qv" Sep 11 04:00:14.682226 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:14.682200 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/be71696e-0b7d-4584-9761-4d5a7cf3bd5f-cert\") pod \"odh-model-controller-7998c67dc4-f6fj4\" (UID: \"be71696e-0b7d-4584-9761-4d5a7cf3bd5f\") " pod="opendatahub/odh-model-controller-7998c67dc4-f6fj4" Sep 11 04:00:14.780926 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:14.780835 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzq8m\" (UniqueName: \"kubernetes.io/projected/b0f7ac31-8a89-46b9-b805-4c19ecc6eb8f-kube-api-access-rzq8m\") pod \"model-serving-api-54d89bd5fc-v24qv\" (UID: \"b0f7ac31-8a89-46b9-b805-4c19ecc6eb8f\") " pod="opendatahub/model-serving-api-54d89bd5fc-v24qv" Sep 11 04:00:14.780926 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:14.780888 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0f7ac31-8a89-46b9-b805-4c19ecc6eb8f-tls-certs\") pod \"model-serving-api-54d89bd5fc-v24qv\" (UID: \"b0f7ac31-8a89-46b9-b805-4c19ecc6eb8f\") " pod="opendatahub/model-serving-api-54d89bd5fc-v24qv" Sep 11 04:00:14.781188 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:00:14.781038 2568 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Sep 11 04:00:14.781188 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:00:14.781129 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0f7ac31-8a89-46b9-b805-4c19ecc6eb8f-tls-certs podName:b0f7ac31-8a89-46b9-b805-4c19ecc6eb8f nodeName:}" failed. No retries permitted until 2026-09-11 04:00:15.281106768 +0000 UTC m=+430.034738153 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b0f7ac31-8a89-46b9-b805-4c19ecc6eb8f-tls-certs") pod "model-serving-api-54d89bd5fc-v24qv" (UID: "b0f7ac31-8a89-46b9-b805-4c19ecc6eb8f") : secret "model-serving-api-tls" not found Sep 11 04:00:14.789470 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:14.789443 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzq8m\" (UniqueName: \"kubernetes.io/projected/b0f7ac31-8a89-46b9-b805-4c19ecc6eb8f-kube-api-access-rzq8m\") pod \"model-serving-api-54d89bd5fc-v24qv\" (UID: \"b0f7ac31-8a89-46b9-b805-4c19ecc6eb8f\") " pod="opendatahub/model-serving-api-54d89bd5fc-v24qv" Sep 11 04:00:14.844407 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:14.844364 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-f6fj4" Sep 11 04:00:15.142498 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:15.142469 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-f6fj4"] Sep 11 04:00:15.144858 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:00:15.144822 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe71696e_0b7d_4584_9761_4d5a7cf3bd5f.slice/crio-ffaa3aecf5263c5df6a224ee192ebc9ee946dde0a09a370a02e6c65558bc6c95 WatchSource:0}: Error finding container ffaa3aecf5263c5df6a224ee192ebc9ee946dde0a09a370a02e6c65558bc6c95: Status 404 returned error can't find the container with id ffaa3aecf5263c5df6a224ee192ebc9ee946dde0a09a370a02e6c65558bc6c95 Sep 11 04:00:15.284919 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:15.284886 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0f7ac31-8a89-46b9-b805-4c19ecc6eb8f-tls-certs\") pod \"model-serving-api-54d89bd5fc-v24qv\" (UID: \"b0f7ac31-8a89-46b9-b805-4c19ecc6eb8f\") " pod="opendatahub/model-serving-api-54d89bd5fc-v24qv" Sep 11 04:00:15.285105 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:00:15.285027 2568 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Sep 11 04:00:15.285172 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:00:15.285111 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0f7ac31-8a89-46b9-b805-4c19ecc6eb8f-tls-certs podName:b0f7ac31-8a89-46b9-b805-4c19ecc6eb8f nodeName:}" failed. No retries permitted until 2026-09-11 04:00:16.2850923 +0000 UTC m=+431.038723682 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/b0f7ac31-8a89-46b9-b805-4c19ecc6eb8f-tls-certs") pod "model-serving-api-54d89bd5fc-v24qv" (UID: "b0f7ac31-8a89-46b9-b805-4c19ecc6eb8f") : secret "model-serving-api-tls" not found Sep 11 04:00:15.401493 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:15.401250 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc28v55" event={"ID":"ff80e193-9c38-4f74-a9db-ab69a8bd5ded","Type":"ContainerStarted","Data":"ffa0c382d689e4a4583c5b151f1f419dea42ad6d53ab2b01eafc28b42337d9a5"} Sep 11 04:00:15.402560 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:15.402531 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-f6fj4" event={"ID":"be71696e-0b7d-4584-9761-4d5a7cf3bd5f","Type":"ContainerStarted","Data":"ffaa3aecf5263c5df6a224ee192ebc9ee946dde0a09a370a02e6c65558bc6c95"} Sep 11 04:00:15.421144 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:15.421099 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc28v55" podStartSLOduration=2.078259125 podStartE2EDuration="5.42108512s" totalImagesPullingTime="3.342825995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:00:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:00:15.419683827 +0000 UTC m=+430.173315228" watchObservedRunningTime="2026-09-11 04:00:15.42108512 +0000 UTC m=+430.174716522" Sep 11 04:00:15.641271 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:15.641230 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc28v55"] Sep 11 04:00:15.668789 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:15.668695 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:00:16.040190 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:16.040094 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc28v55" Sep 11 04:00:16.041869 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:16.041821 2568 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bc28v55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.21:15021/healthz/ready\": dial tcp 10.132.0.21:15021: connect: connection refused" start-of-body= Sep 11 04:00:16.042026 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:16.041907 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc28v55" podUID="ff80e193-9c38-4f74-a9db-ab69a8bd5ded" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.21:15021/healthz/ready\": dial tcp 10.132.0.21:15021: connect: connection refused" Sep 11 04:00:16.293957 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:16.293860 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0f7ac31-8a89-46b9-b805-4c19ecc6eb8f-tls-certs\") pod \"model-serving-api-54d89bd5fc-v24qv\" (UID: \"b0f7ac31-8a89-46b9-b805-4c19ecc6eb8f\") " pod="opendatahub/model-serving-api-54d89bd5fc-v24qv" Sep 11 04:00:16.297309 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:16.297273 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b0f7ac31-8a89-46b9-b805-4c19ecc6eb8f-tls-certs\") pod \"model-serving-api-54d89bd5fc-v24qv\" (UID: \"b0f7ac31-8a89-46b9-b805-4c19ecc6eb8f\") " pod="opendatahub/model-serving-api-54d89bd5fc-v24qv" Sep 11 04:00:16.355013 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:16.354974 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-v24qv" Sep 11 04:00:16.650813 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:16.650751 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-v24qv"] Sep 11 04:00:17.040324 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:17.040284 2568 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-6745bb66bc28v55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.21:15021/healthz/ready\": dial tcp 10.132.0.21:15021: connect: connection refused" start-of-body= Sep 11 04:00:17.040513 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:17.040357 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc28v55" podUID="ff80e193-9c38-4f74-a9db-ab69a8bd5ded" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.21:15021/healthz/ready\": dial tcp 10.132.0.21:15021: connect: connection refused" Sep 11 04:00:17.415010 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:17.414967 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-v24qv" event={"ID":"b0f7ac31-8a89-46b9-b805-4c19ecc6eb8f","Type":"ContainerStarted","Data":"cab6d86ae9ffd00038b8070eb9e46cf37dc9c3506ff7860099cb2456a91320b7"} Sep 11 04:00:17.415449 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:17.415081 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc28v55" podUID="ff80e193-9c38-4f74-a9db-ab69a8bd5ded" containerName="istio-proxy" containerID="cri-o://ffa0c382d689e4a4583c5b151f1f419dea42ad6d53ab2b01eafc28b42337d9a5" gracePeriod=30 Sep 11 04:00:18.171890 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:18.171856 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:00:18.223842 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:18.223796 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:00:18.423369 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:18.423328 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-f6fj4" event={"ID":"be71696e-0b7d-4584-9761-4d5a7cf3bd5f","Type":"ContainerStarted","Data":"ef06bbba11086d370aafa31174509dd14a408a72d2507d87464a0e197e4af9d0"} Sep 11 04:00:18.423853 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:18.423534 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-f6fj4" Sep 11 04:00:18.445170 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:18.445107 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-f6fj4" podStartSLOduration=2.490177107 podStartE2EDuration="5.445083095s" totalImagesPullingTime="2.954905988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:00:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:00:18.441324877 +0000 UTC m=+433.194956281" watchObservedRunningTime="2026-09-11 04:00:18.445083095 +0000 UTC m=+433.198714492" Sep 11 04:00:20.371646 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:20.371561 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5f8f45d65c-cn2gs" Sep 11 04:00:20.434421 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:20.434381 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-v24qv" event={"ID":"b0f7ac31-8a89-46b9-b805-4c19ecc6eb8f","Type":"ContainerStarted","Data":"6662d717410fd924cff0c6bcef3df7a0edf356fa9aa88b15165b1d31d1868d4b"} Sep 11 04:00:20.434575 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:20.434531 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-v24qv" Sep 11 04:00:20.462085 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:20.462020 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-v24qv" podStartSLOduration=2.997889526 podStartE2EDuration="6.462004745s" totalImagesPullingTime="3.464115219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:00:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:00:20.460873431 +0000 UTC m=+435.214504832" watchObservedRunningTime="2026-09-11 04:00:20.462004745 +0000 UTC m=+435.215636146" Sep 11 04:00:22.657688 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:22.657666 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc28v55" Sep 11 04:00:22.748612 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:22.748536 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-istiod-ca-cert\") pod \"ff80e193-9c38-4f74-a9db-ab69a8bd5ded\" (UID: \"ff80e193-9c38-4f74-a9db-ab69a8bd5ded\") " Sep 11 04:00:22.748612 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:22.748580 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brgxs\" (UniqueName: \"kubernetes.io/projected/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-kube-api-access-brgxs\") pod \"ff80e193-9c38-4f74-a9db-ab69a8bd5ded\" (UID: \"ff80e193-9c38-4f74-a9db-ab69a8bd5ded\") " Sep 11 04:00:22.748612 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:22.748607 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-workload-socket\") pod \"ff80e193-9c38-4f74-a9db-ab69a8bd5ded\" (UID: \"ff80e193-9c38-4f74-a9db-ab69a8bd5ded\") " Sep 11 04:00:22.748903 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:22.748632 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-istio-token\") pod \"ff80e193-9c38-4f74-a9db-ab69a8bd5ded\" (UID: \"ff80e193-9c38-4f74-a9db-ab69a8bd5ded\") " Sep 11 04:00:22.748903 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:22.748655 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-workload-certs\") pod \"ff80e193-9c38-4f74-a9db-ab69a8bd5ded\" (UID: \"ff80e193-9c38-4f74-a9db-ab69a8bd5ded\") " Sep 11 04:00:22.748903 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:22.748685 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-istio-podinfo\") pod \"ff80e193-9c38-4f74-a9db-ab69a8bd5ded\" (UID: \"ff80e193-9c38-4f74-a9db-ab69a8bd5ded\") " Sep 11 04:00:22.748903 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:22.748722 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-istio-data\") pod \"ff80e193-9c38-4f74-a9db-ab69a8bd5ded\" (UID: \"ff80e193-9c38-4f74-a9db-ab69a8bd5ded\") " Sep 11 04:00:22.748903 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:22.748753 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-credential-socket\") pod \"ff80e193-9c38-4f74-a9db-ab69a8bd5ded\" (UID: \"ff80e193-9c38-4f74-a9db-ab69a8bd5ded\") " Sep 11 04:00:22.748903 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:22.748800 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-istio-envoy\") pod \"ff80e193-9c38-4f74-a9db-ab69a8bd5ded\" (UID: \"ff80e193-9c38-4f74-a9db-ab69a8bd5ded\") " Sep 11 04:00:22.749126 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:22.748920 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ff80e193-9c38-4f74-a9db-ab69a8bd5ded" (UID: "ff80e193-9c38-4f74-a9db-ab69a8bd5ded"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 04:00:22.749126 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:22.749079 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-istiod-ca-cert\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:00:22.750287 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:22.750257 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ff80e193-9c38-4f74-a9db-ab69a8bd5ded" (UID: "ff80e193-9c38-4f74-a9db-ab69a8bd5ded"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:00:22.750426 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:22.750343 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-istio-data" (OuterVolumeSpecName: "istio-data") pod "ff80e193-9c38-4f74-a9db-ab69a8bd5ded" (UID: "ff80e193-9c38-4f74-a9db-ab69a8bd5ded"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:00:22.750426 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:22.750397 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ff80e193-9c38-4f74-a9db-ab69a8bd5ded" (UID: "ff80e193-9c38-4f74-a9db-ab69a8bd5ded"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:00:22.750543 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:22.750477 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ff80e193-9c38-4f74-a9db-ab69a8bd5ded" (UID: "ff80e193-9c38-4f74-a9db-ab69a8bd5ded"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:00:22.751219 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:22.751196 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-kube-api-access-brgxs" (OuterVolumeSpecName: "kube-api-access-brgxs") pod "ff80e193-9c38-4f74-a9db-ab69a8bd5ded" (UID: "ff80e193-9c38-4f74-a9db-ab69a8bd5ded"). InnerVolumeSpecName "kube-api-access-brgxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:00:22.752153 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:22.752135 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ff80e193-9c38-4f74-a9db-ab69a8bd5ded" (UID: "ff80e193-9c38-4f74-a9db-ab69a8bd5ded"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:00:22.752497 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:22.752466 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ff80e193-9c38-4f74-a9db-ab69a8bd5ded" (UID: "ff80e193-9c38-4f74-a9db-ab69a8bd5ded"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 04:00:22.752606 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:22.752474 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-istio-token" (OuterVolumeSpecName: "istio-token") pod "ff80e193-9c38-4f74-a9db-ab69a8bd5ded" (UID: "ff80e193-9c38-4f74-a9db-ab69a8bd5ded"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:00:22.849771 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:22.849734 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brgxs\" (UniqueName: \"kubernetes.io/projected/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-kube-api-access-brgxs\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:00:22.849771 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:22.849766 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-workload-socket\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:00:22.849771 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:22.849776 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-istio-token\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:00:22.850014 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:22.849784 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-workload-certs\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:00:22.850014 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:22.849793 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-istio-podinfo\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:00:22.850014 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:22.849801 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-istio-data\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:00:22.850014 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:22.849809 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-credential-socket\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:00:22.850014 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:22.849817 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff80e193-9c38-4f74-a9db-ab69a8bd5ded-istio-envoy\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:00:23.446655 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:23.446615 2568 generic.go:356] "Generic (PLEG): container finished" podID="ff80e193-9c38-4f74-a9db-ab69a8bd5ded" containerID="ffa0c382d689e4a4583c5b151f1f419dea42ad6d53ab2b01eafc28b42337d9a5" exitCode=0 Sep 11 04:00:23.446655 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:23.446663 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc28v55" Sep 11 04:00:23.446867 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:23.446707 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc28v55" event={"ID":"ff80e193-9c38-4f74-a9db-ab69a8bd5ded","Type":"ContainerDied","Data":"ffa0c382d689e4a4583c5b151f1f419dea42ad6d53ab2b01eafc28b42337d9a5"} Sep 11 04:00:23.446867 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:23.446756 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc28v55" event={"ID":"ff80e193-9c38-4f74-a9db-ab69a8bd5ded","Type":"ContainerDied","Data":"75ba04999b01b8cace80dd07a7595cd306be71a25e0f1beb62088b8dd15319ae"} Sep 11 04:00:23.446867 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:23.446779 2568 scope.go:117] "RemoveContainer" containerID="ffa0c382d689e4a4583c5b151f1f419dea42ad6d53ab2b01eafc28b42337d9a5" Sep 11 04:00:23.455972 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:23.455953 2568 scope.go:117] "RemoveContainer" containerID="ffa0c382d689e4a4583c5b151f1f419dea42ad6d53ab2b01eafc28b42337d9a5" Sep 11 04:00:23.456254 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:00:23.456237 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffa0c382d689e4a4583c5b151f1f419dea42ad6d53ab2b01eafc28b42337d9a5\": container with ID starting with ffa0c382d689e4a4583c5b151f1f419dea42ad6d53ab2b01eafc28b42337d9a5 not found: ID does not exist" containerID="ffa0c382d689e4a4583c5b151f1f419dea42ad6d53ab2b01eafc28b42337d9a5" Sep 11 04:00:23.456314 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:23.456262 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffa0c382d689e4a4583c5b151f1f419dea42ad6d53ab2b01eafc28b42337d9a5"} err="failed to get container status \"ffa0c382d689e4a4583c5b151f1f419dea42ad6d53ab2b01eafc28b42337d9a5\": rpc error: code = NotFound desc = could not find container \"ffa0c382d689e4a4583c5b151f1f419dea42ad6d53ab2b01eafc28b42337d9a5\": container with ID starting with ffa0c382d689e4a4583c5b151f1f419dea42ad6d53ab2b01eafc28b42337d9a5 not found: ID does not exist" Sep 11 04:00:23.468872 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:23.468844 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc28v55"] Sep 11 04:00:23.472734 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:23.472708 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6745bb66bc28v55"] Sep 11 04:00:23.767537 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:23.767446 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff80e193-9c38-4f74-a9db-ab69a8bd5ded" path="/var/lib/kubelet/pods/ff80e193-9c38-4f74-a9db-ab69a8bd5ded/volumes" Sep 11 04:00:24.998925 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:24.998890 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:00:25.114750 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:25.114717 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-plffv"] Sep 11 04:00:25.115172 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:25.115152 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff80e193-9c38-4f74-a9db-ab69a8bd5ded" containerName="istio-proxy" Sep 11 04:00:25.115172 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:25.115174 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff80e193-9c38-4f74-a9db-ab69a8bd5ded" containerName="istio-proxy" Sep 11 04:00:25.115332 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:25.115277 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff80e193-9c38-4f74-a9db-ab69a8bd5ded" containerName="istio-proxy" Sep 11 04:00:25.119789 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:25.119767 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-plffv" Sep 11 04:00:25.124727 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:25.124705 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-brdwn\"" Sep 11 04:00:25.125098 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:25.124788 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Sep 11 04:00:25.142884 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:25.142862 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-plffv"] Sep 11 04:00:25.144232 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:25.144211 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-mxmqw"] Sep 11 04:00:25.147195 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:25.147179 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mxmqw" Sep 11 04:00:25.150409 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:25.150385 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:00:25.150409 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:25.150402 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Sep 11 04:00:25.150608 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:25.150457 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-r9jtl\"" Sep 11 04:00:25.165548 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:25.165528 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-mxmqw"] Sep 11 04:00:25.270487 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:25.270387 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9jpx\" (UniqueName: \"kubernetes.io/projected/586a64b0-ea72-407e-8cac-fb88d47f146d-kube-api-access-b9jpx\") pod \"kserve-controller-manager-d6f4b65b8-plffv\" (UID: \"586a64b0-ea72-407e-8cac-fb88d47f146d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-plffv" Sep 11 04:00:25.270487 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:25.270435 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s2c7\" (UniqueName: \"kubernetes.io/projected/2970cd01-adab-426f-9dbb-96fc7affff82-kube-api-access-5s2c7\") pod \"llmisvc-controller-manager-777c6bd7d5-mxmqw\" (UID: \"2970cd01-adab-426f-9dbb-96fc7affff82\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mxmqw" Sep 11 04:00:25.270717 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:25.270531 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2970cd01-adab-426f-9dbb-96fc7affff82-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-mxmqw\" (UID: \"2970cd01-adab-426f-9dbb-96fc7affff82\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mxmqw" Sep 11 04:00:25.270717 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:25.270613 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/586a64b0-ea72-407e-8cac-fb88d47f146d-cert\") pod \"kserve-controller-manager-d6f4b65b8-plffv\" (UID: \"586a64b0-ea72-407e-8cac-fb88d47f146d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-plffv" Sep 11 04:00:25.371806 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:25.371769 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/586a64b0-ea72-407e-8cac-fb88d47f146d-cert\") pod \"kserve-controller-manager-d6f4b65b8-plffv\" (UID: \"586a64b0-ea72-407e-8cac-fb88d47f146d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-plffv" Sep 11 04:00:25.372018 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:25.371823 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9jpx\" (UniqueName: \"kubernetes.io/projected/586a64b0-ea72-407e-8cac-fb88d47f146d-kube-api-access-b9jpx\") pod \"kserve-controller-manager-d6f4b65b8-plffv\" (UID: \"586a64b0-ea72-407e-8cac-fb88d47f146d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-plffv" Sep 11 04:00:25.372018 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:25.371863 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s2c7\" (UniqueName: \"kubernetes.io/projected/2970cd01-adab-426f-9dbb-96fc7affff82-kube-api-access-5s2c7\") pod \"llmisvc-controller-manager-777c6bd7d5-mxmqw\" (UID: \"2970cd01-adab-426f-9dbb-96fc7affff82\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mxmqw" Sep 11 04:00:25.372018 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:25.371910 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2970cd01-adab-426f-9dbb-96fc7affff82-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-mxmqw\" (UID: \"2970cd01-adab-426f-9dbb-96fc7affff82\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mxmqw" Sep 11 04:00:25.374275 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:25.374252 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/586a64b0-ea72-407e-8cac-fb88d47f146d-cert\") pod \"kserve-controller-manager-d6f4b65b8-plffv\" (UID: \"586a64b0-ea72-407e-8cac-fb88d47f146d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-plffv" Sep 11 04:00:25.374391 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:25.374314 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2970cd01-adab-426f-9dbb-96fc7affff82-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-mxmqw\" (UID: \"2970cd01-adab-426f-9dbb-96fc7affff82\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mxmqw" Sep 11 04:00:25.381037 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:25.381011 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9jpx\" (UniqueName: \"kubernetes.io/projected/586a64b0-ea72-407e-8cac-fb88d47f146d-kube-api-access-b9jpx\") pod \"kserve-controller-manager-d6f4b65b8-plffv\" (UID: \"586a64b0-ea72-407e-8cac-fb88d47f146d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-plffv" Sep 11 04:00:25.381489 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:25.381468 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s2c7\" (UniqueName: \"kubernetes.io/projected/2970cd01-adab-426f-9dbb-96fc7affff82-kube-api-access-5s2c7\") pod \"llmisvc-controller-manager-777c6bd7d5-mxmqw\" (UID: \"2970cd01-adab-426f-9dbb-96fc7affff82\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mxmqw" Sep 11 04:00:25.404079 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:25.404040 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:00:25.429782 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:25.429761 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-plffv" Sep 11 04:00:25.458272 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:25.458243 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mxmqw" Sep 11 04:00:25.726866 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:25.726786 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-plffv"] Sep 11 04:00:25.729312 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:00:25.729290 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod586a64b0_ea72_407e_8cac_fb88d47f146d.slice/crio-49431b1356977f096c85a68258599ec3c346227a841dd5f7d7fb0ec77c66902f WatchSource:0}: Error finding container 49431b1356977f096c85a68258599ec3c346227a841dd5f7d7fb0ec77c66902f: Status 404 returned error can't find the container with id 49431b1356977f096c85a68258599ec3c346227a841dd5f7d7fb0ec77c66902f Sep 11 04:00:25.745306 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:25.745217 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-mxmqw"] Sep 11 04:00:25.747194 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:00:25.747171 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod2970cd01_adab_426f_9dbb_96fc7affff82.slice/crio-9648d2c6e6410765e8c2cbdc448cbc6ed0dc612b5b50f20d8cf0bbb9d7c5286f WatchSource:0}: Error finding container 9648d2c6e6410765e8c2cbdc448cbc6ed0dc612b5b50f20d8cf0bbb9d7c5286f: Status 404 returned error can't find the container with id 9648d2c6e6410765e8c2cbdc448cbc6ed0dc612b5b50f20d8cf0bbb9d7c5286f Sep 11 04:00:26.413716 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:26.413671 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:00:26.462436 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:26.462404 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mxmqw" event={"ID":"2970cd01-adab-426f-9dbb-96fc7affff82","Type":"ContainerStarted","Data":"9648d2c6e6410765e8c2cbdc448cbc6ed0dc612b5b50f20d8cf0bbb9d7c5286f"} Sep 11 04:00:26.463787 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:26.463761 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-plffv" event={"ID":"586a64b0-ea72-407e-8cac-fb88d47f146d","Type":"ContainerStarted","Data":"49431b1356977f096c85a68258599ec3c346227a841dd5f7d7fb0ec77c66902f"} Sep 11 04:00:26.510463 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:26.510428 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:00:26.611254 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:26.611221 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:00:29.430187 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:29.430152 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-f6fj4" Sep 11 04:00:29.481918 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:29.481862 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mxmqw" event={"ID":"2970cd01-adab-426f-9dbb-96fc7affff82","Type":"ContainerStarted","Data":"0bae7fcae60ae55f8a73fe2c75f4812fcdd19b11ac5367a255f33c37f45e6045"} Sep 11 04:00:29.482386 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:29.481957 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mxmqw" Sep 11 04:00:29.484069 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:29.484020 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-plffv" event={"ID":"586a64b0-ea72-407e-8cac-fb88d47f146d","Type":"ContainerStarted","Data":"ffdd62ffb7bdd4fa4aca18ea3a746d5b75c3248f21570636f65cfecc95d5b3d5"} Sep 11 04:00:29.484187 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:29.484100 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-plffv" Sep 11 04:00:29.503637 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:29.503590 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mxmqw" podStartSLOduration=1.594635917 podStartE2EDuration="4.503573887s" totalImagesPullingTime="2.90893797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:00:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:00:29.502398727 +0000 UTC m=+444.256030130" watchObservedRunningTime="2026-09-11 04:00:29.503573887 +0000 UTC m=+444.257205289" Sep 11 04:00:29.521927 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:29.521756 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-plffv" podStartSLOduration=1.598248668 podStartE2EDuration="4.521738918s" totalImagesPullingTime="2.92349025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:00:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:00:29.520991193 +0000 UTC m=+444.274622592" watchObservedRunningTime="2026-09-11 04:00:29.521738918 +0000 UTC m=+444.275370343" Sep 11 04:00:31.445568 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:00:31.445534 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-v24qv" Sep 11 04:01:00.490479 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:00.490441 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-mxmqw" Sep 11 04:01:00.493261 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:00.493242 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-plffv" Sep 11 04:01:14.957754 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:14.957708 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:01:15.055265 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:15.055224 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:01:30.248475 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:30.248439 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:01:33.023395 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:33.023356 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:01:34.099590 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:34.099552 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:01:34.673952 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:34.673922 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:01:34.818933 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:34.818898 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:01:35.421456 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:35.421416 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:01:35.821526 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:35.821443 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:01:36.218472 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:36.218438 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:01:36.373955 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:36.373920 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:01:36.477654 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:36.477560 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:01:36.903454 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:36.903422 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:01:37.002824 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:37.002786 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:01:38.933620 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:38.933587 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qbcww"] Sep 11 04:01:38.936870 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:38.936854 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qbcww" Sep 11 04:01:38.939511 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:38.939485 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 11 04:01:38.939678 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:38.939496 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 11 04:01:38.939678 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:38.939585 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-dzb9t\"" Sep 11 04:01:38.952388 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:38.952365 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qbcww"] Sep 11 04:01:39.020818 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:39.020788 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:01:39.039874 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:39.039840 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4b4ce251-ae82-48a3-8c1c-81bfcf8e7bd5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-qbcww\" (UID: \"4b4ce251-ae82-48a3-8c1c-81bfcf8e7bd5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qbcww" Sep 11 04:01:39.039973 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:39.039882 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65hxg\" (UniqueName: \"kubernetes.io/projected/4b4ce251-ae82-48a3-8c1c-81bfcf8e7bd5-kube-api-access-65hxg\") pod \"kuadrant-operator-controller-manager-76f8f686bf-qbcww\" (UID: \"4b4ce251-ae82-48a3-8c1c-81bfcf8e7bd5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qbcww" Sep 11 04:01:39.140571 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:39.140540 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4b4ce251-ae82-48a3-8c1c-81bfcf8e7bd5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-qbcww\" (UID: \"4b4ce251-ae82-48a3-8c1c-81bfcf8e7bd5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qbcww" Sep 11 04:01:39.140700 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:39.140585 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65hxg\" (UniqueName: \"kubernetes.io/projected/4b4ce251-ae82-48a3-8c1c-81bfcf8e7bd5-kube-api-access-65hxg\") pod \"kuadrant-operator-controller-manager-76f8f686bf-qbcww\" (UID: \"4b4ce251-ae82-48a3-8c1c-81bfcf8e7bd5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qbcww" Sep 11 04:01:39.141006 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:39.140986 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4b4ce251-ae82-48a3-8c1c-81bfcf8e7bd5-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-qbcww\" (UID: \"4b4ce251-ae82-48a3-8c1c-81bfcf8e7bd5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qbcww" Sep 11 04:01:39.150255 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:39.150226 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-65hxg\" (UniqueName: \"kubernetes.io/projected/4b4ce251-ae82-48a3-8c1c-81bfcf8e7bd5-kube-api-access-65hxg\") pod \"kuadrant-operator-controller-manager-76f8f686bf-qbcww\" (UID: \"4b4ce251-ae82-48a3-8c1c-81bfcf8e7bd5\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qbcww" Sep 11 04:01:39.246917 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:39.246824 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qbcww" Sep 11 04:01:39.422447 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:39.422417 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:01:39.549709 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:39.549685 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qbcww"] Sep 11 04:01:39.551884 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:01:39.551855 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b4ce251_ae82_48a3_8c1c_81bfcf8e7bd5.slice/crio-399ae95a32078192565214bfb0bff88d0e0deb52db14f07a86222e25cea059d1 WatchSource:0}: Error finding container 399ae95a32078192565214bfb0bff88d0e0deb52db14f07a86222e25cea059d1: Status 404 returned error can't find the container with id 399ae95a32078192565214bfb0bff88d0e0deb52db14f07a86222e25cea059d1 Sep 11 04:01:39.776661 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:39.776625 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qbcww" event={"ID":"4b4ce251-ae82-48a3-8c1c-81bfcf8e7bd5","Type":"ContainerStarted","Data":"399ae95a32078192565214bfb0bff88d0e0deb52db14f07a86222e25cea059d1"} Sep 11 04:01:40.861724 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:40.861686 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:01:42.471860 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:42.471814 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:01:42.575225 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:42.575185 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:01:42.673235 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:42.673200 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:01:42.849034 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:42.848948 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:01:42.943198 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:42.943163 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:01:43.800206 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:43.800173 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qbcww" event={"ID":"4b4ce251-ae82-48a3-8c1c-81bfcf8e7bd5","Type":"ContainerStarted","Data":"d656d9449127c7345549a8d27072ec996d4d8396d8bdf6292f56a8c3369ee3c5"} Sep 11 04:01:43.800586 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:43.800284 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qbcww" Sep 11 04:01:43.823144 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:43.823095 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qbcww" podStartSLOduration=1.7376109020000001 podStartE2EDuration="5.823079443s" totalImagesPullingTime="4.085468541s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:01:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:01:43.820742897 +0000 UTC m=+518.574374310" watchObservedRunningTime="2026-09-11 04:01:43.823079443 +0000 UTC m=+518.576710842" Sep 11 04:01:46.441095 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:46.441034 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:01:46.537958 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:46.537919 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-ddlnd"] Sep 11 04:01:46.541512 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:46.541486 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ddlnd" Sep 11 04:01:46.542751 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:46.542714 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:01:46.544467 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:46.544443 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Sep 11 04:01:46.544603 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:46.544444 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Sep 11 04:01:46.544603 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:46.544488 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-cpmw2\"" Sep 11 04:01:46.557260 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:46.557227 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-ddlnd"] Sep 11 04:01:46.629041 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:46.628999 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:01:46.709612 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:46.709523 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-796zz\" (UniqueName: \"kubernetes.io/projected/413cad97-0870-48a3-a7ff-07bba8820ce5-kube-api-access-796zz\") pod \"kuadrant-console-plugin-fbdd4f445-ddlnd\" (UID: \"413cad97-0870-48a3-a7ff-07bba8820ce5\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ddlnd" Sep 11 04:01:46.709785 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:46.709692 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/413cad97-0870-48a3-a7ff-07bba8820ce5-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-ddlnd\" (UID: \"413cad97-0870-48a3-a7ff-07bba8820ce5\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ddlnd" Sep 11 04:01:46.709785 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:46.709731 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/413cad97-0870-48a3-a7ff-07bba8820ce5-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-ddlnd\" (UID: \"413cad97-0870-48a3-a7ff-07bba8820ce5\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ddlnd" Sep 11 04:01:46.810230 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:46.810197 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/413cad97-0870-48a3-a7ff-07bba8820ce5-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-ddlnd\" (UID: \"413cad97-0870-48a3-a7ff-07bba8820ce5\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ddlnd" Sep 11 04:01:46.810390 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:46.810241 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/413cad97-0870-48a3-a7ff-07bba8820ce5-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-ddlnd\" (UID: \"413cad97-0870-48a3-a7ff-07bba8820ce5\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ddlnd" Sep 11 04:01:46.810390 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:46.810279 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-796zz\" (UniqueName: \"kubernetes.io/projected/413cad97-0870-48a3-a7ff-07bba8820ce5-kube-api-access-796zz\") pod \"kuadrant-console-plugin-fbdd4f445-ddlnd\" (UID: \"413cad97-0870-48a3-a7ff-07bba8820ce5\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ddlnd" Sep 11 04:01:46.810390 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:01:46.810331 2568 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Sep 11 04:01:46.810519 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:01:46.810403 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/413cad97-0870-48a3-a7ff-07bba8820ce5-plugin-serving-cert podName:413cad97-0870-48a3-a7ff-07bba8820ce5 nodeName:}" failed. No retries permitted until 2026-09-11 04:01:47.310386117 +0000 UTC m=+522.064017497 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/413cad97-0870-48a3-a7ff-07bba8820ce5-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-ddlnd" (UID: "413cad97-0870-48a3-a7ff-07bba8820ce5") : secret "plugin-serving-cert" not found Sep 11 04:01:46.811092 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:46.811069 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/413cad97-0870-48a3-a7ff-07bba8820ce5-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-ddlnd\" (UID: \"413cad97-0870-48a3-a7ff-07bba8820ce5\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ddlnd" Sep 11 04:01:46.819106 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:46.819082 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-796zz\" (UniqueName: \"kubernetes.io/projected/413cad97-0870-48a3-a7ff-07bba8820ce5-kube-api-access-796zz\") pod \"kuadrant-console-plugin-fbdd4f445-ddlnd\" (UID: \"413cad97-0870-48a3-a7ff-07bba8820ce5\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ddlnd" Sep 11 04:01:47.028969 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:47.028871 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:01:47.313831 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:47.313723 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/413cad97-0870-48a3-a7ff-07bba8820ce5-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-ddlnd\" (UID: \"413cad97-0870-48a3-a7ff-07bba8820ce5\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ddlnd" Sep 11 04:01:47.316273 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:47.316247 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/413cad97-0870-48a3-a7ff-07bba8820ce5-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-ddlnd\" (UID: \"413cad97-0870-48a3-a7ff-07bba8820ce5\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ddlnd" Sep 11 04:01:47.451599 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:47.451564 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ddlnd" Sep 11 04:01:47.722759 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:47.722734 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-ddlnd"] Sep 11 04:01:47.724821 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:01:47.724794 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod413cad97_0870_48a3_a7ff_07bba8820ce5.slice/crio-aeecb1f744fa9a3ba425ca0631c0b66856db4813a3b7cf076cc5bbdbef4b3a70 WatchSource:0}: Error finding container aeecb1f744fa9a3ba425ca0631c0b66856db4813a3b7cf076cc5bbdbef4b3a70: Status 404 returned error can't find the container with id aeecb1f744fa9a3ba425ca0631c0b66856db4813a3b7cf076cc5bbdbef4b3a70 Sep 11 04:01:47.817340 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:47.817309 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ddlnd" event={"ID":"413cad97-0870-48a3-a7ff-07bba8820ce5","Type":"ContainerStarted","Data":"aeecb1f744fa9a3ba425ca0631c0b66856db4813a3b7cf076cc5bbdbef4b3a70"} Sep 11 04:01:50.449825 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:50.449782 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:01:50.543314 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:50.543280 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:01:50.680653 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:50.680596 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:01:52.845965 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:52.845926 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ddlnd" event={"ID":"413cad97-0870-48a3-a7ff-07bba8820ce5","Type":"ContainerStarted","Data":"aef62c49300849d0e43fd1d6cd9b7ba8dc71e3ebcf9681ca11febc19164bf52d"} Sep 11 04:01:52.863792 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:52.863736 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-ddlnd" podStartSLOduration=2.470749291 podStartE2EDuration="6.863719462s" totalImagesPullingTime="4.392970171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:01:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:01:52.862215469 +0000 UTC m=+527.615846870" watchObservedRunningTime="2026-09-11 04:01:52.863719462 +0000 UTC m=+527.617350911" Sep 11 04:01:54.806465 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:54.806431 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qbcww" Sep 11 04:01:56.696137 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:56.694329 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dm56v"] Sep 11 04:01:56.699550 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:56.699526 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dm56v" Sep 11 04:01:56.718748 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:56.718720 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dm56v"] Sep 11 04:01:56.797580 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:56.797542 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9c8k\" (UniqueName: \"kubernetes.io/projected/47c946e4-cc77-4815-aca0-4bd309aa9706-kube-api-access-z9c8k\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-dm56v\" (UID: \"47c946e4-cc77-4815-aca0-4bd309aa9706\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dm56v" Sep 11 04:01:56.797739 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:56.797596 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/47c946e4-cc77-4815-aca0-4bd309aa9706-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-dm56v\" (UID: \"47c946e4-cc77-4815-aca0-4bd309aa9706\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dm56v" Sep 11 04:01:56.898464 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:56.898436 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9c8k\" (UniqueName: \"kubernetes.io/projected/47c946e4-cc77-4815-aca0-4bd309aa9706-kube-api-access-z9c8k\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-dm56v\" (UID: \"47c946e4-cc77-4815-aca0-4bd309aa9706\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dm56v" Sep 11 04:01:56.898664 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:56.898472 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/47c946e4-cc77-4815-aca0-4bd309aa9706-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-dm56v\" (UID: \"47c946e4-cc77-4815-aca0-4bd309aa9706\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dm56v" Sep 11 04:01:56.898821 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:56.898802 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/47c946e4-cc77-4815-aca0-4bd309aa9706-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-dm56v\" (UID: \"47c946e4-cc77-4815-aca0-4bd309aa9706\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dm56v" Sep 11 04:01:56.912203 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:56.912172 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9c8k\" (UniqueName: \"kubernetes.io/projected/47c946e4-cc77-4815-aca0-4bd309aa9706-kube-api-access-z9c8k\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-dm56v\" (UID: \"47c946e4-cc77-4815-aca0-4bd309aa9706\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dm56v" Sep 11 04:01:57.010227 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:57.010135 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dm56v" Sep 11 04:01:57.147523 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:57.147484 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qbcww"] Sep 11 04:01:57.147755 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:57.147727 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qbcww" podUID="4b4ce251-ae82-48a3-8c1c-81bfcf8e7bd5" containerName="manager" containerID="cri-o://d656d9449127c7345549a8d27072ec996d4d8396d8bdf6292f56a8c3369ee3c5" gracePeriod=2 Sep 11 04:01:57.159941 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:57.159905 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qbcww"] Sep 11 04:01:57.169189 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:01:57.169145 2568 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"4b4ce251-ae82-48a3-8c1c-81bfcf8e7bd5\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-qbcww\": pods \"kuadrant-operator-controller-manager-76f8f686bf-qbcww\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qbcww" Sep 11 04:01:57.178234 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:57.178205 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dm56v"] Sep 11 04:01:57.183106 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:01:57.183071 2568 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b4ce251_ae82_48a3_8c1c_81bfcf8e7bd5.slice/crio-conmon-d656d9449127c7345549a8d27072ec996d4d8396d8bdf6292f56a8c3369ee3c5.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b4ce251_ae82_48a3_8c1c_81bfcf8e7bd5.slice/crio-d656d9449127c7345549a8d27072ec996d4d8396d8bdf6292f56a8c3369ee3c5.scope\": RecentStats: unable to find data in memory cache]" Sep 11 04:01:57.195106 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:57.195081 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dm56v"] Sep 11 04:01:57.213488 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:57.213461 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q8hvz"] Sep 11 04:01:57.213947 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:57.213927 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b4ce251-ae82-48a3-8c1c-81bfcf8e7bd5" containerName="manager" Sep 11 04:01:57.214036 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:57.213949 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b4ce251-ae82-48a3-8c1c-81bfcf8e7bd5" containerName="manager" Sep 11 04:01:57.214123 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:57.214039 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b4ce251-ae82-48a3-8c1c-81bfcf8e7bd5" containerName="manager" Sep 11 04:01:57.217315 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:57.217294 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q8hvz" Sep 11 04:01:57.235304 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:57.235275 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q8hvz"] Sep 11 04:01:57.240070 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:57.240028 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-46rbd"] Sep 11 04:01:57.243199 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:57.243175 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-46rbd" Sep 11 04:01:57.246549 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:57.246530 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-cts89\"" Sep 11 04:01:57.258493 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:57.258469 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-46rbd"] Sep 11 04:01:57.266705 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:01:57.266646 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-qbcww\" is forbidden: User \"system:node:ip-10-0-140-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-30.ec2.internal' and this object" podUID="4b4ce251-ae82-48a3-8c1c-81bfcf8e7bd5" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qbcww" Sep 11 04:01:57.302042 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:01:57.301683 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-qbcww\" is forbidden: User \"system:node:ip-10-0-140-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-30.ec2.internal' and this object" podUID="4b4ce251-ae82-48a3-8c1c-81bfcf8e7bd5" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qbcww" Sep 11 04:01:57.302042 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:57.301840 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/64a6175b-243d-40dd-87bd-b59e6aa8804e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-q8hvz\" (UID: \"64a6175b-243d-40dd-87bd-b59e6aa8804e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q8hvz" Sep 11 04:01:57.302042 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:57.301880 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fwzs\" (UniqueName: \"kubernetes.io/projected/64a6175b-243d-40dd-87bd-b59e6aa8804e-kube-api-access-6fwzs\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-q8hvz\" (UID: \"64a6175b-243d-40dd-87bd-b59e6aa8804e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q8hvz" Sep 11 04:01:57.302042 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:57.301938 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc7s2\" (UniqueName: \"kubernetes.io/projected/b5b8e568-1d15-43e6-a1fb-44ba5128d5c0-kube-api-access-pc7s2\") pod \"limitador-operator-controller-manager-74495c69f8-46rbd\" (UID: \"b5b8e568-1d15-43e6-a1fb-44ba5128d5c0\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-46rbd" Sep 11 04:01:57.390381 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:57.390351 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qbcww" Sep 11 04:01:57.392897 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:01:57.392867 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-qbcww\" is forbidden: User \"system:node:ip-10-0-140-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-30.ec2.internal' and this object" podUID="4b4ce251-ae82-48a3-8c1c-81bfcf8e7bd5" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qbcww" Sep 11 04:01:57.402471 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:57.402333 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4b4ce251-ae82-48a3-8c1c-81bfcf8e7bd5-extensions-socket-volume\") pod \"4b4ce251-ae82-48a3-8c1c-81bfcf8e7bd5\" (UID: \"4b4ce251-ae82-48a3-8c1c-81bfcf8e7bd5\") " Sep 11 04:01:57.402471 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:57.402373 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65hxg\" (UniqueName: \"kubernetes.io/projected/4b4ce251-ae82-48a3-8c1c-81bfcf8e7bd5-kube-api-access-65hxg\") pod \"4b4ce251-ae82-48a3-8c1c-81bfcf8e7bd5\" (UID: \"4b4ce251-ae82-48a3-8c1c-81bfcf8e7bd5\") " Sep 11 04:01:57.402651 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:57.402538 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/64a6175b-243d-40dd-87bd-b59e6aa8804e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-q8hvz\" (UID: \"64a6175b-243d-40dd-87bd-b59e6aa8804e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q8hvz" Sep 11 04:01:57.402651 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:57.402581 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fwzs\" (UniqueName: \"kubernetes.io/projected/64a6175b-243d-40dd-87bd-b59e6aa8804e-kube-api-access-6fwzs\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-q8hvz\" (UID: \"64a6175b-243d-40dd-87bd-b59e6aa8804e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q8hvz" Sep 11 04:01:57.402651 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:01:57.402630 2568 log.go:32] "RunPodSandbox from runtime service failed" err=< Sep 11 04:01:57.402651 ip-10-0-140-30 kubenswrapper[2568]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-7978cbdfbc-dm56v_kuadrant-system_47c946e4-cc77-4815-aca0-4bd309aa9706_0(b536475156f2a5a0412f9f4a9ab1a9b3dd0750741b770e38975dab5277071dcd): error adding pod kuadrant-system_kuadrant-operator-controller-manager-7978cbdfbc-dm56v to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"b536475156f2a5a0412f9f4a9ab1a9b3dd0750741b770e38975dab5277071dcd" Netns:"/var/run/netns/ed8bc2be-09d2-4fb2-adcf-0d4b3a53ccb7" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-7978cbdfbc-dm56v;K8S_POD_INFRA_CONTAINER_ID=b536475156f2a5a0412f9f4a9ab1a9b3dd0750741b770e38975dab5277071dcd;K8S_POD_UID=47c946e4-cc77-4815-aca0-4bd309aa9706" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dm56v] networking: [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dm56v/47c946e4-cc77-4815-aca0-4bd309aa9706:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dm56v b536475156f2a5a0412f9f4a9ab1a9b3dd0750741b770e38975dab5277071dcd network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:84:00:1c [10.132.0.28/23] Sep 11 04:01:57.402651 ip-10-0-140-30 kubenswrapper[2568]: ' Sep 11 04:01:57.402651 ip-10-0-140-30 kubenswrapper[2568]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Sep 11 04:01:57.402651 ip-10-0-140-30 kubenswrapper[2568]: > Sep 11 04:01:57.402961 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:57.402653 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc7s2\" (UniqueName: \"kubernetes.io/projected/b5b8e568-1d15-43e6-a1fb-44ba5128d5c0-kube-api-access-pc7s2\") pod \"limitador-operator-controller-manager-74495c69f8-46rbd\" (UID: \"b5b8e568-1d15-43e6-a1fb-44ba5128d5c0\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-46rbd" Sep 11 04:01:57.402961 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:01:57.402684 2568 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Sep 11 04:01:57.402961 ip-10-0-140-30 kubenswrapper[2568]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-7978cbdfbc-dm56v_kuadrant-system_47c946e4-cc77-4815-aca0-4bd309aa9706_0(b536475156f2a5a0412f9f4a9ab1a9b3dd0750741b770e38975dab5277071dcd): error adding pod kuadrant-system_kuadrant-operator-controller-manager-7978cbdfbc-dm56v to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"b536475156f2a5a0412f9f4a9ab1a9b3dd0750741b770e38975dab5277071dcd" Netns:"/var/run/netns/ed8bc2be-09d2-4fb2-adcf-0d4b3a53ccb7" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-7978cbdfbc-dm56v;K8S_POD_INFRA_CONTAINER_ID=b536475156f2a5a0412f9f4a9ab1a9b3dd0750741b770e38975dab5277071dcd;K8S_POD_UID=47c946e4-cc77-4815-aca0-4bd309aa9706" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dm56v] networking: [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dm56v/47c946e4-cc77-4815-aca0-4bd309aa9706:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dm56v b536475156f2a5a0412f9f4a9ab1a9b3dd0750741b770e38975dab5277071dcd network default NAD default NAD key default] failed to configure pod interface: pod deleted waiting for OVS port binding for 0a:58:0a:84:00:1c [10.132.0.28/23] Sep 11 04:01:57.402961 ip-10-0-140-30 kubenswrapper[2568]: ' Sep 11 04:01:57.402961 ip-10-0-140-30 kubenswrapper[2568]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Sep 11 04:01:57.402961 ip-10-0-140-30 kubenswrapper[2568]: > pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dm56v" Sep 11 04:01:57.402961 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:57.402881 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b4ce251-ae82-48a3-8c1c-81bfcf8e7bd5-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "4b4ce251-ae82-48a3-8c1c-81bfcf8e7bd5" (UID: "4b4ce251-ae82-48a3-8c1c-81bfcf8e7bd5"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:01:57.402961 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:57.402925 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/64a6175b-243d-40dd-87bd-b59e6aa8804e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-q8hvz\" (UID: \"64a6175b-243d-40dd-87bd-b59e6aa8804e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q8hvz" Sep 11 04:01:57.404501 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:57.404484 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b4ce251-ae82-48a3-8c1c-81bfcf8e7bd5-kube-api-access-65hxg" (OuterVolumeSpecName: "kube-api-access-65hxg") pod "4b4ce251-ae82-48a3-8c1c-81bfcf8e7bd5" (UID: "4b4ce251-ae82-48a3-8c1c-81bfcf8e7bd5"). InnerVolumeSpecName "kube-api-access-65hxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:01:57.417874 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:57.417851 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc7s2\" (UniqueName: \"kubernetes.io/projected/b5b8e568-1d15-43e6-a1fb-44ba5128d5c0-kube-api-access-pc7s2\") pod \"limitador-operator-controller-manager-74495c69f8-46rbd\" (UID: \"b5b8e568-1d15-43e6-a1fb-44ba5128d5c0\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-46rbd" Sep 11 04:01:57.422825 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:57.422802 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fwzs\" (UniqueName: \"kubernetes.io/projected/64a6175b-243d-40dd-87bd-b59e6aa8804e-kube-api-access-6fwzs\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-q8hvz\" (UID: \"64a6175b-243d-40dd-87bd-b59e6aa8804e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q8hvz" Sep 11 04:01:57.504042 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:57.504009 2568 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4b4ce251-ae82-48a3-8c1c-81bfcf8e7bd5-extensions-socket-volume\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:01:57.504042 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:57.504037 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65hxg\" (UniqueName: \"kubernetes.io/projected/4b4ce251-ae82-48a3-8c1c-81bfcf8e7bd5-kube-api-access-65hxg\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:01:57.554089 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:57.553972 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q8hvz" Sep 11 04:01:57.563037 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:57.563006 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-46rbd" Sep 11 04:01:57.768276 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:57.768241 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b4ce251-ae82-48a3-8c1c-81bfcf8e7bd5" path="/var/lib/kubelet/pods/4b4ce251-ae82-48a3-8c1c-81bfcf8e7bd5/volumes" Sep 11 04:01:57.864442 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:57.864421 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q8hvz"] Sep 11 04:01:57.866706 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:01:57.866680 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64a6175b_243d_40dd_87bd_b59e6aa8804e.slice/crio-467a4595a0da398f77e8edad4b0894af887ee22fc1dfaa93dea87e28e03ee41a WatchSource:0}: Error finding container 467a4595a0da398f77e8edad4b0894af887ee22fc1dfaa93dea87e28e03ee41a: Status 404 returned error can't find the container with id 467a4595a0da398f77e8edad4b0894af887ee22fc1dfaa93dea87e28e03ee41a Sep 11 04:01:57.868186 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:57.868162 2568 generic.go:356] "Generic (PLEG): container finished" podID="4b4ce251-ae82-48a3-8c1c-81bfcf8e7bd5" containerID="d656d9449127c7345549a8d27072ec996d4d8396d8bdf6292f56a8c3369ee3c5" exitCode=0 Sep 11 04:01:57.868274 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:57.868204 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-qbcww" Sep 11 04:01:57.868274 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:57.868263 2568 scope.go:117] "RemoveContainer" containerID="d656d9449127c7345549a8d27072ec996d4d8396d8bdf6292f56a8c3369ee3c5" Sep 11 04:01:57.868358 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:57.868290 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dm56v" Sep 11 04:01:57.876543 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:57.876522 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dm56v" Sep 11 04:01:57.877725 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:57.877707 2568 scope.go:117] "RemoveContainer" containerID="d656d9449127c7345549a8d27072ec996d4d8396d8bdf6292f56a8c3369ee3c5" Sep 11 04:01:57.878144 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:01:57.878002 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d656d9449127c7345549a8d27072ec996d4d8396d8bdf6292f56a8c3369ee3c5\": container with ID starting with d656d9449127c7345549a8d27072ec996d4d8396d8bdf6292f56a8c3369ee3c5 not found: ID does not exist" containerID="d656d9449127c7345549a8d27072ec996d4d8396d8bdf6292f56a8c3369ee3c5" Sep 11 04:01:57.878144 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:57.878027 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d656d9449127c7345549a8d27072ec996d4d8396d8bdf6292f56a8c3369ee3c5"} err="failed to get container status \"d656d9449127c7345549a8d27072ec996d4d8396d8bdf6292f56a8c3369ee3c5\": rpc error: code = NotFound desc = could not find container \"d656d9449127c7345549a8d27072ec996d4d8396d8bdf6292f56a8c3369ee3c5\": container with ID starting with d656d9449127c7345549a8d27072ec996d4d8396d8bdf6292f56a8c3369ee3c5 not found: ID does not exist" Sep 11 04:01:57.883399 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:57.883378 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-46rbd"] Sep 11 04:01:57.884602 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:01:57.884580 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-dm56v\" is forbidden: User \"system:node:ip-10-0-140-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-30.ec2.internal' and this object" podUID="47c946e4-cc77-4815-aca0-4bd309aa9706" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dm56v" Sep 11 04:01:57.885553 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:01:57.885529 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5b8e568_1d15_43e6_a1fb_44ba5128d5c0.slice/crio-30a46c9b47f60714e8f73bc6a7ab8df268b3e7c178935619b3b368f52a0d9d07 WatchSource:0}: Error finding container 30a46c9b47f60714e8f73bc6a7ab8df268b3e7c178935619b3b368f52a0d9d07: Status 404 returned error can't find the container with id 30a46c9b47f60714e8f73bc6a7ab8df268b3e7c178935619b3b368f52a0d9d07 Sep 11 04:01:57.907675 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:57.907645 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/47c946e4-cc77-4815-aca0-4bd309aa9706-extensions-socket-volume\") pod \"47c946e4-cc77-4815-aca0-4bd309aa9706\" (UID: \"47c946e4-cc77-4815-aca0-4bd309aa9706\") " Sep 11 04:01:57.907772 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:57.907698 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9c8k\" (UniqueName: \"kubernetes.io/projected/47c946e4-cc77-4815-aca0-4bd309aa9706-kube-api-access-z9c8k\") pod \"47c946e4-cc77-4815-aca0-4bd309aa9706\" (UID: \"47c946e4-cc77-4815-aca0-4bd309aa9706\") " Sep 11 04:01:57.907889 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:57.907866 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47c946e4-cc77-4815-aca0-4bd309aa9706-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "47c946e4-cc77-4815-aca0-4bd309aa9706" (UID: "47c946e4-cc77-4815-aca0-4bd309aa9706"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:01:57.909796 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:57.909771 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47c946e4-cc77-4815-aca0-4bd309aa9706-kube-api-access-z9c8k" (OuterVolumeSpecName: "kube-api-access-z9c8k") pod "47c946e4-cc77-4815-aca0-4bd309aa9706" (UID: "47c946e4-cc77-4815-aca0-4bd309aa9706"). InnerVolumeSpecName "kube-api-access-z9c8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:01:58.009029 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:58.008959 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9c8k\" (UniqueName: \"kubernetes.io/projected/47c946e4-cc77-4815-aca0-4bd309aa9706-kube-api-access-z9c8k\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:01:58.009029 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:58.008988 2568 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/47c946e4-cc77-4815-aca0-4bd309aa9706-extensions-socket-volume\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:01:58.875257 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:58.875216 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-46rbd" event={"ID":"b5b8e568-1d15-43e6-a1fb-44ba5128d5c0","Type":"ContainerStarted","Data":"30a46c9b47f60714e8f73bc6a7ab8df268b3e7c178935619b3b368f52a0d9d07"} Sep 11 04:01:58.882005 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:58.881975 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dm56v" Sep 11 04:01:58.882138 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:58.881969 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q8hvz" event={"ID":"64a6175b-243d-40dd-87bd-b59e6aa8804e","Type":"ContainerStarted","Data":"ad7d1bb8780ccb57987f2a93abfd5dc8c9fa68972760d7954406fbdd4661b6ff"} Sep 11 04:01:58.882264 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:58.882239 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q8hvz" event={"ID":"64a6175b-243d-40dd-87bd-b59e6aa8804e","Type":"ContainerStarted","Data":"467a4595a0da398f77e8edad4b0894af887ee22fc1dfaa93dea87e28e03ee41a"} Sep 11 04:01:58.885109 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:01:58.885080 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-dm56v\" is forbidden: User \"system:node:ip-10-0-140-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-30.ec2.internal' and this object" podUID="47c946e4-cc77-4815-aca0-4bd309aa9706" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dm56v" Sep 11 04:01:58.907349 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:58.907307 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q8hvz" podStartSLOduration=1.907290717 podStartE2EDuration="1.907290717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:01:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:01:58.905524047 +0000 UTC m=+533.659155450" watchObservedRunningTime="2026-09-11 04:01:58.907290717 +0000 UTC m=+533.660922121" Sep 11 04:01:58.909477 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:01:58.909456 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-dm56v\" is forbidden: User \"system:node:ip-10-0-140-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-30.ec2.internal' and this object" podUID="47c946e4-cc77-4815-aca0-4bd309aa9706" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dm56v" Sep 11 04:01:58.911422 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:01:58.911405 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-dm56v\" is forbidden: User \"system:node:ip-10-0-140-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-30.ec2.internal' and this object" podUID="47c946e4-cc77-4815-aca0-4bd309aa9706" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-dm56v" Sep 11 04:01:59.297773 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:59.297727 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:01:59.398731 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:59.398685 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:01:59.515623 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:59.515584 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:01:59.768756 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:59.768727 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47c946e4-cc77-4815-aca0-4bd309aa9706" path="/var/lib/kubelet/pods/47c946e4-cc77-4815-aca0-4bd309aa9706/volumes" Sep 11 04:01:59.888297 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:59.888260 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-46rbd" event={"ID":"b5b8e568-1d15-43e6-a1fb-44ba5128d5c0","Type":"ContainerStarted","Data":"a4676071fdcf516a01819527122e14c2ab324efc8228c31f28a2695d82a6d7c1"} Sep 11 04:01:59.888648 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:59.888357 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q8hvz" Sep 11 04:01:59.914388 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:01:59.914343 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-46rbd" podStartSLOduration=0.999775026 podStartE2EDuration="2.914329118s" totalImagesPullingTime="1.914554092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:01:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:01:59.912436802 +0000 UTC m=+534.666068203" watchObservedRunningTime="2026-09-11 04:01:59.914329118 +0000 UTC m=+534.667960522" Sep 11 04:02:00.893155 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:00.893115 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-46rbd" Sep 11 04:02:01.909321 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:01.909286 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:02:02.014463 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:02.014425 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:02:05.251509 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:05.251471 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:02:05.349332 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:05.349292 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:02:09.702403 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:09.702364 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:02:09.792193 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:09.792158 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:02:09.916766 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:09.916728 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:02:09.979743 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:09.979648 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:02:10.094722 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:10.094684 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:02:10.162393 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:10.162348 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:02:10.895463 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:10.895435 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q8hvz" Sep 11 04:02:11.267545 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:11.267514 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l9blm"] Sep 11 04:02:11.273483 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:11.273462 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l9blm" Sep 11 04:02:11.317913 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:11.317884 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3fc8812f-fd87-492c-b430-973faacc06d8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-l9blm\" (UID: \"3fc8812f-fd87-492c-b430-973faacc06d8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l9blm" Sep 11 04:02:11.318017 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:11.317936 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgm6d\" (UniqueName: \"kubernetes.io/projected/3fc8812f-fd87-492c-b430-973faacc06d8-kube-api-access-tgm6d\") pod \"kuadrant-operator-controller-manager-5769f99d9f-l9blm\" (UID: \"3fc8812f-fd87-492c-b430-973faacc06d8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l9blm" Sep 11 04:02:11.332195 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:11.332169 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l9blm"] Sep 11 04:02:11.418731 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:11.418701 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3fc8812f-fd87-492c-b430-973faacc06d8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-l9blm\" (UID: \"3fc8812f-fd87-492c-b430-973faacc06d8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l9blm" Sep 11 04:02:11.418881 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:11.418760 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgm6d\" (UniqueName: \"kubernetes.io/projected/3fc8812f-fd87-492c-b430-973faacc06d8-kube-api-access-tgm6d\") pod \"kuadrant-operator-controller-manager-5769f99d9f-l9blm\" (UID: \"3fc8812f-fd87-492c-b430-973faacc06d8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l9blm" Sep 11 04:02:11.419099 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:11.419078 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3fc8812f-fd87-492c-b430-973faacc06d8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-l9blm\" (UID: \"3fc8812f-fd87-492c-b430-973faacc06d8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l9blm" Sep 11 04:02:11.445768 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:11.445707 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgm6d\" (UniqueName: \"kubernetes.io/projected/3fc8812f-fd87-492c-b430-973faacc06d8-kube-api-access-tgm6d\") pod \"kuadrant-operator-controller-manager-5769f99d9f-l9blm\" (UID: \"3fc8812f-fd87-492c-b430-973faacc06d8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l9blm" Sep 11 04:02:11.583853 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:11.583814 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l9blm" Sep 11 04:02:11.882493 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:11.882413 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l9blm"] Sep 11 04:02:11.886408 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:02:11.886377 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fc8812f_fd87_492c_b430_973faacc06d8.slice/crio-a38a5013cc48169cda5fdce8419b22e92cb1b5870ce49d7c783d47b2593a4f4c WatchSource:0}: Error finding container a38a5013cc48169cda5fdce8419b22e92cb1b5870ce49d7c783d47b2593a4f4c: Status 404 returned error can't find the container with id a38a5013cc48169cda5fdce8419b22e92cb1b5870ce49d7c783d47b2593a4f4c Sep 11 04:02:11.899947 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:11.899926 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-46rbd" Sep 11 04:02:11.948322 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:11.948292 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l9blm" event={"ID":"3fc8812f-fd87-492c-b430-973faacc06d8","Type":"ContainerStarted","Data":"a38a5013cc48169cda5fdce8419b22e92cb1b5870ce49d7c783d47b2593a4f4c"} Sep 11 04:02:12.954518 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:12.954485 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l9blm" event={"ID":"3fc8812f-fd87-492c-b430-973faacc06d8","Type":"ContainerStarted","Data":"8cbacb61806252760d034b1f19e5b40a1710cf19d187456c9295c60e6114c493"} Sep 11 04:02:12.954907 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:12.954582 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l9blm" Sep 11 04:02:12.990418 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:12.990350 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l9blm" podStartSLOduration=1.990329845 podStartE2EDuration="1.990329845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:02:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:02:12.987714505 +0000 UTC m=+547.741345907" watchObservedRunningTime="2026-09-11 04:02:12.990329845 +0000 UTC m=+547.743961248" Sep 11 04:02:18.506016 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:18.505981 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:02:18.606616 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:18.606579 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:02:23.960188 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:23.960153 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l9blm" Sep 11 04:02:24.012683 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:24.012651 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q8hvz"] Sep 11 04:02:24.012907 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:24.012875 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q8hvz" podUID="64a6175b-243d-40dd-87bd-b59e6aa8804e" containerName="manager" containerID="cri-o://ad7d1bb8780ccb57987f2a93abfd5dc8c9fa68972760d7954406fbdd4661b6ff" gracePeriod=10 Sep 11 04:02:24.261698 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:24.261669 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q8hvz" Sep 11 04:02:24.321963 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:24.321928 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fwzs\" (UniqueName: \"kubernetes.io/projected/64a6175b-243d-40dd-87bd-b59e6aa8804e-kube-api-access-6fwzs\") pod \"64a6175b-243d-40dd-87bd-b59e6aa8804e\" (UID: \"64a6175b-243d-40dd-87bd-b59e6aa8804e\") " Sep 11 04:02:24.322171 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:24.321992 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/64a6175b-243d-40dd-87bd-b59e6aa8804e-extensions-socket-volume\") pod \"64a6175b-243d-40dd-87bd-b59e6aa8804e\" (UID: \"64a6175b-243d-40dd-87bd-b59e6aa8804e\") " Sep 11 04:02:24.322436 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:24.322403 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64a6175b-243d-40dd-87bd-b59e6aa8804e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "64a6175b-243d-40dd-87bd-b59e6aa8804e" (UID: "64a6175b-243d-40dd-87bd-b59e6aa8804e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:02:24.324115 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:24.324094 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64a6175b-243d-40dd-87bd-b59e6aa8804e-kube-api-access-6fwzs" (OuterVolumeSpecName: "kube-api-access-6fwzs") pod "64a6175b-243d-40dd-87bd-b59e6aa8804e" (UID: "64a6175b-243d-40dd-87bd-b59e6aa8804e"). InnerVolumeSpecName "kube-api-access-6fwzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:02:24.423413 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:24.423373 2568 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/64a6175b-243d-40dd-87bd-b59e6aa8804e-extensions-socket-volume\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:02:24.423413 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:24.423406 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fwzs\" (UniqueName: \"kubernetes.io/projected/64a6175b-243d-40dd-87bd-b59e6aa8804e-kube-api-access-6fwzs\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:02:25.009125 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:25.009084 2568 generic.go:356] "Generic (PLEG): container finished" podID="64a6175b-243d-40dd-87bd-b59e6aa8804e" containerID="ad7d1bb8780ccb57987f2a93abfd5dc8c9fa68972760d7954406fbdd4661b6ff" exitCode=0 Sep 11 04:02:25.009521 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:25.009132 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q8hvz" Sep 11 04:02:25.009521 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:25.009162 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q8hvz" event={"ID":"64a6175b-243d-40dd-87bd-b59e6aa8804e","Type":"ContainerDied","Data":"ad7d1bb8780ccb57987f2a93abfd5dc8c9fa68972760d7954406fbdd4661b6ff"} Sep 11 04:02:25.009521 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:25.009215 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q8hvz" event={"ID":"64a6175b-243d-40dd-87bd-b59e6aa8804e","Type":"ContainerDied","Data":"467a4595a0da398f77e8edad4b0894af887ee22fc1dfaa93dea87e28e03ee41a"} Sep 11 04:02:25.009521 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:25.009239 2568 scope.go:117] "RemoveContainer" containerID="ad7d1bb8780ccb57987f2a93abfd5dc8c9fa68972760d7954406fbdd4661b6ff" Sep 11 04:02:25.018647 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:25.018486 2568 scope.go:117] "RemoveContainer" containerID="ad7d1bb8780ccb57987f2a93abfd5dc8c9fa68972760d7954406fbdd4661b6ff" Sep 11 04:02:25.018822 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:02:25.018799 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad7d1bb8780ccb57987f2a93abfd5dc8c9fa68972760d7954406fbdd4661b6ff\": container with ID starting with ad7d1bb8780ccb57987f2a93abfd5dc8c9fa68972760d7954406fbdd4661b6ff not found: ID does not exist" containerID="ad7d1bb8780ccb57987f2a93abfd5dc8c9fa68972760d7954406fbdd4661b6ff" Sep 11 04:02:25.018876 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:25.018831 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad7d1bb8780ccb57987f2a93abfd5dc8c9fa68972760d7954406fbdd4661b6ff"} err="failed to get container status \"ad7d1bb8780ccb57987f2a93abfd5dc8c9fa68972760d7954406fbdd4661b6ff\": rpc error: code = NotFound desc = could not find container \"ad7d1bb8780ccb57987f2a93abfd5dc8c9fa68972760d7954406fbdd4661b6ff\": container with ID starting with ad7d1bb8780ccb57987f2a93abfd5dc8c9fa68972760d7954406fbdd4661b6ff not found: ID does not exist" Sep 11 04:02:25.032585 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:25.032557 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q8hvz"] Sep 11 04:02:25.041028 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:25.040994 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-q8hvz"] Sep 11 04:02:25.768058 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:25.768016 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64a6175b-243d-40dd-87bd-b59e6aa8804e" path="/var/lib/kubelet/pods/64a6175b-243d-40dd-87bd-b59e6aa8804e/volumes" Sep 11 04:02:27.443929 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:27.443894 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:02:27.742280 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:27.742188 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:02:28.025391 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:28.025304 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:02:29.437885 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:29.437847 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:02:41.170961 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:41.170922 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:02:41.266976 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:41.266936 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:02:41.335202 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:41.335167 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:02:41.986585 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:41.986543 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:02:42.086901 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:42.086865 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:02:42.108406 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:42.108372 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-cl67h"] Sep 11 04:02:42.108706 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:42.108693 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64a6175b-243d-40dd-87bd-b59e6aa8804e" containerName="manager" Sep 11 04:02:42.108751 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:42.108711 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="64a6175b-243d-40dd-87bd-b59e6aa8804e" containerName="manager" Sep 11 04:02:42.108796 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:42.108785 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="64a6175b-243d-40dd-87bd-b59e6aa8804e" containerName="manager" Sep 11 04:02:42.111722 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:42.111705 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-cl67h" Sep 11 04:02:42.114415 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:42.114390 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rmb4f\"" Sep 11 04:02:42.116344 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:42.116319 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-cl67h"] Sep 11 04:02:42.273296 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:42.273212 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx22f\" (UniqueName: \"kubernetes.io/projected/2068ab44-7d1d-4e27-99b3-42e44573ba7a-kube-api-access-rx22f\") pod \"authorino-59c974bd48-cl67h\" (UID: \"2068ab44-7d1d-4e27-99b3-42e44573ba7a\") " pod="kuadrant-system/authorino-59c974bd48-cl67h" Sep 11 04:02:42.374308 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:42.374270 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx22f\" (UniqueName: \"kubernetes.io/projected/2068ab44-7d1d-4e27-99b3-42e44573ba7a-kube-api-access-rx22f\") pod \"authorino-59c974bd48-cl67h\" (UID: \"2068ab44-7d1d-4e27-99b3-42e44573ba7a\") " pod="kuadrant-system/authorino-59c974bd48-cl67h" Sep 11 04:02:42.382965 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:42.382943 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx22f\" (UniqueName: \"kubernetes.io/projected/2068ab44-7d1d-4e27-99b3-42e44573ba7a-kube-api-access-rx22f\") pod \"authorino-59c974bd48-cl67h\" (UID: \"2068ab44-7d1d-4e27-99b3-42e44573ba7a\") " pod="kuadrant-system/authorino-59c974bd48-cl67h" Sep 11 04:02:42.421695 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:42.421651 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-cl67h" Sep 11 04:02:42.690067 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:42.690026 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-cl67h"] Sep 11 04:02:42.692221 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:02:42.692180 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2068ab44_7d1d_4e27_99b3_42e44573ba7a.slice/crio-1d313e9dff9a6f2b22ec083b1a0cb7fa00d5c89a3225fac9ceef61aebea5c974 WatchSource:0}: Error finding container 1d313e9dff9a6f2b22ec083b1a0cb7fa00d5c89a3225fac9ceef61aebea5c974: Status 404 returned error can't find the container with id 1d313e9dff9a6f2b22ec083b1a0cb7fa00d5c89a3225fac9ceef61aebea5c974 Sep 11 04:02:43.096002 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:43.095917 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-cl67h" event={"ID":"2068ab44-7d1d-4e27-99b3-42e44573ba7a","Type":"ContainerStarted","Data":"1d313e9dff9a6f2b22ec083b1a0cb7fa00d5c89a3225fac9ceef61aebea5c974"} Sep 11 04:02:45.106487 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:45.106447 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-cl67h" event={"ID":"2068ab44-7d1d-4e27-99b3-42e44573ba7a","Type":"ContainerStarted","Data":"f9cfff9d9c2d948d8b766fcb97b8eadf5404ec21065a69f484d461d6bd078d53"} Sep 11 04:02:45.125507 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:45.125458 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-cl67h" podStartSLOduration=1.505673989 podStartE2EDuration="3.125442218s" totalImagesPullingTime="1.619768229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:02:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:02:45.123360055 +0000 UTC m=+579.876991457" watchObservedRunningTime="2026-09-11 04:02:45.125442218 +0000 UTC m=+579.879073619" Sep 11 04:02:55.813588 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:55.813513 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:02:55.908072 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:55.908002 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:02:56.341665 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:02:56.341630 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:03:05.703703 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:05.703677 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-5tpdc_9112557d-d0ef-44c7-8b2a-f6ecd0a980d6/console-operator/1.log" Sep 11 04:03:05.707296 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:05.707271 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-5tpdc_9112557d-d0ef-44c7-8b2a-f6ecd0a980d6/console-operator/1.log" Sep 11 04:03:08.822355 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:08.822318 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:03:09.102896 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:09.102798 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:03:10.971490 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:10.971443 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-cl67h"] Sep 11 04:03:10.971994 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:10.971642 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-cl67h" podUID="2068ab44-7d1d-4e27-99b3-42e44573ba7a" containerName="authorino" containerID="cri-o://f9cfff9d9c2d948d8b766fcb97b8eadf5404ec21065a69f484d461d6bd078d53" gracePeriod=30 Sep 11 04:03:11.212635 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:11.212612 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-cl67h" Sep 11 04:03:11.223177 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:11.223103 2568 generic.go:356] "Generic (PLEG): container finished" podID="2068ab44-7d1d-4e27-99b3-42e44573ba7a" containerID="f9cfff9d9c2d948d8b766fcb97b8eadf5404ec21065a69f484d461d6bd078d53" exitCode=0 Sep 11 04:03:11.223177 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:11.223136 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-cl67h" Sep 11 04:03:11.223358 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:11.223185 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-cl67h" event={"ID":"2068ab44-7d1d-4e27-99b3-42e44573ba7a","Type":"ContainerDied","Data":"f9cfff9d9c2d948d8b766fcb97b8eadf5404ec21065a69f484d461d6bd078d53"} Sep 11 04:03:11.223358 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:11.223241 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-cl67h" event={"ID":"2068ab44-7d1d-4e27-99b3-42e44573ba7a","Type":"ContainerDied","Data":"1d313e9dff9a6f2b22ec083b1a0cb7fa00d5c89a3225fac9ceef61aebea5c974"} Sep 11 04:03:11.223358 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:11.223265 2568 scope.go:117] "RemoveContainer" containerID="f9cfff9d9c2d948d8b766fcb97b8eadf5404ec21065a69f484d461d6bd078d53" Sep 11 04:03:11.231706 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:11.231684 2568 scope.go:117] "RemoveContainer" containerID="f9cfff9d9c2d948d8b766fcb97b8eadf5404ec21065a69f484d461d6bd078d53" Sep 11 04:03:11.232004 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:03:11.231985 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9cfff9d9c2d948d8b766fcb97b8eadf5404ec21065a69f484d461d6bd078d53\": container with ID starting with f9cfff9d9c2d948d8b766fcb97b8eadf5404ec21065a69f484d461d6bd078d53 not found: ID does not exist" containerID="f9cfff9d9c2d948d8b766fcb97b8eadf5404ec21065a69f484d461d6bd078d53" Sep 11 04:03:11.232114 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:11.232012 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9cfff9d9c2d948d8b766fcb97b8eadf5404ec21065a69f484d461d6bd078d53"} err="failed to get container status \"f9cfff9d9c2d948d8b766fcb97b8eadf5404ec21065a69f484d461d6bd078d53\": rpc error: code = NotFound desc = could not find container \"f9cfff9d9c2d948d8b766fcb97b8eadf5404ec21065a69f484d461d6bd078d53\": container with ID starting with f9cfff9d9c2d948d8b766fcb97b8eadf5404ec21065a69f484d461d6bd078d53 not found: ID does not exist" Sep 11 04:03:11.312364 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:11.312334 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rx22f\" (UniqueName: \"kubernetes.io/projected/2068ab44-7d1d-4e27-99b3-42e44573ba7a-kube-api-access-rx22f\") pod \"2068ab44-7d1d-4e27-99b3-42e44573ba7a\" (UID: \"2068ab44-7d1d-4e27-99b3-42e44573ba7a\") " Sep 11 04:03:11.314243 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:11.314216 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2068ab44-7d1d-4e27-99b3-42e44573ba7a-kube-api-access-rx22f" (OuterVolumeSpecName: "kube-api-access-rx22f") pod "2068ab44-7d1d-4e27-99b3-42e44573ba7a" (UID: "2068ab44-7d1d-4e27-99b3-42e44573ba7a"). InnerVolumeSpecName "kube-api-access-rx22f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:03:11.413274 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:11.413237 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx22f\" (UniqueName: \"kubernetes.io/projected/2068ab44-7d1d-4e27-99b3-42e44573ba7a-kube-api-access-rx22f\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:03:11.548332 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:11.548301 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-cl67h"] Sep 11 04:03:11.554839 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:11.554812 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-cl67h"] Sep 11 04:03:11.767847 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:11.767818 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2068ab44-7d1d-4e27-99b3-42e44573ba7a" path="/var/lib/kubelet/pods/2068ab44-7d1d-4e27-99b3-42e44573ba7a/volumes" Sep 11 04:03:16.003414 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:16.003378 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:03:16.102714 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:16.102676 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:03:16.195533 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:16.195496 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:03:16.250549 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:16.250514 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:03:16.385806 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:16.385712 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:03:19.326198 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:19.326146 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:03:19.418160 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:19.418117 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:03:43.467182 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:43.467093 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:03:50.946978 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:50.946932 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:03:51.032262 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:51.032215 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:03:52.331632 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:52.331596 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:03:52.443998 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:52.443963 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:03:52.536882 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:52.536850 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:03:52.717483 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:52.717439 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:03:53.051571 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:53.051478 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:03:53.157709 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:53.157670 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:03:53.171775 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:53.171737 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-7v7hl"] Sep 11 04:03:53.172133 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:53.172117 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2068ab44-7d1d-4e27-99b3-42e44573ba7a" containerName="authorino" Sep 11 04:03:53.172223 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:53.172135 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2068ab44-7d1d-4e27-99b3-42e44573ba7a" containerName="authorino" Sep 11 04:03:53.172276 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:53.172237 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2068ab44-7d1d-4e27-99b3-42e44573ba7a" containerName="authorino" Sep 11 04:03:53.175245 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:53.175222 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-7v7hl" Sep 11 04:03:53.177623 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:53.177604 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-cz8bc\"" Sep 11 04:03:53.177737 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:53.177651 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 11 04:03:53.178585 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:53.178565 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 11 04:03:53.178713 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:53.178609 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 11 04:03:53.184592 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:53.184572 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-7v7hl"] Sep 11 04:03:53.268679 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:53.268644 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67w4d\" (UniqueName: \"kubernetes.io/projected/879b00c8-d09a-4fc0-a80d-30261c01c32a-kube-api-access-67w4d\") pod \"payload-processing-5b54b7fbb7-7v7hl\" (UID: \"879b00c8-d09a-4fc0-a80d-30261c01c32a\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7v7hl" Sep 11 04:03:53.268679 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:53.268684 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/879b00c8-d09a-4fc0-a80d-30261c01c32a-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-7v7hl\" (UID: \"879b00c8-d09a-4fc0-a80d-30261c01c32a\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7v7hl" Sep 11 04:03:53.369447 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:53.369357 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67w4d\" (UniqueName: \"kubernetes.io/projected/879b00c8-d09a-4fc0-a80d-30261c01c32a-kube-api-access-67w4d\") pod \"payload-processing-5b54b7fbb7-7v7hl\" (UID: \"879b00c8-d09a-4fc0-a80d-30261c01c32a\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7v7hl" Sep 11 04:03:53.369447 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:53.369401 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/879b00c8-d09a-4fc0-a80d-30261c01c32a-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-7v7hl\" (UID: \"879b00c8-d09a-4fc0-a80d-30261c01c32a\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7v7hl" Sep 11 04:03:53.370256 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:53.370231 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/879b00c8-d09a-4fc0-a80d-30261c01c32a-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-7v7hl\" (UID: \"879b00c8-d09a-4fc0-a80d-30261c01c32a\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7v7hl" Sep 11 04:03:53.377901 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:53.377877 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67w4d\" (UniqueName: \"kubernetes.io/projected/879b00c8-d09a-4fc0-a80d-30261c01c32a-kube-api-access-67w4d\") pod \"payload-processing-5b54b7fbb7-7v7hl\" (UID: \"879b00c8-d09a-4fc0-a80d-30261c01c32a\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-7v7hl" Sep 11 04:03:53.488724 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:53.488680 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-7v7hl" Sep 11 04:03:53.760034 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:53.760009 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-7v7hl"] Sep 11 04:03:53.762277 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:03:53.762240 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod879b00c8_d09a_4fc0_a80d_30261c01c32a.slice/crio-b2805847f55797c8d0f3b75d5d37c04aa6a9b6f8bc5cced8949b50fc63e4e3c1 WatchSource:0}: Error finding container b2805847f55797c8d0f3b75d5d37c04aa6a9b6f8bc5cced8949b50fc63e4e3c1: Status 404 returned error can't find the container with id b2805847f55797c8d0f3b75d5d37c04aa6a9b6f8bc5cced8949b50fc63e4e3c1 Sep 11 04:03:54.414090 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:54.414032 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-7v7hl" event={"ID":"879b00c8-d09a-4fc0-a80d-30261c01c32a","Type":"ContainerStarted","Data":"b2805847f55797c8d0f3b75d5d37c04aa6a9b6f8bc5cced8949b50fc63e4e3c1"} Sep 11 04:03:56.423434 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:56.423396 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-7v7hl" event={"ID":"879b00c8-d09a-4fc0-a80d-30261c01c32a","Type":"ContainerStarted","Data":"ffdc5e3a18fc4d03c2c2fb6a857640f76a7e1eef81cc8253cfb274c4651eea1b"} Sep 11 04:03:56.423834 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:56.423457 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-7v7hl" Sep 11 04:03:56.441034 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:03:56.440983 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-7v7hl" podStartSLOduration=1.8191746119999999 podStartE2EDuration="3.440968488s" totalImagesPullingTime="1.621793876s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:03:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:03:56.439673181 +0000 UTC m=+651.193304586" watchObservedRunningTime="2026-09-11 04:03:56.440968488 +0000 UTC m=+651.194599890" Sep 11 04:04:02.787456 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:04:02.787419 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:04:02.884427 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:04:02.884395 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:04:07.429509 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:04:07.429476 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-7v7hl" Sep 11 04:04:16.794720 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:04:16.794686 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:04:32.809430 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:04:32.809384 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:04:34.771309 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:04:34.771277 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:04:35.907629 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:04:35.907594 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:04:58.630702 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:04:58.630598 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:04:59.052533 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:04:59.052495 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:05:05.925852 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:05:05.925810 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:05:17.417677 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:05:17.417638 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:05:24.233482 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:05:24.233443 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:05:38.930637 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:05:38.930591 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:05:47.147979 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:05:47.147939 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:06:59.246577 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:06:59.246540 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6b59f59dfb-ntnj2"] Sep 11 04:06:59.249690 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:06:59.249668 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b59f59dfb-ntnj2" Sep 11 04:06:59.252935 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:06:59.252915 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 11 04:06:59.253027 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:06:59.252915 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rmb4f\"" Sep 11 04:06:59.257255 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:06:59.257234 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6b59f59dfb-ntnj2"] Sep 11 04:06:59.356689 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:06:59.356651 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7bsk\" (UniqueName: \"kubernetes.io/projected/861f1c75-8df2-4d2a-97ac-ae57e5478404-kube-api-access-d7bsk\") pod \"authorino-6b59f59dfb-ntnj2\" (UID: \"861f1c75-8df2-4d2a-97ac-ae57e5478404\") " pod="kuadrant-system/authorino-6b59f59dfb-ntnj2" Sep 11 04:06:59.356689 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:06:59.356700 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/861f1c75-8df2-4d2a-97ac-ae57e5478404-tls-cert\") pod \"authorino-6b59f59dfb-ntnj2\" (UID: \"861f1c75-8df2-4d2a-97ac-ae57e5478404\") " pod="kuadrant-system/authorino-6b59f59dfb-ntnj2" Sep 11 04:06:59.457787 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:06:59.457746 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/861f1c75-8df2-4d2a-97ac-ae57e5478404-tls-cert\") pod \"authorino-6b59f59dfb-ntnj2\" (UID: \"861f1c75-8df2-4d2a-97ac-ae57e5478404\") " pod="kuadrant-system/authorino-6b59f59dfb-ntnj2" Sep 11 04:06:59.457973 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:06:59.457854 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7bsk\" (UniqueName: \"kubernetes.io/projected/861f1c75-8df2-4d2a-97ac-ae57e5478404-kube-api-access-d7bsk\") pod \"authorino-6b59f59dfb-ntnj2\" (UID: \"861f1c75-8df2-4d2a-97ac-ae57e5478404\") " pod="kuadrant-system/authorino-6b59f59dfb-ntnj2" Sep 11 04:06:59.460419 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:06:59.460393 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/861f1c75-8df2-4d2a-97ac-ae57e5478404-tls-cert\") pod \"authorino-6b59f59dfb-ntnj2\" (UID: \"861f1c75-8df2-4d2a-97ac-ae57e5478404\") " pod="kuadrant-system/authorino-6b59f59dfb-ntnj2" Sep 11 04:06:59.464980 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:06:59.464954 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7bsk\" (UniqueName: \"kubernetes.io/projected/861f1c75-8df2-4d2a-97ac-ae57e5478404-kube-api-access-d7bsk\") pod \"authorino-6b59f59dfb-ntnj2\" (UID: \"861f1c75-8df2-4d2a-97ac-ae57e5478404\") " pod="kuadrant-system/authorino-6b59f59dfb-ntnj2" Sep 11 04:06:59.559180 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:06:59.559101 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b59f59dfb-ntnj2" Sep 11 04:06:59.830735 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:06:59.830650 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6b59f59dfb-ntnj2"] Sep 11 04:06:59.835711 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:06:59.835687 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod861f1c75_8df2_4d2a_97ac_ae57e5478404.slice/crio-71f0556cffc5e06c4bb2f826a46400379cfe586804f473365602a39ea7b1b08c WatchSource:0}: Error finding container 71f0556cffc5e06c4bb2f826a46400379cfe586804f473365602a39ea7b1b08c: Status 404 returned error can't find the container with id 71f0556cffc5e06c4bb2f826a46400379cfe586804f473365602a39ea7b1b08c Sep 11 04:06:59.837088 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:06:59.837039 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 04:07:00.224929 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:00.224892 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b59f59dfb-ntnj2" event={"ID":"861f1c75-8df2-4d2a-97ac-ae57e5478404","Type":"ContainerStarted","Data":"71f0556cffc5e06c4bb2f826a46400379cfe586804f473365602a39ea7b1b08c"} Sep 11 04:07:01.231508 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:01.231467 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b59f59dfb-ntnj2" event={"ID":"861f1c75-8df2-4d2a-97ac-ae57e5478404","Type":"ContainerStarted","Data":"917b596bfae0ecc9cb635d6d9e2f36a390a8b80f67e7b2368c12650bf148f215"} Sep 11 04:07:01.249227 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:01.249163 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6b59f59dfb-ntnj2" podStartSLOduration=1.5090928460000002 podStartE2EDuration="2.249141084s" totalImagesPullingTime="740.048238ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:06:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:07:01.248492284 +0000 UTC m=+836.002123685" watchObservedRunningTime="2026-09-11 04:07:01.249141084 +0000 UTC m=+836.002772486" Sep 11 04:07:01.857861 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:01.857795 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:01.962523 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:01.962464 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:03.109037 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:03.108961 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:03.201640 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:03.201603 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:19.740944 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:19.740901 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:19.840254 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:19.840191 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:20.043603 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:20.043528 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s"] Sep 11 04:07:20.047005 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:20.046970 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" Sep 11 04:07:20.050166 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:20.050143 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 11 04:07:20.050334 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:20.050223 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a28c9f-openshift-default-dockercfg-tk64p\"" Sep 11 04:07:20.061243 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:20.061218 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s"] Sep 11 04:07:20.074316 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:20.074280 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:20.127529 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:20.127491 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/136cff62-ad79-4976-81dd-ac786e4d4cec-istio-podinfo\") pod \"e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s\" (UID: \"136cff62-ad79-4976-81dd-ac786e4d4cec\") " pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" Sep 11 04:07:20.127529 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:20.127532 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/136cff62-ad79-4976-81dd-ac786e4d4cec-workload-socket\") pod \"e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s\" (UID: \"136cff62-ad79-4976-81dd-ac786e4d4cec\") " pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" Sep 11 04:07:20.127748 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:20.127555 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/136cff62-ad79-4976-81dd-ac786e4d4cec-istiod-ca-cert\") pod \"e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s\" (UID: \"136cff62-ad79-4976-81dd-ac786e4d4cec\") " pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" Sep 11 04:07:20.127748 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:20.127578 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/136cff62-ad79-4976-81dd-ac786e4d4cec-workload-certs\") pod \"e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s\" (UID: \"136cff62-ad79-4976-81dd-ac786e4d4cec\") " pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" Sep 11 04:07:20.127748 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:20.127663 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/136cff62-ad79-4976-81dd-ac786e4d4cec-istio-envoy\") pod \"e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s\" (UID: \"136cff62-ad79-4976-81dd-ac786e4d4cec\") " pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" Sep 11 04:07:20.127748 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:20.127712 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdnpl\" (UniqueName: \"kubernetes.io/projected/136cff62-ad79-4976-81dd-ac786e4d4cec-kube-api-access-gdnpl\") pod \"e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s\" (UID: \"136cff62-ad79-4976-81dd-ac786e4d4cec\") " pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" Sep 11 04:07:20.127877 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:20.127758 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/136cff62-ad79-4976-81dd-ac786e4d4cec-istio-data\") pod \"e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s\" (UID: \"136cff62-ad79-4976-81dd-ac786e4d4cec\") " pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" Sep 11 04:07:20.127877 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:20.127833 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/136cff62-ad79-4976-81dd-ac786e4d4cec-credential-socket\") pod \"e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s\" (UID: \"136cff62-ad79-4976-81dd-ac786e4d4cec\") " pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" Sep 11 04:07:20.127942 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:20.127876 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/136cff62-ad79-4976-81dd-ac786e4d4cec-istio-token\") pod \"e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s\" (UID: \"136cff62-ad79-4976-81dd-ac786e4d4cec\") " pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" Sep 11 04:07:20.228583 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:20.228544 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/136cff62-ad79-4976-81dd-ac786e4d4cec-istio-data\") pod \"e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s\" (UID: \"136cff62-ad79-4976-81dd-ac786e4d4cec\") " pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" Sep 11 04:07:20.228771 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:20.228602 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/136cff62-ad79-4976-81dd-ac786e4d4cec-credential-socket\") pod \"e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s\" (UID: \"136cff62-ad79-4976-81dd-ac786e4d4cec\") " pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" Sep 11 04:07:20.228771 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:20.228630 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/136cff62-ad79-4976-81dd-ac786e4d4cec-istio-token\") pod \"e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s\" (UID: \"136cff62-ad79-4976-81dd-ac786e4d4cec\") " pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" Sep 11 04:07:20.228771 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:20.228671 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/136cff62-ad79-4976-81dd-ac786e4d4cec-istio-podinfo\") pod \"e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s\" (UID: \"136cff62-ad79-4976-81dd-ac786e4d4cec\") " pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" Sep 11 04:07:20.228771 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:20.228700 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/136cff62-ad79-4976-81dd-ac786e4d4cec-workload-socket\") pod \"e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s\" (UID: \"136cff62-ad79-4976-81dd-ac786e4d4cec\") " pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" Sep 11 04:07:20.228771 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:20.228731 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/136cff62-ad79-4976-81dd-ac786e4d4cec-istiod-ca-cert\") pod \"e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s\" (UID: \"136cff62-ad79-4976-81dd-ac786e4d4cec\") " pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" Sep 11 04:07:20.228771 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:20.228766 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/136cff62-ad79-4976-81dd-ac786e4d4cec-workload-certs\") pod \"e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s\" (UID: \"136cff62-ad79-4976-81dd-ac786e4d4cec\") " pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" Sep 11 04:07:20.229117 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:20.228802 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/136cff62-ad79-4976-81dd-ac786e4d4cec-istio-envoy\") pod \"e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s\" (UID: \"136cff62-ad79-4976-81dd-ac786e4d4cec\") " pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" Sep 11 04:07:20.229117 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:20.228838 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdnpl\" (UniqueName: \"kubernetes.io/projected/136cff62-ad79-4976-81dd-ac786e4d4cec-kube-api-access-gdnpl\") pod \"e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s\" (UID: \"136cff62-ad79-4976-81dd-ac786e4d4cec\") " pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" Sep 11 04:07:20.229117 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:20.229015 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/136cff62-ad79-4976-81dd-ac786e4d4cec-istio-data\") pod \"e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s\" (UID: \"136cff62-ad79-4976-81dd-ac786e4d4cec\") " pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" Sep 11 04:07:20.229117 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:20.229073 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/136cff62-ad79-4976-81dd-ac786e4d4cec-credential-socket\") pod \"e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s\" (UID: \"136cff62-ad79-4976-81dd-ac786e4d4cec\") " pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" Sep 11 04:07:20.229276 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:20.229137 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/136cff62-ad79-4976-81dd-ac786e4d4cec-workload-certs\") pod \"e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s\" (UID: \"136cff62-ad79-4976-81dd-ac786e4d4cec\") " pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" Sep 11 04:07:20.229492 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:20.229474 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/136cff62-ad79-4976-81dd-ac786e4d4cec-istiod-ca-cert\") pod \"e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s\" (UID: \"136cff62-ad79-4976-81dd-ac786e4d4cec\") " pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" Sep 11 04:07:20.229675 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:20.229659 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/136cff62-ad79-4976-81dd-ac786e4d4cec-workload-socket\") pod \"e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s\" (UID: \"136cff62-ad79-4976-81dd-ac786e4d4cec\") " pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" Sep 11 04:07:20.231183 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:20.231164 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/136cff62-ad79-4976-81dd-ac786e4d4cec-istio-envoy\") pod \"e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s\" (UID: \"136cff62-ad79-4976-81dd-ac786e4d4cec\") " pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" Sep 11 04:07:20.231527 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:20.231511 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/136cff62-ad79-4976-81dd-ac786e4d4cec-istio-podinfo\") pod \"e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s\" (UID: \"136cff62-ad79-4976-81dd-ac786e4d4cec\") " pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" Sep 11 04:07:20.236719 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:20.236694 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdnpl\" (UniqueName: \"kubernetes.io/projected/136cff62-ad79-4976-81dd-ac786e4d4cec-kube-api-access-gdnpl\") pod \"e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s\" (UID: \"136cff62-ad79-4976-81dd-ac786e4d4cec\") " pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" Sep 11 04:07:20.237136 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:20.237113 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/136cff62-ad79-4976-81dd-ac786e4d4cec-istio-token\") pod \"e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s\" (UID: \"136cff62-ad79-4976-81dd-ac786e4d4cec\") " pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" Sep 11 04:07:20.291824 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:20.291784 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:20.358892 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:20.358822 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" Sep 11 04:07:20.635765 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:20.635737 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s"] Sep 11 04:07:20.638879 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:07:20.638847 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod136cff62_ad79_4976_81dd_ac786e4d4cec.slice/crio-78b2467b2a72285920c7f8234c12f0c225e650cb00adaf87c5cffcd215d1864b WatchSource:0}: Error finding container 78b2467b2a72285920c7f8234c12f0c225e650cb00adaf87c5cffcd215d1864b: Status 404 returned error can't find the container with id 78b2467b2a72285920c7f8234c12f0c225e650cb00adaf87c5cffcd215d1864b Sep 11 04:07:20.641801 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:20.641758 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 04:07:20.641900 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:20.641854 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 04:07:20.641963 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:20.641899 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 04:07:21.271475 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:21.271439 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:21.329907 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:21.329876 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" event={"ID":"136cff62-ad79-4976-81dd-ac786e4d4cec","Type":"ContainerStarted","Data":"2d01931adcd8522c7f52f6336c29b07b3a6ca37a8e83d26653caf9834e041989"} Sep 11 04:07:21.330096 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:21.329928 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" event={"ID":"136cff62-ad79-4976-81dd-ac786e4d4cec","Type":"ContainerStarted","Data":"78b2467b2a72285920c7f8234c12f0c225e650cb00adaf87c5cffcd215d1864b"} Sep 11 04:07:21.353306 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:21.353255 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" podStartSLOduration=1.353237079 podStartE2EDuration="1.353237079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:07:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:07:21.352609942 +0000 UTC m=+856.106241351" watchObservedRunningTime="2026-09-11 04:07:21.353237079 +0000 UTC m=+856.106868486" Sep 11 04:07:21.357175 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:21.357146 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:21.360004 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:21.359981 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" Sep 11 04:07:21.364753 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:21.364725 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" Sep 11 04:07:21.765648 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:21.765614 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:21.881453 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:21.881422 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8089-96970e"] Sep 11 04:07:21.884797 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:21.884779 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8089-96970e" Sep 11 04:07:21.887079 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:21.887060 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zqmz7\"" Sep 11 04:07:21.891368 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:21.891306 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8089-96970e"] Sep 11 04:07:21.944973 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:21.944943 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glw6f\" (UniqueName: \"kubernetes.io/projected/9aba86d6-56f9-44e6-8f79-7f48f9ad55a3-kube-api-access-glw6f\") pod \"test-curl-8089-96970e\" (UID: \"9aba86d6-56f9-44e6-8f79-7f48f9ad55a3\") " pod="opendatahub/test-curl-8089-96970e" Sep 11 04:07:22.046309 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:22.046232 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glw6f\" (UniqueName: \"kubernetes.io/projected/9aba86d6-56f9-44e6-8f79-7f48f9ad55a3-kube-api-access-glw6f\") pod \"test-curl-8089-96970e\" (UID: \"9aba86d6-56f9-44e6-8f79-7f48f9ad55a3\") " pod="opendatahub/test-curl-8089-96970e" Sep 11 04:07:22.054235 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:22.054205 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glw6f\" (UniqueName: \"kubernetes.io/projected/9aba86d6-56f9-44e6-8f79-7f48f9ad55a3-kube-api-access-glw6f\") pod \"test-curl-8089-96970e\" (UID: \"9aba86d6-56f9-44e6-8f79-7f48f9ad55a3\") " pod="opendatahub/test-curl-8089-96970e" Sep 11 04:07:22.195861 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:22.195823 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8089-96970e" Sep 11 04:07:22.335159 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:22.335087 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" Sep 11 04:07:22.336039 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:22.336021 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" Sep 11 04:07:22.470206 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:22.470178 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8089-96970e"] Sep 11 04:07:22.472482 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:07:22.472450 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9aba86d6_56f9_44e6_8f79_7f48f9ad55a3.slice/crio-eaf0cc9530feb9e5b0dd816600f3fe2a978eed2a05d25175c1a15ad66a5617bc WatchSource:0}: Error finding container eaf0cc9530feb9e5b0dd816600f3fe2a978eed2a05d25175c1a15ad66a5617bc: Status 404 returned error can't find the container with id eaf0cc9530feb9e5b0dd816600f3fe2a978eed2a05d25175c1a15ad66a5617bc Sep 11 04:07:23.277582 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:23.277547 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:23.341820 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:23.341724 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8089-96970e" event={"ID":"9aba86d6-56f9-44e6-8f79-7f48f9ad55a3","Type":"ContainerStarted","Data":"d9697e35029b601c838ebc8c3854c8e2500d4c1e772ef0b08b2437aee13c45b7"} Sep 11 04:07:23.341820 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:23.341779 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8089-96970e" event={"ID":"9aba86d6-56f9-44e6-8f79-7f48f9ad55a3","Type":"ContainerStarted","Data":"eaf0cc9530feb9e5b0dd816600f3fe2a978eed2a05d25175c1a15ad66a5617bc"} Sep 11 04:07:23.358249 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:23.358176 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8089-96970e" podStartSLOduration=1.6661449240000001 podStartE2EDuration="2.35815439s" totalImagesPullingTime="692.009466ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:07:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:07:23.357777504 +0000 UTC m=+858.111408907" watchObservedRunningTime="2026-09-11 04:07:23.35815439 +0000 UTC m=+858.111785796" Sep 11 04:07:23.363168 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:23.363143 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:23.981818 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:23.981777 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8089-96970e"] Sep 11 04:07:23.986132 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:23.986103 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8089-96970e"] Sep 11 04:07:24.111093 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:24.111033 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8089-2513c6"] Sep 11 04:07:24.111430 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:24.111415 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aba86d6-56f9-44e6-8f79-7f48f9ad55a3" containerName="test-curl-8089-96970e" Sep 11 04:07:24.111479 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:24.111433 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aba86d6-56f9-44e6-8f79-7f48f9ad55a3" containerName="test-curl-8089-96970e" Sep 11 04:07:24.111526 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:24.111513 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="9aba86d6-56f9-44e6-8f79-7f48f9ad55a3" containerName="test-curl-8089-96970e" Sep 11 04:07:24.114546 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:24.114529 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8089-2513c6" Sep 11 04:07:24.121628 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:24.121285 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8089-2513c6"] Sep 11 04:07:24.163856 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:24.163816 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69p8w\" (UniqueName: \"kubernetes.io/projected/a23b7c29-0454-484a-ba19-3f387d8b1bba-kube-api-access-69p8w\") pod \"test-curl-8089-2513c6\" (UID: \"a23b7c29-0454-484a-ba19-3f387d8b1bba\") " pod="opendatahub/test-curl-8089-2513c6" Sep 11 04:07:24.264401 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:24.264312 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69p8w\" (UniqueName: \"kubernetes.io/projected/a23b7c29-0454-484a-ba19-3f387d8b1bba-kube-api-access-69p8w\") pod \"test-curl-8089-2513c6\" (UID: \"a23b7c29-0454-484a-ba19-3f387d8b1bba\") " pod="opendatahub/test-curl-8089-2513c6" Sep 11 04:07:24.277467 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:24.277438 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69p8w\" (UniqueName: \"kubernetes.io/projected/a23b7c29-0454-484a-ba19-3f387d8b1bba-kube-api-access-69p8w\") pod \"test-curl-8089-2513c6\" (UID: \"a23b7c29-0454-484a-ba19-3f387d8b1bba\") " pod="opendatahub/test-curl-8089-2513c6" Sep 11 04:07:24.346910 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:07:24.346877 2568 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8089-96970e" container="test-curl-8089-96970e" Sep 11 04:07:24.424886 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:24.424853 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8089-2513c6" Sep 11 04:07:24.624239 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:24.624159 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:24.715931 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:24.715893 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8089-2513c6"] Sep 11 04:07:24.723868 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:07:24.723835 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda23b7c29_0454_484a_ba19_3f387d8b1bba.slice/crio-bf544ebbf38e7e07b2964b49c1e1c74bba4c419d366cd2f2ff0a8a7edd014849 WatchSource:0}: Error finding container bf544ebbf38e7e07b2964b49c1e1c74bba4c419d366cd2f2ff0a8a7edd014849: Status 404 returned error can't find the container with id bf544ebbf38e7e07b2964b49c1e1c74bba4c419d366cd2f2ff0a8a7edd014849 Sep 11 04:07:24.742207 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:24.742177 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:24.906679 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:24.906643 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:25.256450 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:25.256336 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:25.353501 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:25.353464 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8089-2513c6" event={"ID":"a23b7c29-0454-484a-ba19-3f387d8b1bba","Type":"ContainerStarted","Data":"11cb3f6bc3d295ebbe1c83cb66f1a9018e81760567910148bb9b1bcd18f4c2bc"} Sep 11 04:07:25.353938 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:25.353507 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8089-2513c6" event={"ID":"a23b7c29-0454-484a-ba19-3f387d8b1bba","Type":"ContainerStarted","Data":"bf544ebbf38e7e07b2964b49c1e1c74bba4c419d366cd2f2ff0a8a7edd014849"} Sep 11 04:07:25.353938 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:25.353571 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8089-96970e" podUID="9aba86d6-56f9-44e6-8f79-7f48f9ad55a3" containerName="test-curl-8089-96970e" containerID="cri-o://d9697e35029b601c838ebc8c3854c8e2500d4c1e772ef0b08b2437aee13c45b7" gracePeriod=2 Sep 11 04:07:25.356093 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:25.356027 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:25.374857 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:25.374822 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a28c9f-5b6c74bcbc-j7hwv"] Sep 11 04:07:25.378066 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:25.378024 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a28c9f-5b6c74bcbc-j7hwv" Sep 11 04:07:25.378718 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:25.378675 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8089-2513c6" podStartSLOduration=1.3786620219999999 podStartE2EDuration="1.378662022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:07:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:07:25.377320538 +0000 UTC m=+860.130951941" watchObservedRunningTime="2026-09-11 04:07:25.378662022 +0000 UTC m=+860.132293423" Sep 11 04:07:25.379075 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:07:25.379029 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8089-96970e\" is forbidden: User \"system:node:ip-10-0-140-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-30.ec2.internal' and this object" podUID="9aba86d6-56f9-44e6-8f79-7f48f9ad55a3" pod="opendatahub/test-curl-8089-96970e" Sep 11 04:07:25.380366 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:25.380348 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a28c9f-dockercfg-gjv57\"" Sep 11 04:07:25.380423 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:25.380349 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a28c9f\"" Sep 11 04:07:25.393544 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:25.393515 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a28c9f-5b6c74bcbc-j7hwv"] Sep 11 04:07:25.407205 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:07:25.407181 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8089-96970e\" is forbidden: User \"system:node:ip-10-0-140-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-30.ec2.internal' and this object" podUID="9aba86d6-56f9-44e6-8f79-7f48f9ad55a3" pod="opendatahub/test-curl-8089-96970e" Sep 11 04:07:25.473684 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:25.473652 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8xgb\" (UniqueName: \"kubernetes.io/projected/3818a5e1-5ed9-4a12-8f89-7c7b58ccb0ec-kube-api-access-k8xgb\") pod \"payload-processing-e2e-inf-a-a28c9f-5b6c74bcbc-j7hwv\" (UID: \"3818a5e1-5ed9-4a12-8f89-7c7b58ccb0ec\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a28c9f-5b6c74bcbc-j7hwv" Sep 11 04:07:25.473844 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:25.473704 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3818a5e1-5ed9-4a12-8f89-7c7b58ccb0ec-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a28c9f-5b6c74bcbc-j7hwv\" (UID: \"3818a5e1-5ed9-4a12-8f89-7c7b58ccb0ec\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a28c9f-5b6c74bcbc-j7hwv" Sep 11 04:07:25.575154 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:25.575119 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8xgb\" (UniqueName: \"kubernetes.io/projected/3818a5e1-5ed9-4a12-8f89-7c7b58ccb0ec-kube-api-access-k8xgb\") pod \"payload-processing-e2e-inf-a-a28c9f-5b6c74bcbc-j7hwv\" (UID: \"3818a5e1-5ed9-4a12-8f89-7c7b58ccb0ec\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a28c9f-5b6c74bcbc-j7hwv" Sep 11 04:07:25.575313 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:25.575175 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3818a5e1-5ed9-4a12-8f89-7c7b58ccb0ec-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a28c9f-5b6c74bcbc-j7hwv\" (UID: \"3818a5e1-5ed9-4a12-8f89-7c7b58ccb0ec\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a28c9f-5b6c74bcbc-j7hwv" Sep 11 04:07:25.575959 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:25.575934 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3818a5e1-5ed9-4a12-8f89-7c7b58ccb0ec-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a28c9f-5b6c74bcbc-j7hwv\" (UID: \"3818a5e1-5ed9-4a12-8f89-7c7b58ccb0ec\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a28c9f-5b6c74bcbc-j7hwv" Sep 11 04:07:25.583903 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:25.583879 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8xgb\" (UniqueName: \"kubernetes.io/projected/3818a5e1-5ed9-4a12-8f89-7c7b58ccb0ec-kube-api-access-k8xgb\") pod \"payload-processing-e2e-inf-a-a28c9f-5b6c74bcbc-j7hwv\" (UID: \"3818a5e1-5ed9-4a12-8f89-7c7b58ccb0ec\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a28c9f-5b6c74bcbc-j7hwv" Sep 11 04:07:25.689872 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:25.689830 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a28c9f-5b6c74bcbc-j7hwv" Sep 11 04:07:25.766989 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:07:25.766948 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8089-96970e\" is forbidden: User \"system:node:ip-10-0-140-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-30.ec2.internal' and this object" podUID="9aba86d6-56f9-44e6-8f79-7f48f9ad55a3" pod="opendatahub/test-curl-8089-96970e" Sep 11 04:07:25.967221 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:25.967195 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a28c9f-5b6c74bcbc-j7hwv"] Sep 11 04:07:25.969458 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:07:25.969418 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3818a5e1_5ed9_4a12_8f89_7c7b58ccb0ec.slice/crio-455a114ae498fd5023367e51d3aefa290370915e8149e44dd462fda00c1e94f2 WatchSource:0}: Error finding container 455a114ae498fd5023367e51d3aefa290370915e8149e44dd462fda00c1e94f2: Status 404 returned error can't find the container with id 455a114ae498fd5023367e51d3aefa290370915e8149e44dd462fda00c1e94f2 Sep 11 04:07:25.985514 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:25.985494 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8089-2513c6"] Sep 11 04:07:25.992709 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:25.992687 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8089-2513c6"] Sep 11 04:07:26.360414 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:26.360314 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a28c9f-5b6c74bcbc-j7hwv" event={"ID":"3818a5e1-5ed9-4a12-8f89-7c7b58ccb0ec","Type":"ContainerStarted","Data":"4c09006dfc68014aa0b63f96f4e3d3c0c83c82fd1bd233e160668f760a8d7690"} Sep 11 04:07:26.360414 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:26.360380 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a28c9f-5b6c74bcbc-j7hwv" event={"ID":"3818a5e1-5ed9-4a12-8f89-7c7b58ccb0ec","Type":"ContainerStarted","Data":"455a114ae498fd5023367e51d3aefa290370915e8149e44dd462fda00c1e94f2"} Sep 11 04:07:26.360797 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:26.360568 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a28c9f-5b6c74bcbc-j7hwv" Sep 11 04:07:26.360797 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:07:26.360569 2568 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8089-2513c6" container="test-curl-8089-2513c6" Sep 11 04:07:26.379137 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:26.379095 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-a28c9f-5b6c74bcbc-j7hwv" podStartSLOduration=1.379081671 podStartE2EDuration="1.379081671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:07:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:07:26.377391592 +0000 UTC m=+861.131022995" watchObservedRunningTime="2026-09-11 04:07:26.379081671 +0000 UTC m=+861.132713070" Sep 11 04:07:26.565700 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:26.565660 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:26.833399 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:26.833367 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5"] Sep 11 04:07:26.833853 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:26.833822 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a23b7c29-0454-484a-ba19-3f387d8b1bba" containerName="test-curl-8089-2513c6" Sep 11 04:07:26.833853 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:26.833846 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a23b7c29-0454-484a-ba19-3f387d8b1bba" containerName="test-curl-8089-2513c6" Sep 11 04:07:26.834074 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:26.833950 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="a23b7c29-0454-484a-ba19-3f387d8b1bba" containerName="test-curl-8089-2513c6" Sep 11 04:07:26.837767 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:26.837745 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" Sep 11 04:07:26.840928 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:26.840904 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-eda9e4-openshift-default-dockercfg-c6tcp\"" Sep 11 04:07:26.856945 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:26.856913 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5"] Sep 11 04:07:26.869385 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:26.869357 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:26.885618 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:26.885588 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-istio-envoy\") pod \"e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5\" (UID: \"82fa5c18-c332-45ab-8bea-ac64f4b2caf3\") " pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" Sep 11 04:07:26.885729 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:26.885655 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-istiod-ca-cert\") pod \"e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5\" (UID: \"82fa5c18-c332-45ab-8bea-ac64f4b2caf3\") " pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" Sep 11 04:07:26.885729 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:26.885688 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-istio-token\") pod \"e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5\" (UID: \"82fa5c18-c332-45ab-8bea-ac64f4b2caf3\") " pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" Sep 11 04:07:26.885729 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:26.885723 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-istio-podinfo\") pod \"e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5\" (UID: \"82fa5c18-c332-45ab-8bea-ac64f4b2caf3\") " pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" Sep 11 04:07:26.885886 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:26.885754 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-workload-socket\") pod \"e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5\" (UID: \"82fa5c18-c332-45ab-8bea-ac64f4b2caf3\") " pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" Sep 11 04:07:26.885886 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:26.885826 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-istio-data\") pod \"e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5\" (UID: \"82fa5c18-c332-45ab-8bea-ac64f4b2caf3\") " pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" Sep 11 04:07:26.885886 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:26.885878 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jb5p\" (UniqueName: \"kubernetes.io/projected/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-kube-api-access-7jb5p\") pod \"e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5\" (UID: \"82fa5c18-c332-45ab-8bea-ac64f4b2caf3\") " pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" Sep 11 04:07:26.886066 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:26.885911 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-workload-certs\") pod \"e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5\" (UID: \"82fa5c18-c332-45ab-8bea-ac64f4b2caf3\") " pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" Sep 11 04:07:26.886066 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:26.885976 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-credential-socket\") pod \"e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5\" (UID: \"82fa5c18-c332-45ab-8bea-ac64f4b2caf3\") " pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" Sep 11 04:07:26.986688 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:26.986656 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-istio-envoy\") pod \"e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5\" (UID: \"82fa5c18-c332-45ab-8bea-ac64f4b2caf3\") " pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" Sep 11 04:07:26.986839 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:26.986706 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-istiod-ca-cert\") pod \"e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5\" (UID: \"82fa5c18-c332-45ab-8bea-ac64f4b2caf3\") " pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" Sep 11 04:07:26.986839 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:26.986726 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-istio-token\") pod \"e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5\" (UID: \"82fa5c18-c332-45ab-8bea-ac64f4b2caf3\") " pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" Sep 11 04:07:26.986839 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:26.986756 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-istio-podinfo\") pod \"e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5\" (UID: \"82fa5c18-c332-45ab-8bea-ac64f4b2caf3\") " pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" Sep 11 04:07:26.986839 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:26.986777 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-workload-socket\") pod \"e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5\" (UID: \"82fa5c18-c332-45ab-8bea-ac64f4b2caf3\") " pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" Sep 11 04:07:26.986839 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:26.986796 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-istio-data\") pod \"e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5\" (UID: \"82fa5c18-c332-45ab-8bea-ac64f4b2caf3\") " pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" Sep 11 04:07:26.986839 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:26.986828 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jb5p\" (UniqueName: \"kubernetes.io/projected/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-kube-api-access-7jb5p\") pod \"e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5\" (UID: \"82fa5c18-c332-45ab-8bea-ac64f4b2caf3\") " pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" Sep 11 04:07:26.987122 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:26.986877 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-workload-certs\") pod \"e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5\" (UID: \"82fa5c18-c332-45ab-8bea-ac64f4b2caf3\") " pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" Sep 11 04:07:26.987122 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:26.986924 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-credential-socket\") pod \"e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5\" (UID: \"82fa5c18-c332-45ab-8bea-ac64f4b2caf3\") " pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" Sep 11 04:07:26.987465 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:26.987348 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-istio-data\") pod \"e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5\" (UID: \"82fa5c18-c332-45ab-8bea-ac64f4b2caf3\") " pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" Sep 11 04:07:26.987465 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:26.987369 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-workload-socket\") pod \"e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5\" (UID: \"82fa5c18-c332-45ab-8bea-ac64f4b2caf3\") " pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" Sep 11 04:07:26.987641 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:26.987493 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-istiod-ca-cert\") pod \"e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5\" (UID: \"82fa5c18-c332-45ab-8bea-ac64f4b2caf3\") " pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" Sep 11 04:07:26.987641 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:26.987556 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-workload-certs\") pod \"e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5\" (UID: \"82fa5c18-c332-45ab-8bea-ac64f4b2caf3\") " pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" Sep 11 04:07:26.987810 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:26.987788 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-credential-socket\") pod \"e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5\" (UID: \"82fa5c18-c332-45ab-8bea-ac64f4b2caf3\") " pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" Sep 11 04:07:26.989182 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:26.989160 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-istio-envoy\") pod \"e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5\" (UID: \"82fa5c18-c332-45ab-8bea-ac64f4b2caf3\") " pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" Sep 11 04:07:26.989537 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:26.989518 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-istio-podinfo\") pod \"e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5\" (UID: \"82fa5c18-c332-45ab-8bea-ac64f4b2caf3\") " pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" Sep 11 04:07:26.995357 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:26.995338 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jb5p\" (UniqueName: \"kubernetes.io/projected/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-kube-api-access-7jb5p\") pod \"e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5\" (UID: \"82fa5c18-c332-45ab-8bea-ac64f4b2caf3\") " pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" Sep 11 04:07:26.995490 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:26.995425 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-istio-token\") pod \"e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5\" (UID: \"82fa5c18-c332-45ab-8bea-ac64f4b2caf3\") " pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" Sep 11 04:07:27.022410 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:27.022371 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:27.150906 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:27.150809 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" Sep 11 04:07:27.365759 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:27.365690 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8089-2513c6" podUID="a23b7c29-0454-484a-ba19-3f387d8b1bba" containerName="test-curl-8089-2513c6" containerID="cri-o://11cb3f6bc3d295ebbe1c83cb66f1a9018e81760567910148bb9b1bcd18f4c2bc" gracePeriod=2 Sep 11 04:07:27.367634 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:07:27.367598 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8089-2513c6\" is forbidden: User \"system:node:ip-10-0-140-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-30.ec2.internal' and this object" podUID="a23b7c29-0454-484a-ba19-3f387d8b1bba" pod="opendatahub/test-curl-8089-2513c6" Sep 11 04:07:27.433168 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:27.433132 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5"] Sep 11 04:07:27.437507 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:07:27.437475 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82fa5c18_c332_45ab_8bea_ac64f4b2caf3.slice/crio-3f1cd958a54b0dbaa5b8434a1e3d1d33178579bdf8680cda19e81133a77f401c WatchSource:0}: Error finding container 3f1cd958a54b0dbaa5b8434a1e3d1d33178579bdf8680cda19e81133a77f401c: Status 404 returned error can't find the container with id 3f1cd958a54b0dbaa5b8434a1e3d1d33178579bdf8680cda19e81133a77f401c Sep 11 04:07:27.440387 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:27.440345 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 04:07:27.440458 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:27.440434 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 04:07:27.440491 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:27.440478 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 04:07:27.528704 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:27.528681 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8089-96970e" Sep 11 04:07:27.530726 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:07:27.530701 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8089-2513c6\" is forbidden: User \"system:node:ip-10-0-140-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-30.ec2.internal' and this object" podUID="a23b7c29-0454-484a-ba19-3f387d8b1bba" pod="opendatahub/test-curl-8089-2513c6" Sep 11 04:07:27.592701 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:27.592665 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glw6f\" (UniqueName: \"kubernetes.io/projected/9aba86d6-56f9-44e6-8f79-7f48f9ad55a3-kube-api-access-glw6f\") pod \"9aba86d6-56f9-44e6-8f79-7f48f9ad55a3\" (UID: \"9aba86d6-56f9-44e6-8f79-7f48f9ad55a3\") " Sep 11 04:07:27.595009 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:27.594974 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aba86d6-56f9-44e6-8f79-7f48f9ad55a3-kube-api-access-glw6f" (OuterVolumeSpecName: "kube-api-access-glw6f") pod "9aba86d6-56f9-44e6-8f79-7f48f9ad55a3" (UID: "9aba86d6-56f9-44e6-8f79-7f48f9ad55a3"). InnerVolumeSpecName "kube-api-access-glw6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:07:27.693861 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:27.693771 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glw6f\" (UniqueName: \"kubernetes.io/projected/9aba86d6-56f9-44e6-8f79-7f48f9ad55a3-kube-api-access-glw6f\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:07:27.770164 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:27.770131 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9aba86d6-56f9-44e6-8f79-7f48f9ad55a3" path="/var/lib/kubelet/pods/9aba86d6-56f9-44e6-8f79-7f48f9ad55a3/volumes" Sep 11 04:07:28.031097 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:28.031065 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:28.101313 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:28.101272 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:28.260757 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:28.260668 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:28.371727 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:28.371692 2568 generic.go:356] "Generic (PLEG): container finished" podID="9aba86d6-56f9-44e6-8f79-7f48f9ad55a3" containerID="d9697e35029b601c838ebc8c3854c8e2500d4c1e772ef0b08b2437aee13c45b7" exitCode=137 Sep 11 04:07:28.371727 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:28.371729 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8089-96970e" Sep 11 04:07:28.372269 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:28.371793 2568 scope.go:117] "RemoveContainer" containerID="d9697e35029b601c838ebc8c3854c8e2500d4c1e772ef0b08b2437aee13c45b7" Sep 11 04:07:28.373871 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:28.373828 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" event={"ID":"82fa5c18-c332-45ab-8bea-ac64f4b2caf3","Type":"ContainerStarted","Data":"2b901d7d94ac2c752e7edec8edd0e1290da11947f049812a3c5179ad1233910c"} Sep 11 04:07:28.374022 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:28.373887 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" event={"ID":"82fa5c18-c332-45ab-8bea-ac64f4b2caf3","Type":"ContainerStarted","Data":"3f1cd958a54b0dbaa5b8434a1e3d1d33178579bdf8680cda19e81133a77f401c"} Sep 11 04:07:28.381944 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:28.381924 2568 scope.go:117] "RemoveContainer" containerID="d9697e35029b601c838ebc8c3854c8e2500d4c1e772ef0b08b2437aee13c45b7" Sep 11 04:07:28.382303 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:07:28.382285 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9697e35029b601c838ebc8c3854c8e2500d4c1e772ef0b08b2437aee13c45b7\": container with ID starting with d9697e35029b601c838ebc8c3854c8e2500d4c1e772ef0b08b2437aee13c45b7 not found: ID does not exist" containerID="d9697e35029b601c838ebc8c3854c8e2500d4c1e772ef0b08b2437aee13c45b7" Sep 11 04:07:28.382364 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:28.382311 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9697e35029b601c838ebc8c3854c8e2500d4c1e772ef0b08b2437aee13c45b7"} err="failed to get container status \"d9697e35029b601c838ebc8c3854c8e2500d4c1e772ef0b08b2437aee13c45b7\": rpc error: code = NotFound desc = could not find container \"d9697e35029b601c838ebc8c3854c8e2500d4c1e772ef0b08b2437aee13c45b7\": container with ID starting with d9697e35029b601c838ebc8c3854c8e2500d4c1e772ef0b08b2437aee13c45b7 not found: ID does not exist" Sep 11 04:07:28.397394 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:28.397349 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" podStartSLOduration=2.3973354479999998 podStartE2EDuration="2.397335448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:07:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:07:28.396070248 +0000 UTC m=+863.149701652" watchObservedRunningTime="2026-09-11 04:07:28.397335448 +0000 UTC m=+863.150966850" Sep 11 04:07:28.587843 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:28.587754 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:28.780484 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:28.780448 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:29.151779 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:29.151735 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" Sep 11 04:07:29.157732 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:29.157704 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" Sep 11 04:07:29.380094 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:29.379982 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" Sep 11 04:07:29.381072 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:29.381034 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" Sep 11 04:07:29.545030 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:29.545007 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8089-2513c6" Sep 11 04:07:29.613907 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:29.613871 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69p8w\" (UniqueName: \"kubernetes.io/projected/a23b7c29-0454-484a-ba19-3f387d8b1bba-kube-api-access-69p8w\") pod \"a23b7c29-0454-484a-ba19-3f387d8b1bba\" (UID: \"a23b7c29-0454-484a-ba19-3f387d8b1bba\") " Sep 11 04:07:29.615989 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:29.615966 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a23b7c29-0454-484a-ba19-3f387d8b1bba-kube-api-access-69p8w" (OuterVolumeSpecName: "kube-api-access-69p8w") pod "a23b7c29-0454-484a-ba19-3f387d8b1bba" (UID: "a23b7c29-0454-484a-ba19-3f387d8b1bba"). InnerVolumeSpecName "kube-api-access-69p8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:07:29.714983 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:29.714892 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69p8w\" (UniqueName: \"kubernetes.io/projected/a23b7c29-0454-484a-ba19-3f387d8b1bba-kube-api-access-69p8w\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:07:29.769295 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:29.769260 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a23b7c29-0454-484a-ba19-3f387d8b1bba" path="/var/lib/kubelet/pods/a23b7c29-0454-484a-ba19-3f387d8b1bba/volumes" Sep 11 04:07:30.385774 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:30.385736 2568 generic.go:356] "Generic (PLEG): container finished" podID="a23b7c29-0454-484a-ba19-3f387d8b1bba" containerID="11cb3f6bc3d295ebbe1c83cb66f1a9018e81760567910148bb9b1bcd18f4c2bc" exitCode=137 Sep 11 04:07:30.386282 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:30.385791 2568 scope.go:117] "RemoveContainer" containerID="11cb3f6bc3d295ebbe1c83cb66f1a9018e81760567910148bb9b1bcd18f4c2bc" Sep 11 04:07:30.386282 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:30.385797 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8089-2513c6" Sep 11 04:07:30.395919 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:30.395876 2568 scope.go:117] "RemoveContainer" containerID="11cb3f6bc3d295ebbe1c83cb66f1a9018e81760567910148bb9b1bcd18f4c2bc" Sep 11 04:07:30.396439 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:07:30.396415 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11cb3f6bc3d295ebbe1c83cb66f1a9018e81760567910148bb9b1bcd18f4c2bc\": container with ID starting with 11cb3f6bc3d295ebbe1c83cb66f1a9018e81760567910148bb9b1bcd18f4c2bc not found: ID does not exist" containerID="11cb3f6bc3d295ebbe1c83cb66f1a9018e81760567910148bb9b1bcd18f4c2bc" Sep 11 04:07:30.396527 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:30.396450 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11cb3f6bc3d295ebbe1c83cb66f1a9018e81760567910148bb9b1bcd18f4c2bc"} err="failed to get container status \"11cb3f6bc3d295ebbe1c83cb66f1a9018e81760567910148bb9b1bcd18f4c2bc\": rpc error: code = NotFound desc = could not find container \"11cb3f6bc3d295ebbe1c83cb66f1a9018e81760567910148bb9b1bcd18f4c2bc\": container with ID starting with 11cb3f6bc3d295ebbe1c83cb66f1a9018e81760567910148bb9b1bcd18f4c2bc not found: ID does not exist" Sep 11 04:07:30.520769 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:30.520721 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:30.614418 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:30.614380 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:31.285961 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:31.285929 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:31.390516 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:31.390477 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:31.735450 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:31.735407 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:31.924709 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:31.924673 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:32.333644 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:32.333610 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:32.426892 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:32.426854 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:32.428363 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:32.428339 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-eda9e4-695967bd58-t2g26"] Sep 11 04:07:32.432298 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:32.432281 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-eda9e4-695967bd58-t2g26" Sep 11 04:07:32.435634 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:32.435613 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-eda9e4-dockercfg-mh9kh\"" Sep 11 04:07:32.436015 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:32.435993 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-eda9e4\"" Sep 11 04:07:32.443226 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:32.443198 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-eda9e4-695967bd58-t2g26"] Sep 11 04:07:32.537481 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:32.537444 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqmtm\" (UniqueName: \"kubernetes.io/projected/20581566-083b-489a-9d5f-62ccbb4bb771-kube-api-access-lqmtm\") pod \"payload-pre-processing-e2e-autores-eda9e4-695967bd58-t2g26\" (UID: \"20581566-083b-489a-9d5f-62ccbb4bb771\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-eda9e4-695967bd58-t2g26" Sep 11 04:07:32.537651 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:32.537499 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20581566-083b-489a-9d5f-62ccbb4bb771-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-eda9e4-695967bd58-t2g26\" (UID: \"20581566-083b-489a-9d5f-62ccbb4bb771\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-eda9e4-695967bd58-t2g26" Sep 11 04:07:32.561679 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:32.561576 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:32.639004 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:32.638903 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqmtm\" (UniqueName: \"kubernetes.io/projected/20581566-083b-489a-9d5f-62ccbb4bb771-kube-api-access-lqmtm\") pod \"payload-pre-processing-e2e-autores-eda9e4-695967bd58-t2g26\" (UID: \"20581566-083b-489a-9d5f-62ccbb4bb771\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-eda9e4-695967bd58-t2g26" Sep 11 04:07:32.639357 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:32.639332 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20581566-083b-489a-9d5f-62ccbb4bb771-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-eda9e4-695967bd58-t2g26\" (UID: \"20581566-083b-489a-9d5f-62ccbb4bb771\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-eda9e4-695967bd58-t2g26" Sep 11 04:07:32.640257 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:32.640230 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20581566-083b-489a-9d5f-62ccbb4bb771-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-eda9e4-695967bd58-t2g26\" (UID: \"20581566-083b-489a-9d5f-62ccbb4bb771\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-eda9e4-695967bd58-t2g26" Sep 11 04:07:32.648799 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:32.648777 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqmtm\" (UniqueName: \"kubernetes.io/projected/20581566-083b-489a-9d5f-62ccbb4bb771-kube-api-access-lqmtm\") pod \"payload-pre-processing-e2e-autores-eda9e4-695967bd58-t2g26\" (UID: \"20581566-083b-489a-9d5f-62ccbb4bb771\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-eda9e4-695967bd58-t2g26" Sep 11 04:07:32.770071 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:32.769207 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-eda9e4-695967bd58-t2g26" Sep 11 04:07:32.773241 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:32.772567 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:32.872120 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:32.871256 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:33.051449 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:33.046646 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-eda9e4-695967bd58-t2g26"] Sep 11 04:07:33.175105 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:33.175060 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:33.361088 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:33.360975 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:33.402784 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:33.402750 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-eda9e4-695967bd58-t2g26" event={"ID":"20581566-083b-489a-9d5f-62ccbb4bb771","Type":"ContainerStarted","Data":"6b0e6e744cd432e7ced08e3550f2636f5e800463ef0a0c84e4d4ca77cb313f22"} Sep 11 04:07:33.402950 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:33.402802 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-eda9e4-695967bd58-t2g26" event={"ID":"20581566-083b-489a-9d5f-62ccbb4bb771","Type":"ContainerStarted","Data":"f32e4c38d886225a2230e4f7fc258e9fbc34ed90097180e88846d248e6d387e4"} Sep 11 04:07:33.402950 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:33.402823 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-eda9e4-695967bd58-t2g26" Sep 11 04:07:33.417718 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:33.417672 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-eda9e4-695967bd58-t2g26" podStartSLOduration=1.417658097 podStartE2EDuration="1.417658097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:07:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:07:33.416896882 +0000 UTC m=+868.170528284" watchObservedRunningTime="2026-09-11 04:07:33.417658097 +0000 UTC m=+868.171289498" Sep 11 04:07:33.458081 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:33.457989 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:33.487148 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:33.487117 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d6904f79-86fdb86df-cnkrc"] Sep 11 04:07:33.490625 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:33.490597 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d6904f79-86fdb86df-cnkrc" Sep 11 04:07:33.494528 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:33.494511 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-d6904f79-dockercfg-5hdsf\"" Sep 11 04:07:33.494651 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:33.494542 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-d6904f79\"" Sep 11 04:07:33.501586 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:33.501512 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d6904f79-86fdb86df-cnkrc"] Sep 11 04:07:33.580734 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:33.580684 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsk8r\" (UniqueName: \"kubernetes.io/projected/5c081926-0a59-4aa3-baee-a7a0bead055e-kube-api-access-wsk8r\") pod \"payload-processing-e2e-ait-d6904f79-86fdb86df-cnkrc\" (UID: \"5c081926-0a59-4aa3-baee-a7a0bead055e\") " pod="openshift-ingress/payload-processing-e2e-ait-d6904f79-86fdb86df-cnkrc" Sep 11 04:07:33.580953 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:33.580786 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c081926-0a59-4aa3-baee-a7a0bead055e-plugins-config-volume\") pod \"payload-processing-e2e-ait-d6904f79-86fdb86df-cnkrc\" (UID: \"5c081926-0a59-4aa3-baee-a7a0bead055e\") " pod="openshift-ingress/payload-processing-e2e-ait-d6904f79-86fdb86df-cnkrc" Sep 11 04:07:33.682304 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:33.682264 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c081926-0a59-4aa3-baee-a7a0bead055e-plugins-config-volume\") pod \"payload-processing-e2e-ait-d6904f79-86fdb86df-cnkrc\" (UID: \"5c081926-0a59-4aa3-baee-a7a0bead055e\") " pod="openshift-ingress/payload-processing-e2e-ait-d6904f79-86fdb86df-cnkrc" Sep 11 04:07:33.682483 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:33.682370 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsk8r\" (UniqueName: \"kubernetes.io/projected/5c081926-0a59-4aa3-baee-a7a0bead055e-kube-api-access-wsk8r\") pod \"payload-processing-e2e-ait-d6904f79-86fdb86df-cnkrc\" (UID: \"5c081926-0a59-4aa3-baee-a7a0bead055e\") " pod="openshift-ingress/payload-processing-e2e-ait-d6904f79-86fdb86df-cnkrc" Sep 11 04:07:33.683176 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:33.683154 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c081926-0a59-4aa3-baee-a7a0bead055e-plugins-config-volume\") pod \"payload-processing-e2e-ait-d6904f79-86fdb86df-cnkrc\" (UID: \"5c081926-0a59-4aa3-baee-a7a0bead055e\") " pod="openshift-ingress/payload-processing-e2e-ait-d6904f79-86fdb86df-cnkrc" Sep 11 04:07:33.691061 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:33.691031 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsk8r\" (UniqueName: \"kubernetes.io/projected/5c081926-0a59-4aa3-baee-a7a0bead055e-kube-api-access-wsk8r\") pod \"payload-processing-e2e-ait-d6904f79-86fdb86df-cnkrc\" (UID: \"5c081926-0a59-4aa3-baee-a7a0bead055e\") " pod="openshift-ingress/payload-processing-e2e-ait-d6904f79-86fdb86df-cnkrc" Sep 11 04:07:33.801591 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:33.801562 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d6904f79-86fdb86df-cnkrc" Sep 11 04:07:33.814719 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:33.814682 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:34.106682 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:34.106653 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d6904f79-86fdb86df-cnkrc"] Sep 11 04:07:34.108805 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:07:34.108761 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c081926_0a59_4aa3_baee_a7a0bead055e.slice/crio-da169948249079061976866b39e80b26ec0fd2cf61749491a503c0acffe0c317 WatchSource:0}: Error finding container da169948249079061976866b39e80b26ec0fd2cf61749491a503c0acffe0c317: Status 404 returned error can't find the container with id da169948249079061976866b39e80b26ec0fd2cf61749491a503c0acffe0c317 Sep 11 04:07:34.409782 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:34.409635 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d6904f79-86fdb86df-cnkrc" event={"ID":"5c081926-0a59-4aa3-baee-a7a0bead055e","Type":"ContainerStarted","Data":"95ebe809ad1ee023b66658d69d45fd4375d9f95857c1dc43761134ad2635e605"} Sep 11 04:07:34.409782 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:34.409713 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-d6904f79-86fdb86df-cnkrc" Sep 11 04:07:34.409782 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:34.409733 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d6904f79-86fdb86df-cnkrc" event={"ID":"5c081926-0a59-4aa3-baee-a7a0bead055e","Type":"ContainerStarted","Data":"da169948249079061976866b39e80b26ec0fd2cf61749491a503c0acffe0c317"} Sep 11 04:07:34.428117 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:34.428066 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-d6904f79-86fdb86df-cnkrc" podStartSLOduration=1.42803359 podStartE2EDuration="1.42803359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:07:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:07:34.426230006 +0000 UTC m=+869.179861409" watchObservedRunningTime="2026-09-11 04:07:34.42803359 +0000 UTC m=+869.181664995" Sep 11 04:07:34.508940 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:34.508911 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:37.367445 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:37.367365 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a28c9f-5b6c74bcbc-j7hwv" Sep 11 04:07:38.377272 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:38.377227 2568 patch_prober.go:28] interesting pod/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 11 04:07:38.377702 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:38.377291 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" podUID="136cff62-ad79-4976-81dd-ac786e4d4cec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 04:07:38.663800 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:38.663763 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:38.827735 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:38.827692 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:38.913880 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:38.913797 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:39.014486 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:39.014445 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:40.442881 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:40.442846 2568 generic.go:356] "Generic (PLEG): container finished" podID="136cff62-ad79-4976-81dd-ac786e4d4cec" containerID="2d01931adcd8522c7f52f6336c29b07b3a6ca37a8e83d26653caf9834e041989" exitCode=0 Sep 11 04:07:40.443430 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:40.442931 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" event={"ID":"136cff62-ad79-4976-81dd-ac786e4d4cec","Type":"ContainerDied","Data":"2d01931adcd8522c7f52f6336c29b07b3a6ca37a8e83d26653caf9834e041989"} Sep 11 04:07:40.443499 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:40.443471 2568 scope.go:117] "RemoveContainer" containerID="2d01931adcd8522c7f52f6336c29b07b3a6ca37a8e83d26653caf9834e041989" Sep 11 04:07:40.444579 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:40.444546 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 04:07:40.444678 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:40.444613 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 04:07:40.444678 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:40.444648 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 04:07:41.468077 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:41.465231 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" event={"ID":"136cff62-ad79-4976-81dd-ac786e4d4cec","Type":"ContainerStarted","Data":"97d702943b0b252e3faccfa9c9c412d3cce055fb30a5c6d437351f01592afe5f"} Sep 11 04:07:41.755776 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:41.755649 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:42.359149 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:42.359101 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" Sep 11 04:07:42.359384 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:42.359353 2568 patch_prober.go:28] interesting pod/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Sep 11 04:07:42.359467 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:42.359419 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" podUID="136cff62-ad79-4976-81dd-ac786e4d4cec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Sep 11 04:07:43.359947 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:43.359903 2568 patch_prober.go:28] interesting pod/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Sep 11 04:07:43.360473 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:43.359977 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" podUID="136cff62-ad79-4976-81dd-ac786e4d4cec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Sep 11 04:07:43.493839 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:43.493788 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:44.359468 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:44.359420 2568 patch_prober.go:28] interesting pod/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Sep 11 04:07:44.359815 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:44.359757 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" podUID="136cff62-ad79-4976-81dd-ac786e4d4cec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Sep 11 04:07:44.411611 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:44.411568 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-eda9e4-695967bd58-t2g26" Sep 11 04:07:45.359482 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:45.359433 2568 patch_prober.go:28] interesting pod/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Sep 11 04:07:45.359704 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:45.359506 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" podUID="136cff62-ad79-4976-81dd-ac786e4d4cec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Sep 11 04:07:45.416813 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:45.416780 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-d6904f79-86fdb86df-cnkrc" Sep 11 04:07:45.676411 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:45.676373 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:46.359384 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:46.359350 2568 patch_prober.go:28] interesting pod/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Sep 11 04:07:46.359678 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:46.359647 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" podUID="136cff62-ad79-4976-81dd-ac786e4d4cec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Sep 11 04:07:47.369136 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:47.369103 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" Sep 11 04:07:47.369576 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:47.369475 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" Sep 11 04:07:47.370330 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:47.370309 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" Sep 11 04:07:48.284062 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:48.284002 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:48.627952 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:48.627854 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:48.997678 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:48.997637 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt"] Sep 11 04:07:49.001705 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:49.001682 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" Sep 11 04:07:49.004749 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:49.004727 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-fca08f-openshift-default-dockercfg-6vqrk\"" Sep 11 04:07:49.023452 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:49.023423 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt"] Sep 11 04:07:49.030251 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:49.030217 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:49.041572 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:49.041179 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0046d7-abc0-4be9-857d-b1e9d93baeee-workload-socket\") pod \"e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt\" (UID: \"7e0046d7-abc0-4be9-857d-b1e9d93baeee\") " pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" Sep 11 04:07:49.041572 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:49.041236 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e0046d7-abc0-4be9-857d-b1e9d93baeee-istio-envoy\") pod \"e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt\" (UID: \"7e0046d7-abc0-4be9-857d-b1e9d93baeee\") " pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" Sep 11 04:07:49.041572 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:49.041261 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e0046d7-abc0-4be9-857d-b1e9d93baeee-istio-data\") pod \"e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt\" (UID: \"7e0046d7-abc0-4be9-857d-b1e9d93baeee\") " pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" Sep 11 04:07:49.041572 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:49.041284 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e0046d7-abc0-4be9-857d-b1e9d93baeee-istio-podinfo\") pod \"e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt\" (UID: \"7e0046d7-abc0-4be9-857d-b1e9d93baeee\") " pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" Sep 11 04:07:49.041572 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:49.041320 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e0046d7-abc0-4be9-857d-b1e9d93baeee-istiod-ca-cert\") pod \"e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt\" (UID: \"7e0046d7-abc0-4be9-857d-b1e9d93baeee\") " pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" Sep 11 04:07:49.041572 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:49.041353 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e0046d7-abc0-4be9-857d-b1e9d93baeee-workload-certs\") pod \"e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt\" (UID: \"7e0046d7-abc0-4be9-857d-b1e9d93baeee\") " pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" Sep 11 04:07:49.041572 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:49.041376 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e0046d7-abc0-4be9-857d-b1e9d93baeee-istio-token\") pod \"e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt\" (UID: \"7e0046d7-abc0-4be9-857d-b1e9d93baeee\") " pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" Sep 11 04:07:49.041572 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:49.041449 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0046d7-abc0-4be9-857d-b1e9d93baeee-credential-socket\") pod \"e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt\" (UID: \"7e0046d7-abc0-4be9-857d-b1e9d93baeee\") " pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" Sep 11 04:07:49.041572 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:49.041474 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52pbs\" (UniqueName: \"kubernetes.io/projected/7e0046d7-abc0-4be9-857d-b1e9d93baeee-kube-api-access-52pbs\") pod \"e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt\" (UID: \"7e0046d7-abc0-4be9-857d-b1e9d93baeee\") " pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" Sep 11 04:07:49.141976 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:49.141935 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e0046d7-abc0-4be9-857d-b1e9d93baeee-istio-envoy\") pod \"e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt\" (UID: \"7e0046d7-abc0-4be9-857d-b1e9d93baeee\") " pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" Sep 11 04:07:49.142191 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:49.141983 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e0046d7-abc0-4be9-857d-b1e9d93baeee-istio-data\") pod \"e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt\" (UID: \"7e0046d7-abc0-4be9-857d-b1e9d93baeee\") " pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" Sep 11 04:07:49.142191 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:49.142011 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e0046d7-abc0-4be9-857d-b1e9d93baeee-istio-podinfo\") pod \"e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt\" (UID: \"7e0046d7-abc0-4be9-857d-b1e9d93baeee\") " pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" Sep 11 04:07:49.142191 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:49.142065 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e0046d7-abc0-4be9-857d-b1e9d93baeee-istiod-ca-cert\") pod \"e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt\" (UID: \"7e0046d7-abc0-4be9-857d-b1e9d93baeee\") " pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" Sep 11 04:07:49.142191 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:49.142101 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e0046d7-abc0-4be9-857d-b1e9d93baeee-workload-certs\") pod \"e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt\" (UID: \"7e0046d7-abc0-4be9-857d-b1e9d93baeee\") " pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" Sep 11 04:07:49.142191 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:49.142125 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e0046d7-abc0-4be9-857d-b1e9d93baeee-istio-token\") pod \"e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt\" (UID: \"7e0046d7-abc0-4be9-857d-b1e9d93baeee\") " pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" Sep 11 04:07:49.142463 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:49.142200 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0046d7-abc0-4be9-857d-b1e9d93baeee-credential-socket\") pod \"e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt\" (UID: \"7e0046d7-abc0-4be9-857d-b1e9d93baeee\") " pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" Sep 11 04:07:49.142463 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:49.142227 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52pbs\" (UniqueName: \"kubernetes.io/projected/7e0046d7-abc0-4be9-857d-b1e9d93baeee-kube-api-access-52pbs\") pod \"e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt\" (UID: \"7e0046d7-abc0-4be9-857d-b1e9d93baeee\") " pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" Sep 11 04:07:49.142463 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:49.142298 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0046d7-abc0-4be9-857d-b1e9d93baeee-workload-socket\") pod \"e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt\" (UID: \"7e0046d7-abc0-4be9-857d-b1e9d93baeee\") " pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" Sep 11 04:07:49.142773 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:49.142748 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0046d7-abc0-4be9-857d-b1e9d93baeee-workload-socket\") pod \"e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt\" (UID: \"7e0046d7-abc0-4be9-857d-b1e9d93baeee\") " pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" Sep 11 04:07:49.144236 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:49.144203 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e0046d7-abc0-4be9-857d-b1e9d93baeee-workload-certs\") pod \"e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt\" (UID: \"7e0046d7-abc0-4be9-857d-b1e9d93baeee\") " pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" Sep 11 04:07:49.144485 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:49.144464 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e0046d7-abc0-4be9-857d-b1e9d93baeee-istio-data\") pod \"e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt\" (UID: \"7e0046d7-abc0-4be9-857d-b1e9d93baeee\") " pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" Sep 11 04:07:49.145201 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:49.145180 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0046d7-abc0-4be9-857d-b1e9d93baeee-credential-socket\") pod \"e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt\" (UID: \"7e0046d7-abc0-4be9-857d-b1e9d93baeee\") " pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" Sep 11 04:07:49.145559 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:49.145501 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e0046d7-abc0-4be9-857d-b1e9d93baeee-istio-envoy\") pod \"e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt\" (UID: \"7e0046d7-abc0-4be9-857d-b1e9d93baeee\") " pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" Sep 11 04:07:49.145817 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:49.145771 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e0046d7-abc0-4be9-857d-b1e9d93baeee-istiod-ca-cert\") pod \"e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt\" (UID: \"7e0046d7-abc0-4be9-857d-b1e9d93baeee\") " pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" Sep 11 04:07:49.147251 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:49.147226 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e0046d7-abc0-4be9-857d-b1e9d93baeee-istio-podinfo\") pod \"e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt\" (UID: \"7e0046d7-abc0-4be9-857d-b1e9d93baeee\") " pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" Sep 11 04:07:49.152955 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:49.152933 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e0046d7-abc0-4be9-857d-b1e9d93baeee-istio-token\") pod \"e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt\" (UID: \"7e0046d7-abc0-4be9-857d-b1e9d93baeee\") " pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" Sep 11 04:07:49.153316 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:49.153293 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52pbs\" (UniqueName: \"kubernetes.io/projected/7e0046d7-abc0-4be9-857d-b1e9d93baeee-kube-api-access-52pbs\") pod \"e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt\" (UID: \"7e0046d7-abc0-4be9-857d-b1e9d93baeee\") " pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" Sep 11 04:07:49.220809 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:49.220773 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:49.314652 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:49.314554 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" Sep 11 04:07:49.593733 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:49.593692 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt"] Sep 11 04:07:49.595540 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:07:49.595508 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e0046d7_abc0_4be9_857d_b1e9d93baeee.slice/crio-fd3232e5a27182f2d2d637db7dcf59768cf979c489cbca0acb39025140a7d58f WatchSource:0}: Error finding container fd3232e5a27182f2d2d637db7dcf59768cf979c489cbca0acb39025140a7d58f: Status 404 returned error can't find the container with id fd3232e5a27182f2d2d637db7dcf59768cf979c489cbca0acb39025140a7d58f Sep 11 04:07:49.597916 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:49.597875 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 04:07:49.597994 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:49.597961 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 04:07:49.598037 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:49.598000 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 04:07:50.529932 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:50.529896 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" event={"ID":"7e0046d7-abc0-4be9-857d-b1e9d93baeee","Type":"ContainerStarted","Data":"ae092688161795ed7c50a17cd3ef8cb77b52281e9a945e1a8859d44966b068e0"} Sep 11 04:07:50.530376 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:50.529950 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" event={"ID":"7e0046d7-abc0-4be9-857d-b1e9d93baeee","Type":"ContainerStarted","Data":"fd3232e5a27182f2d2d637db7dcf59768cf979c489cbca0acb39025140a7d58f"} Sep 11 04:07:50.554837 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:50.554788 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" podStartSLOduration=2.554771326 podStartE2EDuration="2.554771326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:07:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:07:50.553616951 +0000 UTC m=+885.307248357" watchObservedRunningTime="2026-09-11 04:07:50.554771326 +0000 UTC m=+885.308402727" Sep 11 04:07:51.315785 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:51.315748 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" Sep 11 04:07:51.320635 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:51.320609 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" Sep 11 04:07:51.535881 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:51.535848 2568 generic.go:356] "Generic (PLEG): container finished" podID="82fa5c18-c332-45ab-8bea-ac64f4b2caf3" containerID="2b901d7d94ac2c752e7edec8edd0e1290da11947f049812a3c5179ad1233910c" exitCode=0 Sep 11 04:07:51.536294 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:51.535921 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" event={"ID":"82fa5c18-c332-45ab-8bea-ac64f4b2caf3","Type":"ContainerDied","Data":"2b901d7d94ac2c752e7edec8edd0e1290da11947f049812a3c5179ad1233910c"} Sep 11 04:07:51.536350 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:51.536331 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" Sep 11 04:07:51.536456 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:51.536439 2568 scope.go:117] "RemoveContainer" containerID="2b901d7d94ac2c752e7edec8edd0e1290da11947f049812a3c5179ad1233910c" Sep 11 04:07:51.537304 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:51.537187 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" Sep 11 04:07:51.537598 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:51.537568 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 04:07:51.537689 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:51.537655 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 04:07:51.537758 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:51.537703 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 04:07:52.505226 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:52.505180 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:52.545494 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:52.545390 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" event={"ID":"82fa5c18-c332-45ab-8bea-ac64f4b2caf3","Type":"ContainerStarted","Data":"d341be0b05ce8b48489cb0bd1e8fef8a12319c7df9084d7723118544eabae99d"} Sep 11 04:07:52.578882 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:52.578846 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:52.943738 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:52.943708 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:53.160916 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:53.160441 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" Sep 11 04:07:53.160916 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:53.160566 2568 patch_prober.go:28] interesting pod/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Sep 11 04:07:53.160916 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:53.160612 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" podUID="82fa5c18-c332-45ab-8bea-ac64f4b2caf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Sep 11 04:07:53.723443 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:53.723402 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:53.894871 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:53.894835 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:53.983386 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:53.983282 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:54.151405 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:54.151369 2568 patch_prober.go:28] interesting pod/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Sep 11 04:07:54.151684 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:54.151656 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" podUID="82fa5c18-c332-45ab-8bea-ac64f4b2caf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Sep 11 04:07:54.155755 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:54.155718 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:54.571268 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:54.571231 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:54.667240 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:54.667192 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:55.151997 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:55.151947 2568 patch_prober.go:28] interesting pod/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Sep 11 04:07:55.152506 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:55.152022 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" podUID="82fa5c18-c332-45ab-8bea-ac64f4b2caf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Sep 11 04:07:55.375246 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:55.375211 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:55.518016 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:55.517941 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:56.137270 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:56.137235 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:56.151497 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:56.151461 2568 patch_prober.go:28] interesting pod/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Sep 11 04:07:56.151768 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:56.151741 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" podUID="82fa5c18-c332-45ab-8bea-ac64f4b2caf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Sep 11 04:07:57.126800 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:57.126761 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:57.151496 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:57.151460 2568 patch_prober.go:28] interesting pod/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Sep 11 04:07:57.151784 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:57.151749 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" podUID="82fa5c18-c332-45ab-8bea-ac64f4b2caf3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Sep 11 04:07:57.950983 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:57.950942 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:07:58.155455 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:58.155426 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" Sep 11 04:07:58.155853 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:58.155718 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" Sep 11 04:07:58.156440 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:58.156422 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" Sep 11 04:07:59.958534 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:07:59.958496 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:08:00.064108 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:00.064068 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:08:00.159843 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:00.159810 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:08:00.445194 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:00.445161 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:08:02.017495 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:02.017461 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d6904f79-5dtnv"] Sep 11 04:08:02.020882 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:02.020862 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d6904f79-5dtnv" Sep 11 04:08:02.023557 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:02.023535 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 04:08:02.023684 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:02.023583 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6brsr\"" Sep 11 04:08:02.028334 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:02.028308 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d6904f79-5dtnv"] Sep 11 04:08:02.062746 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:02.062702 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca90b718-89d0-4a2b-86ad-03b099d8fc55-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d6904f79-5dtnv\" (UID: \"ca90b718-89d0-4a2b-86ad-03b099d8fc55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d6904f79-5dtnv" Sep 11 04:08:02.169585 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:02.169545 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca90b718-89d0-4a2b-86ad-03b099d8fc55-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d6904f79-5dtnv\" (UID: \"ca90b718-89d0-4a2b-86ad-03b099d8fc55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d6904f79-5dtnv" Sep 11 04:08:02.170636 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:02.170606 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca90b718-89d0-4a2b-86ad-03b099d8fc55-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d6904f79-5dtnv\" (UID: \"ca90b718-89d0-4a2b-86ad-03b099d8fc55\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d6904f79-5dtnv" Sep 11 04:08:02.332254 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:02.332149 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d6904f79-5dtnv" Sep 11 04:08:02.633910 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:02.633880 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d6904f79-5dtnv"] Sep 11 04:08:02.635988 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:08:02.635956 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca90b718_89d0_4a2b_86ad_03b099d8fc55.slice/crio-9e7ae7786517cfcc148d033e77083a03583ac18442ea6d77ab9cfc83fccee581 WatchSource:0}: Error finding container 9e7ae7786517cfcc148d033e77083a03583ac18442ea6d77ab9cfc83fccee581: Status 404 returned error can't find the container with id 9e7ae7786517cfcc148d033e77083a03583ac18442ea6d77ab9cfc83fccee581 Sep 11 04:08:03.619823 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:03.619777 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d6904f79-5dtnv" event={"ID":"ca90b718-89d0-4a2b-86ad-03b099d8fc55","Type":"ContainerStarted","Data":"9e7ae7786517cfcc148d033e77083a03583ac18442ea6d77ab9cfc83fccee581"} Sep 11 04:08:05.736285 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:05.736252 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-5tpdc_9112557d-d0ef-44c7-8b2a-f6ecd0a980d6/console-operator/1.log" Sep 11 04:08:05.737524 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:05.737500 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-5tpdc_9112557d-d0ef-44c7-8b2a-f6ecd0a980d6/console-operator/1.log" Sep 11 04:08:06.443842 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:06.443743 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:08:06.543243 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:06.543197 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:08:06.637204 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:06.637111 2568 generic.go:356] "Generic (PLEG): container finished" podID="ca90b718-89d0-4a2b-86ad-03b099d8fc55" containerID="903a28f32ae14d7edae13fe93804da1ffb04dbf3ba3e1d4c026c026ff837a417" exitCode=0 Sep 11 04:08:06.637367 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:06.637205 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d6904f79-5dtnv" event={"ID":"ca90b718-89d0-4a2b-86ad-03b099d8fc55","Type":"ContainerDied","Data":"903a28f32ae14d7edae13fe93804da1ffb04dbf3ba3e1d4c026c026ff837a417"} Sep 11 04:08:07.579181 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:07.579141 2568 patch_prober.go:28] interesting pod/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 11 04:08:07.579641 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:07.579220 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" podUID="7e0046d7-abc0-4be9-857d-b1e9d93baeee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 04:08:07.779434 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:07.779409 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d6904f79-5dtnv" Sep 11 04:08:07.822711 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:07.822676 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca90b718-89d0-4a2b-86ad-03b099d8fc55-maas-api-service-ca\") pod \"ca90b718-89d0-4a2b-86ad-03b099d8fc55\" (UID: \"ca90b718-89d0-4a2b-86ad-03b099d8fc55\") " Sep 11 04:08:07.823136 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:07.823109 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca90b718-89d0-4a2b-86ad-03b099d8fc55-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ca90b718-89d0-4a2b-86ad-03b099d8fc55" (UID: "ca90b718-89d0-4a2b-86ad-03b099d8fc55"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 04:08:07.924137 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:07.924103 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca90b718-89d0-4a2b-86ad-03b099d8fc55-maas-api-service-ca\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:08:08.651031 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:08.650997 2568 generic.go:356] "Generic (PLEG): container finished" podID="7e0046d7-abc0-4be9-857d-b1e9d93baeee" containerID="ae092688161795ed7c50a17cd3ef8cb77b52281e9a945e1a8859d44966b068e0" exitCode=0 Sep 11 04:08:08.651499 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:08.651074 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" event={"ID":"7e0046d7-abc0-4be9-857d-b1e9d93baeee","Type":"ContainerDied","Data":"ae092688161795ed7c50a17cd3ef8cb77b52281e9a945e1a8859d44966b068e0"} Sep 11 04:08:08.651572 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:08.651555 2568 scope.go:117] "RemoveContainer" containerID="ae092688161795ed7c50a17cd3ef8cb77b52281e9a945e1a8859d44966b068e0" Sep 11 04:08:08.652612 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:08.652592 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d6904f79-5dtnv" Sep 11 04:08:08.652612 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:08.652593 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d6904f79-5dtnv" event={"ID":"ca90b718-89d0-4a2b-86ad-03b099d8fc55","Type":"ContainerDied","Data":"9e7ae7786517cfcc148d033e77083a03583ac18442ea6d77ab9cfc83fccee581"} Sep 11 04:08:08.652746 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:08.652650 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e7ae7786517cfcc148d033e77083a03583ac18442ea6d77ab9cfc83fccee581" Sep 11 04:08:08.652830 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:08.652797 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 04:08:08.652908 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:08.652891 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 04:08:08.652963 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:08.652944 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 04:08:09.659993 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:09.659958 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" event={"ID":"7e0046d7-abc0-4be9-857d-b1e9d93baeee","Type":"ContainerStarted","Data":"6d0170a66dcd7ebb2cabe82421c3aa1c6cb5bf324e27babfe301eac68a6ac24c"} Sep 11 04:08:10.315304 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:10.315268 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" Sep 11 04:08:10.315484 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:10.315422 2568 patch_prober.go:28] interesting pod/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Sep 11 04:08:10.315530 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:10.315479 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" podUID="7e0046d7-abc0-4be9-857d-b1e9d93baeee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Sep 11 04:08:11.315231 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:11.315193 2568 patch_prober.go:28] interesting pod/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Sep 11 04:08:11.315636 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:11.315259 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" podUID="7e0046d7-abc0-4be9-857d-b1e9d93baeee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Sep 11 04:08:12.315093 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:12.315034 2568 patch_prober.go:28] interesting pod/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Sep 11 04:08:12.315271 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:12.315124 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" podUID="7e0046d7-abc0-4be9-857d-b1e9d93baeee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Sep 11 04:08:13.164989 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:13.164956 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:08:13.186672 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:13.186637 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d6904f79-86fdb86df-cnkrc"] Sep 11 04:08:13.186892 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:13.186864 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-d6904f79-86fdb86df-cnkrc" podUID="5c081926-0a59-4aa3-baee-a7a0bead055e" containerName="payload-processing" containerID="cri-o://95ebe809ad1ee023b66658d69d45fd4375d9f95857c1dc43761134ad2635e605" gracePeriod=30 Sep 11 04:08:13.273620 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:13.273587 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:08:13.315099 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:13.315041 2568 patch_prober.go:28] interesting pod/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Sep 11 04:08:13.315247 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:13.315126 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" podUID="7e0046d7-abc0-4be9-857d-b1e9d93baeee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Sep 11 04:08:13.369068 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:13.364724 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d6904f79-5dtnv"] Sep 11 04:08:13.369068 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:13.364784 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d6904f79-5dtnv"] Sep 11 04:08:13.369635 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:13.369314 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:08:13.437949 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:13.437859 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:08:13.512572 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:13.512548 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d6904f79-86fdb86df-cnkrc" Sep 11 04:08:13.529344 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:13.529316 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:08:13.578591 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:13.578553 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wsk8r\" (UniqueName: \"kubernetes.io/projected/5c081926-0a59-4aa3-baee-a7a0bead055e-kube-api-access-wsk8r\") pod \"5c081926-0a59-4aa3-baee-a7a0bead055e\" (UID: \"5c081926-0a59-4aa3-baee-a7a0bead055e\") " Sep 11 04:08:13.578785 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:13.578596 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c081926-0a59-4aa3-baee-a7a0bead055e-plugins-config-volume\") pod \"5c081926-0a59-4aa3-baee-a7a0bead055e\" (UID: \"5c081926-0a59-4aa3-baee-a7a0bead055e\") " Sep 11 04:08:13.579016 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:13.578987 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c081926-0a59-4aa3-baee-a7a0bead055e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5c081926-0a59-4aa3-baee-a7a0bead055e" (UID: "5c081926-0a59-4aa3-baee-a7a0bead055e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 04:08:13.580605 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:13.580585 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c081926-0a59-4aa3-baee-a7a0bead055e-kube-api-access-wsk8r" (OuterVolumeSpecName: "kube-api-access-wsk8r") pod "5c081926-0a59-4aa3-baee-a7a0bead055e" (UID: "5c081926-0a59-4aa3-baee-a7a0bead055e"). InnerVolumeSpecName "kube-api-access-wsk8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:08:13.679854 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:13.679820 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsk8r\" (UniqueName: \"kubernetes.io/projected/5c081926-0a59-4aa3-baee-a7a0bead055e-kube-api-access-wsk8r\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:08:13.679854 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:13.679852 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c081926-0a59-4aa3-baee-a7a0bead055e-plugins-config-volume\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:08:13.683326 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:13.683297 2568 generic.go:356] "Generic (PLEG): container finished" podID="5c081926-0a59-4aa3-baee-a7a0bead055e" containerID="95ebe809ad1ee023b66658d69d45fd4375d9f95857c1dc43761134ad2635e605" exitCode=0 Sep 11 04:08:13.683469 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:13.683342 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-d6904f79-86fdb86df-cnkrc" Sep 11 04:08:13.683469 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:13.683384 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d6904f79-86fdb86df-cnkrc" event={"ID":"5c081926-0a59-4aa3-baee-a7a0bead055e","Type":"ContainerDied","Data":"95ebe809ad1ee023b66658d69d45fd4375d9f95857c1dc43761134ad2635e605"} Sep 11 04:08:13.683469 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:13.683443 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-d6904f79-86fdb86df-cnkrc" event={"ID":"5c081926-0a59-4aa3-baee-a7a0bead055e","Type":"ContainerDied","Data":"da169948249079061976866b39e80b26ec0fd2cf61749491a503c0acffe0c317"} Sep 11 04:08:13.683634 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:13.683473 2568 scope.go:117] "RemoveContainer" containerID="95ebe809ad1ee023b66658d69d45fd4375d9f95857c1dc43761134ad2635e605" Sep 11 04:08:13.696215 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:13.696194 2568 scope.go:117] "RemoveContainer" containerID="95ebe809ad1ee023b66658d69d45fd4375d9f95857c1dc43761134ad2635e605" Sep 11 04:08:13.696502 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:08:13.696482 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95ebe809ad1ee023b66658d69d45fd4375d9f95857c1dc43761134ad2635e605\": container with ID starting with 95ebe809ad1ee023b66658d69d45fd4375d9f95857c1dc43761134ad2635e605 not found: ID does not exist" containerID="95ebe809ad1ee023b66658d69d45fd4375d9f95857c1dc43761134ad2635e605" Sep 11 04:08:13.696553 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:13.696510 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95ebe809ad1ee023b66658d69d45fd4375d9f95857c1dc43761134ad2635e605"} err="failed to get container status \"95ebe809ad1ee023b66658d69d45fd4375d9f95857c1dc43761134ad2635e605\": rpc error: code = NotFound desc = could not find container \"95ebe809ad1ee023b66658d69d45fd4375d9f95857c1dc43761134ad2635e605\": container with ID starting with 95ebe809ad1ee023b66658d69d45fd4375d9f95857c1dc43761134ad2635e605 not found: ID does not exist" Sep 11 04:08:13.708516 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:13.708486 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d6904f79-86fdb86df-cnkrc"] Sep 11 04:08:13.712845 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:13.712823 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-d6904f79-86fdb86df-cnkrc"] Sep 11 04:08:13.771778 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:13.771738 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c081926-0a59-4aa3-baee-a7a0bead055e" path="/var/lib/kubelet/pods/5c081926-0a59-4aa3-baee-a7a0bead055e/volumes" Sep 11 04:08:13.772186 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:13.772165 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca90b718-89d0-4a2b-86ad-03b099d8fc55" path="/var/lib/kubelet/pods/ca90b718-89d0-4a2b-86ad-03b099d8fc55/volumes" Sep 11 04:08:14.026911 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:14.026810 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:08:14.120289 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:14.120252 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:08:14.264348 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:14.264312 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:08:14.315579 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:14.315503 2568 patch_prober.go:28] interesting pod/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Sep 11 04:08:14.315711 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:14.315581 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" podUID="7e0046d7-abc0-4be9-857d-b1e9d93baeee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Sep 11 04:08:14.578150 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:14.577468 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls"] Sep 11 04:08:14.578150 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:14.578014 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c081926-0a59-4aa3-baee-a7a0bead055e" containerName="payload-processing" Sep 11 04:08:14.578150 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:14.578035 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c081926-0a59-4aa3-baee-a7a0bead055e" containerName="payload-processing" Sep 11 04:08:14.578150 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:14.578083 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca90b718-89d0-4a2b-86ad-03b099d8fc55" containerName="revoke-keys" Sep 11 04:08:14.578150 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:14.578092 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca90b718-89d0-4a2b-86ad-03b099d8fc55" containerName="revoke-keys" Sep 11 04:08:14.578810 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:14.578182 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c081926-0a59-4aa3-baee-a7a0bead055e" containerName="payload-processing" Sep 11 04:08:14.578810 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:14.578198 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca90b718-89d0-4a2b-86ad-03b099d8fc55" containerName="revoke-keys" Sep 11 04:08:14.583429 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:14.583085 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls"] Sep 11 04:08:14.583429 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:14.583203 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" Sep 11 04:08:14.585738 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:14.585715 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-52e7067f-gw-openshift-default-dockercfg-h2mmp\"" Sep 11 04:08:14.590242 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:14.590220 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c8ff345-db48-4038-9ffe-e443ccd672b4-istio-podinfo\") pod \"e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls\" (UID: \"9c8ff345-db48-4038-9ffe-e443ccd672b4\") " pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" Sep 11 04:08:14.590391 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:14.590378 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c8ff345-db48-4038-9ffe-e443ccd672b4-workload-certs\") pod \"e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls\" (UID: \"9c8ff345-db48-4038-9ffe-e443ccd672b4\") " pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" Sep 11 04:08:14.590491 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:14.590481 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7ss8\" (UniqueName: \"kubernetes.io/projected/9c8ff345-db48-4038-9ffe-e443ccd672b4-kube-api-access-j7ss8\") pod \"e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls\" (UID: \"9c8ff345-db48-4038-9ffe-e443ccd672b4\") " pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" Sep 11 04:08:14.590559 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:14.590550 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8ff345-db48-4038-9ffe-e443ccd672b4-workload-socket\") pod \"e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls\" (UID: \"9c8ff345-db48-4038-9ffe-e443ccd672b4\") " pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" Sep 11 04:08:14.590634 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:14.590623 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c8ff345-db48-4038-9ffe-e443ccd672b4-istio-token\") pod \"e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls\" (UID: \"9c8ff345-db48-4038-9ffe-e443ccd672b4\") " pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" Sep 11 04:08:14.590702 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:14.590693 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c8ff345-db48-4038-9ffe-e443ccd672b4-istiod-ca-cert\") pod \"e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls\" (UID: \"9c8ff345-db48-4038-9ffe-e443ccd672b4\") " pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" Sep 11 04:08:14.590773 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:14.590763 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c8ff345-db48-4038-9ffe-e443ccd672b4-istio-data\") pod \"e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls\" (UID: \"9c8ff345-db48-4038-9ffe-e443ccd672b4\") " pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" Sep 11 04:08:14.591257 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:14.591234 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8ff345-db48-4038-9ffe-e443ccd672b4-credential-socket\") pod \"e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls\" (UID: \"9c8ff345-db48-4038-9ffe-e443ccd672b4\") " pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" Sep 11 04:08:14.591418 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:14.591405 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c8ff345-db48-4038-9ffe-e443ccd672b4-istio-envoy\") pod \"e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls\" (UID: \"9c8ff345-db48-4038-9ffe-e443ccd672b4\") " pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" Sep 11 04:08:14.593296 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:14.593214 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:08:14.692509 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:14.692475 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c8ff345-db48-4038-9ffe-e443ccd672b4-istio-podinfo\") pod \"e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls\" (UID: \"9c8ff345-db48-4038-9ffe-e443ccd672b4\") " pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" Sep 11 04:08:14.692924 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:14.692906 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c8ff345-db48-4038-9ffe-e443ccd672b4-workload-certs\") pod \"e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls\" (UID: \"9c8ff345-db48-4038-9ffe-e443ccd672b4\") " pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" Sep 11 04:08:14.692992 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:14.692944 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7ss8\" (UniqueName: \"kubernetes.io/projected/9c8ff345-db48-4038-9ffe-e443ccd672b4-kube-api-access-j7ss8\") pod \"e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls\" (UID: \"9c8ff345-db48-4038-9ffe-e443ccd672b4\") " pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" Sep 11 04:08:14.692992 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:14.692986 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8ff345-db48-4038-9ffe-e443ccd672b4-workload-socket\") pod \"e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls\" (UID: \"9c8ff345-db48-4038-9ffe-e443ccd672b4\") " pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" Sep 11 04:08:14.693120 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:14.693002 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c8ff345-db48-4038-9ffe-e443ccd672b4-istio-token\") pod \"e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls\" (UID: \"9c8ff345-db48-4038-9ffe-e443ccd672b4\") " pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" Sep 11 04:08:14.693120 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:14.693027 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c8ff345-db48-4038-9ffe-e443ccd672b4-istiod-ca-cert\") pod \"e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls\" (UID: \"9c8ff345-db48-4038-9ffe-e443ccd672b4\") " pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" Sep 11 04:08:14.693120 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:14.693078 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c8ff345-db48-4038-9ffe-e443ccd672b4-istio-data\") pod \"e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls\" (UID: \"9c8ff345-db48-4038-9ffe-e443ccd672b4\") " pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" Sep 11 04:08:14.693120 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:14.693098 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8ff345-db48-4038-9ffe-e443ccd672b4-credential-socket\") pod \"e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls\" (UID: \"9c8ff345-db48-4038-9ffe-e443ccd672b4\") " pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" Sep 11 04:08:14.693333 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:14.693151 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c8ff345-db48-4038-9ffe-e443ccd672b4-istio-envoy\") pod \"e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls\" (UID: \"9c8ff345-db48-4038-9ffe-e443ccd672b4\") " pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" Sep 11 04:08:14.693898 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:14.693876 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8ff345-db48-4038-9ffe-e443ccd672b4-credential-socket\") pod \"e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls\" (UID: \"9c8ff345-db48-4038-9ffe-e443ccd672b4\") " pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" Sep 11 04:08:14.694020 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:14.693973 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c8ff345-db48-4038-9ffe-e443ccd672b4-workload-certs\") pod \"e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls\" (UID: \"9c8ff345-db48-4038-9ffe-e443ccd672b4\") " pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" Sep 11 04:08:14.701969 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:14.701256 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c8ff345-db48-4038-9ffe-e443ccd672b4-istio-data\") pod \"e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls\" (UID: \"9c8ff345-db48-4038-9ffe-e443ccd672b4\") " pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" Sep 11 04:08:14.702502 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:14.702273 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8ff345-db48-4038-9ffe-e443ccd672b4-workload-socket\") pod \"e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls\" (UID: \"9c8ff345-db48-4038-9ffe-e443ccd672b4\") " pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" Sep 11 04:08:14.702989 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:14.702913 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c8ff345-db48-4038-9ffe-e443ccd672b4-istio-podinfo\") pod \"e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls\" (UID: \"9c8ff345-db48-4038-9ffe-e443ccd672b4\") " pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" Sep 11 04:08:14.703302 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:14.703281 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c8ff345-db48-4038-9ffe-e443ccd672b4-istiod-ca-cert\") pod \"e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls\" (UID: \"9c8ff345-db48-4038-9ffe-e443ccd672b4\") " pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" Sep 11 04:08:14.704549 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:14.704528 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c8ff345-db48-4038-9ffe-e443ccd672b4-istio-envoy\") pod \"e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls\" (UID: \"9c8ff345-db48-4038-9ffe-e443ccd672b4\") " pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" Sep 11 04:08:14.711904 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:14.711864 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c8ff345-db48-4038-9ffe-e443ccd672b4-istio-token\") pod \"e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls\" (UID: \"9c8ff345-db48-4038-9ffe-e443ccd672b4\") " pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" Sep 11 04:08:14.713310 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:14.713289 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7ss8\" (UniqueName: \"kubernetes.io/projected/9c8ff345-db48-4038-9ffe-e443ccd672b4-kube-api-access-j7ss8\") pod \"e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls\" (UID: \"9c8ff345-db48-4038-9ffe-e443ccd672b4\") " pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" Sep 11 04:08:14.895360 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:14.895272 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" Sep 11 04:08:15.213229 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:15.213203 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls"] Sep 11 04:08:15.216029 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:08:15.215994 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c8ff345_db48_4038_9ffe_e443ccd672b4.slice/crio-1fb92453f5a0eae23fbb250706c1bbde653b39e003e31700de0873799a9311c0 WatchSource:0}: Error finding container 1fb92453f5a0eae23fbb250706c1bbde653b39e003e31700de0873799a9311c0: Status 404 returned error can't find the container with id 1fb92453f5a0eae23fbb250706c1bbde653b39e003e31700de0873799a9311c0 Sep 11 04:08:15.218819 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:15.218783 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 04:08:15.218920 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:15.218857 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 04:08:15.218920 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:15.218894 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 04:08:15.315633 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:15.315524 2568 patch_prober.go:28] interesting pod/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Sep 11 04:08:15.315633 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:15.315594 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" podUID="7e0046d7-abc0-4be9-857d-b1e9d93baeee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Sep 11 04:08:15.331539 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:15.331448 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:08:15.533150 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:15.533061 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:08:15.698339 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:15.698301 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" event={"ID":"9c8ff345-db48-4038-9ffe-e443ccd672b4","Type":"ContainerStarted","Data":"b5b5bfd5b0093b6c307f91b07d0b9a2b320f2cf2b1b2428cc0640c45d463a30b"} Sep 11 04:08:15.698845 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:15.698353 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" event={"ID":"9c8ff345-db48-4038-9ffe-e443ccd672b4","Type":"ContainerStarted","Data":"1fb92453f5a0eae23fbb250706c1bbde653b39e003e31700de0873799a9311c0"} Sep 11 04:08:15.730400 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:15.730342 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" podStartSLOduration=1.730325396 podStartE2EDuration="1.730325396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:08:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:08:15.728431566 +0000 UTC m=+910.482062970" watchObservedRunningTime="2026-09-11 04:08:15.730325396 +0000 UTC m=+910.483956800" Sep 11 04:08:15.896181 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:15.896096 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" Sep 11 04:08:16.318846 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:16.318813 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" Sep 11 04:08:16.319098 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:16.319077 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" Sep 11 04:08:16.319757 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:16.319736 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" Sep 11 04:08:16.900895 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:16.900864 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" Sep 11 04:08:17.304761 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:17.304673 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:08:17.400034 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:17.400001 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:08:17.469829 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:17.469782 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:08:17.570568 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:17.570483 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:08:17.708986 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:17.708946 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" Sep 11 04:08:17.709704 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:17.709680 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" Sep 11 04:08:18.239184 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:18.239145 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:08:18.335512 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:18.335476 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:08:18.438279 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:18.438239 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-52e7067f-674857fb4b-8lb5t"] Sep 11 04:08:18.445222 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:18.445202 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-52e7067f-674857fb4b-8lb5t" Sep 11 04:08:18.447714 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:18.447691 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-52e7067f\"" Sep 11 04:08:18.447842 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:18.447788 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-52e7067f-dockercfg-6j5qm\"" Sep 11 04:08:18.455589 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:18.455567 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-52e7067f-674857fb4b-8lb5t"] Sep 11 04:08:18.530331 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:18.530247 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ee77ee9-bf6d-4beb-b000-da6ac157891d-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-52e7067f-674857fb4b-8lb5t\" (UID: \"1ee77ee9-bf6d-4beb-b000-da6ac157891d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-52e7067f-674857fb4b-8lb5t" Sep 11 04:08:18.530527 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:18.530378 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs2vl\" (UniqueName: \"kubernetes.io/projected/1ee77ee9-bf6d-4beb-b000-da6ac157891d-kube-api-access-qs2vl\") pod \"payload-pre-processing-e2e-migrate-52e7067f-674857fb4b-8lb5t\" (UID: \"1ee77ee9-bf6d-4beb-b000-da6ac157891d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-52e7067f-674857fb4b-8lb5t" Sep 11 04:08:18.631519 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:18.631477 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs2vl\" (UniqueName: \"kubernetes.io/projected/1ee77ee9-bf6d-4beb-b000-da6ac157891d-kube-api-access-qs2vl\") pod \"payload-pre-processing-e2e-migrate-52e7067f-674857fb4b-8lb5t\" (UID: \"1ee77ee9-bf6d-4beb-b000-da6ac157891d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-52e7067f-674857fb4b-8lb5t" Sep 11 04:08:18.631706 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:18.631566 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ee77ee9-bf6d-4beb-b000-da6ac157891d-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-52e7067f-674857fb4b-8lb5t\" (UID: \"1ee77ee9-bf6d-4beb-b000-da6ac157891d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-52e7067f-674857fb4b-8lb5t" Sep 11 04:08:18.632344 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:18.632322 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ee77ee9-bf6d-4beb-b000-da6ac157891d-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-52e7067f-674857fb4b-8lb5t\" (UID: \"1ee77ee9-bf6d-4beb-b000-da6ac157891d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-52e7067f-674857fb4b-8lb5t" Sep 11 04:08:18.639391 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:18.639367 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs2vl\" (UniqueName: \"kubernetes.io/projected/1ee77ee9-bf6d-4beb-b000-da6ac157891d-kube-api-access-qs2vl\") pod \"payload-pre-processing-e2e-migrate-52e7067f-674857fb4b-8lb5t\" (UID: \"1ee77ee9-bf6d-4beb-b000-da6ac157891d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-52e7067f-674857fb4b-8lb5t" Sep 11 04:08:18.757372 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:18.757328 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-52e7067f-674857fb4b-8lb5t" Sep 11 04:08:19.031984 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:19.031935 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-52e7067f-674857fb4b-8lb5t"] Sep 11 04:08:19.034209 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:08:19.034182 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ee77ee9_bf6d_4beb_b000_da6ac157891d.slice/crio-1ca301468440a4c2078ed63f645361abd7716bb48984d2f31b326f51eed1e33b WatchSource:0}: Error finding container 1ca301468440a4c2078ed63f645361abd7716bb48984d2f31b326f51eed1e33b: Status 404 returned error can't find the container with id 1ca301468440a4c2078ed63f645361abd7716bb48984d2f31b326f51eed1e33b Sep 11 04:08:19.719695 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:19.719658 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-52e7067f-674857fb4b-8lb5t" event={"ID":"1ee77ee9-bf6d-4beb-b000-da6ac157891d","Type":"ContainerStarted","Data":"dba8f4f211244358916ce0ed89995e100c1e893c0b1cd1fbac829edb64e04b8d"} Sep 11 04:08:19.720186 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:19.719716 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-52e7067f-674857fb4b-8lb5t" event={"ID":"1ee77ee9-bf6d-4beb-b000-da6ac157891d","Type":"ContainerStarted","Data":"1ca301468440a4c2078ed63f645361abd7716bb48984d2f31b326f51eed1e33b"} Sep 11 04:08:19.745750 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:19.745700 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-52e7067f-674857fb4b-8lb5t" podStartSLOduration=1.745684048 podStartE2EDuration="1.745684048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:08:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:08:19.744549121 +0000 UTC m=+914.498180526" watchObservedRunningTime="2026-09-11 04:08:19.745684048 +0000 UTC m=+914.499315450" Sep 11 04:08:20.725446 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:20.725416 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-52e7067f-674857fb4b-8lb5t" Sep 11 04:08:31.732731 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:31.732700 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-52e7067f-674857fb4b-8lb5t" Sep 11 04:08:32.710256 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:32.710218 2568 patch_prober.go:28] interesting pod/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Sep 11 04:08:32.710476 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:32.710277 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" podUID="9c8ff345-db48-4038-9ffe-e443ccd672b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Sep 11 04:08:32.809346 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:32.809314 2568 generic.go:356] "Generic (PLEG): container finished" podID="9c8ff345-db48-4038-9ffe-e443ccd672b4" containerID="b5b5bfd5b0093b6c307f91b07d0b9a2b320f2cf2b1b2428cc0640c45d463a30b" exitCode=0 Sep 11 04:08:32.809691 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:32.809390 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" event={"ID":"9c8ff345-db48-4038-9ffe-e443ccd672b4","Type":"ContainerDied","Data":"b5b5bfd5b0093b6c307f91b07d0b9a2b320f2cf2b1b2428cc0640c45d463a30b"} Sep 11 04:08:32.809867 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:32.809850 2568 scope.go:117] "RemoveContainer" containerID="b5b5bfd5b0093b6c307f91b07d0b9a2b320f2cf2b1b2428cc0640c45d463a30b" Sep 11 04:08:32.811030 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:32.811001 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 04:08:32.811115 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:32.811100 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 04:08:32.811154 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:32.811142 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 04:08:33.816346 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:33.816314 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" event={"ID":"9c8ff345-db48-4038-9ffe-e443ccd672b4","Type":"ContainerStarted","Data":"23787a237a9f981f582bfee34d22edff8183a9874ef29014cd3f01e5af253f36"} Sep 11 04:08:33.895434 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:33.895390 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" Sep 11 04:08:33.895697 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:33.895674 2568 patch_prober.go:28] interesting pod/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Sep 11 04:08:33.895746 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:33.895722 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" podUID="9c8ff345-db48-4038-9ffe-e443ccd672b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Sep 11 04:08:34.895629 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:34.895597 2568 patch_prober.go:28] interesting pod/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Sep 11 04:08:34.895997 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:34.895645 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" podUID="9c8ff345-db48-4038-9ffe-e443ccd672b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Sep 11 04:08:35.896244 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:35.896202 2568 patch_prober.go:28] interesting pod/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Sep 11 04:08:35.896643 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:35.896259 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" podUID="9c8ff345-db48-4038-9ffe-e443ccd672b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Sep 11 04:08:36.896506 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:36.896470 2568 patch_prober.go:28] interesting pod/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Sep 11 04:08:36.896906 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:36.896522 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" podUID="9c8ff345-db48-4038-9ffe-e443ccd672b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Sep 11 04:08:37.896553 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:37.896505 2568 patch_prober.go:28] interesting pod/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Sep 11 04:08:37.896959 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:37.896572 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" podUID="9c8ff345-db48-4038-9ffe-e443ccd672b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Sep 11 04:08:38.896145 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:38.896106 2568 patch_prober.go:28] interesting pod/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Sep 11 04:08:38.896324 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:38.896177 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" podUID="9c8ff345-db48-4038-9ffe-e443ccd672b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Sep 11 04:08:39.900104 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:39.900073 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" Sep 11 04:08:39.900481 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:39.900400 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" Sep 11 04:08:39.901120 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:39.901100 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" Sep 11 04:08:42.272795 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:42.272758 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:08:59.318168 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:59.318129 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:08:59.359042 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:59.358991 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-52e7067f-674857fb4b-8lb5t"] Sep 11 04:08:59.359351 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:59.359318 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-52e7067f-674857fb4b-8lb5t" podUID="1ee77ee9-bf6d-4beb-b000-da6ac157891d" containerName="payload-pre-processing" containerID="cri-o://dba8f4f211244358916ce0ed89995e100c1e893c0b1cd1fbac829edb64e04b8d" gracePeriod=30 Sep 11 04:08:59.640843 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:59.640808 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-52e7067f-674857fb4b-8lb5t" Sep 11 04:08:59.717156 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:59.717124 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qs2vl\" (UniqueName: \"kubernetes.io/projected/1ee77ee9-bf6d-4beb-b000-da6ac157891d-kube-api-access-qs2vl\") pod \"1ee77ee9-bf6d-4beb-b000-da6ac157891d\" (UID: \"1ee77ee9-bf6d-4beb-b000-da6ac157891d\") " Sep 11 04:08:59.717367 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:59.717237 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ee77ee9-bf6d-4beb-b000-da6ac157891d-plugins-config-volume\") pod \"1ee77ee9-bf6d-4beb-b000-da6ac157891d\" (UID: \"1ee77ee9-bf6d-4beb-b000-da6ac157891d\") " Sep 11 04:08:59.717596 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:59.717570 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ee77ee9-bf6d-4beb-b000-da6ac157891d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1ee77ee9-bf6d-4beb-b000-da6ac157891d" (UID: "1ee77ee9-bf6d-4beb-b000-da6ac157891d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 04:08:59.719244 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:59.719223 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ee77ee9-bf6d-4beb-b000-da6ac157891d-kube-api-access-qs2vl" (OuterVolumeSpecName: "kube-api-access-qs2vl") pod "1ee77ee9-bf6d-4beb-b000-da6ac157891d" (UID: "1ee77ee9-bf6d-4beb-b000-da6ac157891d"). InnerVolumeSpecName "kube-api-access-qs2vl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:08:59.788472 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:59.788441 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls"] Sep 11 04:08:59.788658 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:59.788623 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" podUID="9c8ff345-db48-4038-9ffe-e443ccd672b4" containerName="istio-proxy" containerID="cri-o://23787a237a9f981f582bfee34d22edff8183a9874ef29014cd3f01e5af253f36" gracePeriod=30 Sep 11 04:08:59.818815 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:59.818790 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ee77ee9-bf6d-4beb-b000-da6ac157891d-plugins-config-volume\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:08:59.818815 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:59.818815 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs2vl\" (UniqueName: \"kubernetes.io/projected/1ee77ee9-bf6d-4beb-b000-da6ac157891d-kube-api-access-qs2vl\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:08:59.946789 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:59.946751 2568 generic.go:356] "Generic (PLEG): container finished" podID="1ee77ee9-bf6d-4beb-b000-da6ac157891d" containerID="dba8f4f211244358916ce0ed89995e100c1e893c0b1cd1fbac829edb64e04b8d" exitCode=0 Sep 11 04:08:59.946962 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:59.946806 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-52e7067f-674857fb4b-8lb5t" Sep 11 04:08:59.946962 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:59.946813 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-52e7067f-674857fb4b-8lb5t" event={"ID":"1ee77ee9-bf6d-4beb-b000-da6ac157891d","Type":"ContainerDied","Data":"dba8f4f211244358916ce0ed89995e100c1e893c0b1cd1fbac829edb64e04b8d"} Sep 11 04:08:59.946962 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:59.946871 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-52e7067f-674857fb4b-8lb5t" event={"ID":"1ee77ee9-bf6d-4beb-b000-da6ac157891d","Type":"ContainerDied","Data":"1ca301468440a4c2078ed63f645361abd7716bb48984d2f31b326f51eed1e33b"} Sep 11 04:08:59.946962 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:59.946894 2568 scope.go:117] "RemoveContainer" containerID="dba8f4f211244358916ce0ed89995e100c1e893c0b1cd1fbac829edb64e04b8d" Sep 11 04:08:59.955663 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:59.955643 2568 scope.go:117] "RemoveContainer" containerID="dba8f4f211244358916ce0ed89995e100c1e893c0b1cd1fbac829edb64e04b8d" Sep 11 04:08:59.955942 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:08:59.955923 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dba8f4f211244358916ce0ed89995e100c1e893c0b1cd1fbac829edb64e04b8d\": container with ID starting with dba8f4f211244358916ce0ed89995e100c1e893c0b1cd1fbac829edb64e04b8d not found: ID does not exist" containerID="dba8f4f211244358916ce0ed89995e100c1e893c0b1cd1fbac829edb64e04b8d" Sep 11 04:08:59.956021 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:59.955949 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dba8f4f211244358916ce0ed89995e100c1e893c0b1cd1fbac829edb64e04b8d"} err="failed to get container status \"dba8f4f211244358916ce0ed89995e100c1e893c0b1cd1fbac829edb64e04b8d\": rpc error: code = NotFound desc = could not find container \"dba8f4f211244358916ce0ed89995e100c1e893c0b1cd1fbac829edb64e04b8d\": container with ID starting with dba8f4f211244358916ce0ed89995e100c1e893c0b1cd1fbac829edb64e04b8d not found: ID does not exist" Sep 11 04:08:59.971365 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:59.971345 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-52e7067f-674857fb4b-8lb5t"] Sep 11 04:08:59.988070 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:08:59.988019 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-52e7067f-674857fb4b-8lb5t"] Sep 11 04:09:00.939454 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:00.939410 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:09:01.339664 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:01.339570 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:09:01.768562 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:01.768532 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ee77ee9-bf6d-4beb-b000-da6ac157891d" path="/var/lib/kubelet/pods/1ee77ee9-bf6d-4beb-b000-da6ac157891d/volumes" Sep 11 04:09:02.886254 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:02.886213 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:09:02.999256 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:02.999217 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:09:04.088268 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:04.088226 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:09:04.196702 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:04.196664 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:09:04.395375 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:04.395296 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:09:04.733883 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:04.733844 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:09:04.829174 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:04.829141 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:09:04.943968 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:04.943931 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-26814939-6b4cdc4d6b-gbx9h"] Sep 11 04:09:04.944333 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:04.944316 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ee77ee9-bf6d-4beb-b000-da6ac157891d" containerName="payload-pre-processing" Sep 11 04:09:04.944333 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:04.944331 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ee77ee9-bf6d-4beb-b000-da6ac157891d" containerName="payload-pre-processing" Sep 11 04:09:04.944497 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:04.944398 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ee77ee9-bf6d-4beb-b000-da6ac157891d" containerName="payload-pre-processing" Sep 11 04:09:04.947499 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:04.947476 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-26814939-6b4cdc4d6b-gbx9h" Sep 11 04:09:04.950738 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:04.950491 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-26814939-dockercfg-gwnxw\"" Sep 11 04:09:04.950938 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:04.950880 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-26814939\"" Sep 11 04:09:04.959346 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:04.959322 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-26814939-6b4cdc4d6b-gbx9h"] Sep 11 04:09:04.982506 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:04.982411 2568 generic.go:356] "Generic (PLEG): container finished" podID="9c8ff345-db48-4038-9ffe-e443ccd672b4" containerID="23787a237a9f981f582bfee34d22edff8183a9874ef29014cd3f01e5af253f36" exitCode=0 Sep 11 04:09:04.982506 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:04.982483 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" event={"ID":"9c8ff345-db48-4038-9ffe-e443ccd672b4","Type":"ContainerDied","Data":"23787a237a9f981f582bfee34d22edff8183a9874ef29014cd3f01e5af253f36"} Sep 11 04:09:04.982798 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:04.982564 2568 scope.go:117] "RemoveContainer" containerID="b5b5bfd5b0093b6c307f91b07d0b9a2b320f2cf2b1b2428cc0640c45d463a30b" Sep 11 04:09:05.065703 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:05.065671 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" Sep 11 04:09:05.069081 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:05.069027 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzv6x\" (UniqueName: \"kubernetes.io/projected/5e69d38a-9530-45be-a3b7-8a10d99bd851-kube-api-access-lzv6x\") pod \"payload-pre-processing-e2e-ait-26814939-6b4cdc4d6b-gbx9h\" (UID: \"5e69d38a-9530-45be-a3b7-8a10d99bd851\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-26814939-6b4cdc4d6b-gbx9h" Sep 11 04:09:05.069157 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:05.069141 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e69d38a-9530-45be-a3b7-8a10d99bd851-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-26814939-6b4cdc4d6b-gbx9h\" (UID: \"5e69d38a-9530-45be-a3b7-8a10d99bd851\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-26814939-6b4cdc4d6b-gbx9h" Sep 11 04:09:05.170151 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:05.170109 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c8ff345-db48-4038-9ffe-e443ccd672b4-istiod-ca-cert\") pod \"9c8ff345-db48-4038-9ffe-e443ccd672b4\" (UID: \"9c8ff345-db48-4038-9ffe-e443ccd672b4\") " Sep 11 04:09:05.170656 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:05.170166 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c8ff345-db48-4038-9ffe-e443ccd672b4-istio-data\") pod \"9c8ff345-db48-4038-9ffe-e443ccd672b4\" (UID: \"9c8ff345-db48-4038-9ffe-e443ccd672b4\") " Sep 11 04:09:05.170656 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:05.170202 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c8ff345-db48-4038-9ffe-e443ccd672b4-istio-envoy\") pod \"9c8ff345-db48-4038-9ffe-e443ccd672b4\" (UID: \"9c8ff345-db48-4038-9ffe-e443ccd672b4\") " Sep 11 04:09:05.170656 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:05.170243 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c8ff345-db48-4038-9ffe-e443ccd672b4-istio-podinfo\") pod \"9c8ff345-db48-4038-9ffe-e443ccd672b4\" (UID: \"9c8ff345-db48-4038-9ffe-e443ccd672b4\") " Sep 11 04:09:05.170656 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:05.170301 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c8ff345-db48-4038-9ffe-e443ccd672b4-istio-token\") pod \"9c8ff345-db48-4038-9ffe-e443ccd672b4\" (UID: \"9c8ff345-db48-4038-9ffe-e443ccd672b4\") " Sep 11 04:09:05.170656 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:05.170335 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7ss8\" (UniqueName: \"kubernetes.io/projected/9c8ff345-db48-4038-9ffe-e443ccd672b4-kube-api-access-j7ss8\") pod \"9c8ff345-db48-4038-9ffe-e443ccd672b4\" (UID: \"9c8ff345-db48-4038-9ffe-e443ccd672b4\") " Sep 11 04:09:05.170656 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:05.170359 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8ff345-db48-4038-9ffe-e443ccd672b4-credential-socket\") pod \"9c8ff345-db48-4038-9ffe-e443ccd672b4\" (UID: \"9c8ff345-db48-4038-9ffe-e443ccd672b4\") " Sep 11 04:09:05.170656 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:05.170428 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c8ff345-db48-4038-9ffe-e443ccd672b4-workload-certs\") pod \"9c8ff345-db48-4038-9ffe-e443ccd672b4\" (UID: \"9c8ff345-db48-4038-9ffe-e443ccd672b4\") " Sep 11 04:09:05.170656 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:05.170455 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8ff345-db48-4038-9ffe-e443ccd672b4-workload-socket\") pod \"9c8ff345-db48-4038-9ffe-e443ccd672b4\" (UID: \"9c8ff345-db48-4038-9ffe-e443ccd672b4\") " Sep 11 04:09:05.170656 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:05.170495 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c8ff345-db48-4038-9ffe-e443ccd672b4-istio-data" (OuterVolumeSpecName: "istio-data") pod "9c8ff345-db48-4038-9ffe-e443ccd672b4" (UID: "9c8ff345-db48-4038-9ffe-e443ccd672b4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:09:05.170656 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:05.170528 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c8ff345-db48-4038-9ffe-e443ccd672b4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9c8ff345-db48-4038-9ffe-e443ccd672b4" (UID: "9c8ff345-db48-4038-9ffe-e443ccd672b4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 04:09:05.170656 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:05.170658 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzv6x\" (UniqueName: \"kubernetes.io/projected/5e69d38a-9530-45be-a3b7-8a10d99bd851-kube-api-access-lzv6x\") pod \"payload-pre-processing-e2e-ait-26814939-6b4cdc4d6b-gbx9h\" (UID: \"5e69d38a-9530-45be-a3b7-8a10d99bd851\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-26814939-6b4cdc4d6b-gbx9h" Sep 11 04:09:05.171308 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:05.170630 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c8ff345-db48-4038-9ffe-e443ccd672b4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9c8ff345-db48-4038-9ffe-e443ccd672b4" (UID: "9c8ff345-db48-4038-9ffe-e443ccd672b4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:09:05.171308 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:05.170748 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c8ff345-db48-4038-9ffe-e443ccd672b4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9c8ff345-db48-4038-9ffe-e443ccd672b4" (UID: "9c8ff345-db48-4038-9ffe-e443ccd672b4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:09:05.171308 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:05.170674 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c8ff345-db48-4038-9ffe-e443ccd672b4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9c8ff345-db48-4038-9ffe-e443ccd672b4" (UID: "9c8ff345-db48-4038-9ffe-e443ccd672b4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:09:05.171308 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:05.170825 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e69d38a-9530-45be-a3b7-8a10d99bd851-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-26814939-6b4cdc4d6b-gbx9h\" (UID: \"5e69d38a-9530-45be-a3b7-8a10d99bd851\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-26814939-6b4cdc4d6b-gbx9h" Sep 11 04:09:05.171308 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:05.170928 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8ff345-db48-4038-9ffe-e443ccd672b4-credential-socket\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:09:05.171308 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:05.170944 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c8ff345-db48-4038-9ffe-e443ccd672b4-workload-certs\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:09:05.171308 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:05.170958 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c8ff345-db48-4038-9ffe-e443ccd672b4-workload-socket\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:09:05.171308 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:05.170973 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c8ff345-db48-4038-9ffe-e443ccd672b4-istiod-ca-cert\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:09:05.171308 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:05.170986 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c8ff345-db48-4038-9ffe-e443ccd672b4-istio-data\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:09:05.171865 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:05.171576 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e69d38a-9530-45be-a3b7-8a10d99bd851-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-26814939-6b4cdc4d6b-gbx9h\" (UID: \"5e69d38a-9530-45be-a3b7-8a10d99bd851\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-26814939-6b4cdc4d6b-gbx9h" Sep 11 04:09:05.172738 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:05.172712 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c8ff345-db48-4038-9ffe-e443ccd672b4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9c8ff345-db48-4038-9ffe-e443ccd672b4" (UID: "9c8ff345-db48-4038-9ffe-e443ccd672b4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:09:05.172969 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:05.172946 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c8ff345-db48-4038-9ffe-e443ccd672b4-kube-api-access-j7ss8" (OuterVolumeSpecName: "kube-api-access-j7ss8") pod "9c8ff345-db48-4038-9ffe-e443ccd672b4" (UID: "9c8ff345-db48-4038-9ffe-e443ccd672b4"). InnerVolumeSpecName "kube-api-access-j7ss8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:09:05.173039 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:05.172993 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c8ff345-db48-4038-9ffe-e443ccd672b4-istio-token" (OuterVolumeSpecName: "istio-token") pod "9c8ff345-db48-4038-9ffe-e443ccd672b4" (UID: "9c8ff345-db48-4038-9ffe-e443ccd672b4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:09:05.173133 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:05.173034 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c8ff345-db48-4038-9ffe-e443ccd672b4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9c8ff345-db48-4038-9ffe-e443ccd672b4" (UID: "9c8ff345-db48-4038-9ffe-e443ccd672b4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 04:09:05.182074 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:05.182021 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzv6x\" (UniqueName: \"kubernetes.io/projected/5e69d38a-9530-45be-a3b7-8a10d99bd851-kube-api-access-lzv6x\") pod \"payload-pre-processing-e2e-ait-26814939-6b4cdc4d6b-gbx9h\" (UID: \"5e69d38a-9530-45be-a3b7-8a10d99bd851\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-26814939-6b4cdc4d6b-gbx9h" Sep 11 04:09:05.261157 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:05.261075 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-26814939-6b4cdc4d6b-gbx9h" Sep 11 04:09:05.271391 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:05.271361 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c8ff345-db48-4038-9ffe-e443ccd672b4-istio-podinfo\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:09:05.271391 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:05.271388 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c8ff345-db48-4038-9ffe-e443ccd672b4-istio-token\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:09:05.271546 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:05.271400 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7ss8\" (UniqueName: \"kubernetes.io/projected/9c8ff345-db48-4038-9ffe-e443ccd672b4-kube-api-access-j7ss8\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:09:05.271546 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:05.271409 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c8ff345-db48-4038-9ffe-e443ccd672b4-istio-envoy\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:09:05.562247 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:05.562220 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-26814939-6b4cdc4d6b-gbx9h"] Sep 11 04:09:05.563561 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:09:05.563537 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e69d38a_9530_45be_a3b7_8a10d99bd851.slice/crio-358428d768b8958c332a77e44c6806fb8fd3d05379308daf7e5d5eae4712f5a3 WatchSource:0}: Error finding container 358428d768b8958c332a77e44c6806fb8fd3d05379308daf7e5d5eae4712f5a3: Status 404 returned error can't find the container with id 358428d768b8958c332a77e44c6806fb8fd3d05379308daf7e5d5eae4712f5a3 Sep 11 04:09:05.989019 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:05.988978 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" Sep 11 04:09:05.989232 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:05.989018 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls" event={"ID":"9c8ff345-db48-4038-9ffe-e443ccd672b4","Type":"ContainerDied","Data":"1fb92453f5a0eae23fbb250706c1bbde653b39e003e31700de0873799a9311c0"} Sep 11 04:09:05.989232 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:05.989116 2568 scope.go:117] "RemoveContainer" containerID="23787a237a9f981f582bfee34d22edff8183a9874ef29014cd3f01e5af253f36" Sep 11 04:09:05.991182 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:05.991156 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-26814939-6b4cdc4d6b-gbx9h" event={"ID":"5e69d38a-9530-45be-a3b7-8a10d99bd851","Type":"ContainerStarted","Data":"3ef37dd8880e3b3e48852574f4fa82ef20b2ca52c174139b6abae8021f31b1b9"} Sep 11 04:09:05.991315 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:05.991208 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-26814939-6b4cdc4d6b-gbx9h" event={"ID":"5e69d38a-9530-45be-a3b7-8a10d99bd851","Type":"ContainerStarted","Data":"358428d768b8958c332a77e44c6806fb8fd3d05379308daf7e5d5eae4712f5a3"} Sep 11 04:09:05.991431 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:05.991409 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-26814939-6b4cdc4d6b-gbx9h" Sep 11 04:09:06.015460 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:06.015439 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls"] Sep 11 04:09:06.021241 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:06.021220 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-52e7067f-gw-openshift-default-bffc4bdb4-gx6ls"] Sep 11 04:09:06.039866 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:06.039609 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-26814939-6b4cdc4d6b-gbx9h" podStartSLOduration=2.0395941459999998 podStartE2EDuration="2.039594146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:09:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:09:06.039242839 +0000 UTC m=+960.792874234" watchObservedRunningTime="2026-09-11 04:09:06.039594146 +0000 UTC m=+960.793225549" Sep 11 04:09:07.768915 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:07.768883 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c8ff345-db48-4038-9ffe-e443ccd672b4" path="/var/lib/kubelet/pods/9c8ff345-db48-4038-9ffe-e443ccd672b4/volumes" Sep 11 04:09:08.424870 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:08.424839 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:09:14.323473 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:14.323432 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:09:16.999886 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:16.999855 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-26814939-6b4cdc4d6b-gbx9h" Sep 11 04:09:25.208527 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:25.208494 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:09:28.803111 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:28.803019 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26814939-dl5xp"] Sep 11 04:09:28.803580 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:28.803560 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c8ff345-db48-4038-9ffe-e443ccd672b4" containerName="istio-proxy" Sep 11 04:09:28.803662 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:28.803582 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c8ff345-db48-4038-9ffe-e443ccd672b4" containerName="istio-proxy" Sep 11 04:09:28.803662 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:28.803611 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c8ff345-db48-4038-9ffe-e443ccd672b4" containerName="istio-proxy" Sep 11 04:09:28.803662 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:28.803620 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c8ff345-db48-4038-9ffe-e443ccd672b4" containerName="istio-proxy" Sep 11 04:09:28.803811 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:28.803722 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c8ff345-db48-4038-9ffe-e443ccd672b4" containerName="istio-proxy" Sep 11 04:09:28.803811 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:28.803736 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c8ff345-db48-4038-9ffe-e443ccd672b4" containerName="istio-proxy" Sep 11 04:09:28.808247 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:28.808226 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26814939-dl5xp" Sep 11 04:09:28.810976 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:28.810957 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 04:09:28.811235 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:28.811220 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6brsr\"" Sep 11 04:09:28.816428 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:28.816407 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26814939-dl5xp"] Sep 11 04:09:28.977636 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:28.977597 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d106487b-0bed-43e9-baaa-6bae4ba9a6d7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-26814939-dl5xp\" (UID: \"d106487b-0bed-43e9-baaa-6bae4ba9a6d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26814939-dl5xp" Sep 11 04:09:29.078921 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:29.078817 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d106487b-0bed-43e9-baaa-6bae4ba9a6d7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-26814939-dl5xp\" (UID: \"d106487b-0bed-43e9-baaa-6bae4ba9a6d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26814939-dl5xp" Sep 11 04:09:29.079497 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:29.079474 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d106487b-0bed-43e9-baaa-6bae4ba9a6d7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-26814939-dl5xp\" (UID: \"d106487b-0bed-43e9-baaa-6bae4ba9a6d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26814939-dl5xp" Sep 11 04:09:29.119857 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:29.119824 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26814939-dl5xp" Sep 11 04:09:29.396885 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:29.396858 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26814939-dl5xp"] Sep 11 04:09:29.398860 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:09:29.398830 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd106487b_0bed_43e9_baaa_6bae4ba9a6d7.slice/crio-51a8c5e0bae31b451f1f28b7fb38c88236db7e607f1169a2dfaa676c1f617b0c WatchSource:0}: Error finding container 51a8c5e0bae31b451f1f28b7fb38c88236db7e607f1169a2dfaa676c1f617b0c: Status 404 returned error can't find the container with id 51a8c5e0bae31b451f1f28b7fb38c88236db7e607f1169a2dfaa676c1f617b0c Sep 11 04:09:30.112016 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:30.111984 2568 generic.go:356] "Generic (PLEG): container finished" podID="d106487b-0bed-43e9-baaa-6bae4ba9a6d7" containerID="c5c2beba3104a9de5ce9f872409ed027689757c44a213405feb300e2b524ee1c" exitCode=0 Sep 11 04:09:30.112397 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:30.112071 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26814939-dl5xp" event={"ID":"d106487b-0bed-43e9-baaa-6bae4ba9a6d7","Type":"ContainerDied","Data":"c5c2beba3104a9de5ce9f872409ed027689757c44a213405feb300e2b524ee1c"} Sep 11 04:09:30.112397 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:30.112123 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26814939-dl5xp" event={"ID":"d106487b-0bed-43e9-baaa-6bae4ba9a6d7","Type":"ContainerStarted","Data":"51a8c5e0bae31b451f1f28b7fb38c88236db7e607f1169a2dfaa676c1f617b0c"} Sep 11 04:09:31.248409 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:31.248384 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26814939-dl5xp" Sep 11 04:09:31.398776 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:31.398687 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d106487b-0bed-43e9-baaa-6bae4ba9a6d7-maas-api-service-ca\") pod \"d106487b-0bed-43e9-baaa-6bae4ba9a6d7\" (UID: \"d106487b-0bed-43e9-baaa-6bae4ba9a6d7\") " Sep 11 04:09:31.399078 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:31.399039 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d106487b-0bed-43e9-baaa-6bae4ba9a6d7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d106487b-0bed-43e9-baaa-6bae4ba9a6d7" (UID: "d106487b-0bed-43e9-baaa-6bae4ba9a6d7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 04:09:31.499940 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:31.499902 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d106487b-0bed-43e9-baaa-6bae4ba9a6d7-maas-api-service-ca\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:09:32.125667 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:32.125574 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26814939-dl5xp" event={"ID":"d106487b-0bed-43e9-baaa-6bae4ba9a6d7","Type":"ContainerDied","Data":"51a8c5e0bae31b451f1f28b7fb38c88236db7e607f1169a2dfaa676c1f617b0c"} Sep 11 04:09:32.125667 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:32.125619 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26814939-dl5xp" Sep 11 04:09:32.125667 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:32.125651 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51a8c5e0bae31b451f1f28b7fb38c88236db7e607f1169a2dfaa676c1f617b0c" Sep 11 04:09:32.525186 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:32.525151 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:09:34.668379 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:34.668334 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:09:34.718668 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:34.718629 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-26814939-6b4cdc4d6b-gbx9h"] Sep 11 04:09:34.718993 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:34.718951 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-26814939-6b4cdc4d6b-gbx9h" podUID="5e69d38a-9530-45be-a3b7-8a10d99bd851" containerName="payload-pre-processing" containerID="cri-o://3ef37dd8880e3b3e48852574f4fa82ef20b2ca52c174139b6abae8021f31b1b9" gracePeriod=30 Sep 11 04:09:34.768281 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:34.768249 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:09:34.867887 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:34.867855 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:09:34.887927 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:34.887871 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26814939-dl5xp"] Sep 11 04:09:34.890011 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:34.889986 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-26814939-dl5xp"] Sep 11 04:09:34.942114 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:34.941868 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:09:34.972814 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:34.972793 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-26814939-6b4cdc4d6b-gbx9h" Sep 11 04:09:35.136218 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:35.136188 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e69d38a-9530-45be-a3b7-8a10d99bd851-plugins-config-volume\") pod \"5e69d38a-9530-45be-a3b7-8a10d99bd851\" (UID: \"5e69d38a-9530-45be-a3b7-8a10d99bd851\") " Sep 11 04:09:35.136420 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:35.136279 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzv6x\" (UniqueName: \"kubernetes.io/projected/5e69d38a-9530-45be-a3b7-8a10d99bd851-kube-api-access-lzv6x\") pod \"5e69d38a-9530-45be-a3b7-8a10d99bd851\" (UID: \"5e69d38a-9530-45be-a3b7-8a10d99bd851\") " Sep 11 04:09:35.136631 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:35.136604 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e69d38a-9530-45be-a3b7-8a10d99bd851-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5e69d38a-9530-45be-a3b7-8a10d99bd851" (UID: "5e69d38a-9530-45be-a3b7-8a10d99bd851"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 04:09:35.138875 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:35.138831 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e69d38a-9530-45be-a3b7-8a10d99bd851-kube-api-access-lzv6x" (OuterVolumeSpecName: "kube-api-access-lzv6x") pod "5e69d38a-9530-45be-a3b7-8a10d99bd851" (UID: "5e69d38a-9530-45be-a3b7-8a10d99bd851"). InnerVolumeSpecName "kube-api-access-lzv6x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:09:35.143743 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:35.143716 2568 generic.go:356] "Generic (PLEG): container finished" podID="5e69d38a-9530-45be-a3b7-8a10d99bd851" containerID="3ef37dd8880e3b3e48852574f4fa82ef20b2ca52c174139b6abae8021f31b1b9" exitCode=0 Sep 11 04:09:35.143867 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:35.143764 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-26814939-6b4cdc4d6b-gbx9h" event={"ID":"5e69d38a-9530-45be-a3b7-8a10d99bd851","Type":"ContainerDied","Data":"3ef37dd8880e3b3e48852574f4fa82ef20b2ca52c174139b6abae8021f31b1b9"} Sep 11 04:09:35.143867 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:35.143770 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-26814939-6b4cdc4d6b-gbx9h" Sep 11 04:09:35.143867 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:35.143814 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-26814939-6b4cdc4d6b-gbx9h" event={"ID":"5e69d38a-9530-45be-a3b7-8a10d99bd851","Type":"ContainerDied","Data":"358428d768b8958c332a77e44c6806fb8fd3d05379308daf7e5d5eae4712f5a3"} Sep 11 04:09:35.143867 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:35.143840 2568 scope.go:117] "RemoveContainer" containerID="3ef37dd8880e3b3e48852574f4fa82ef20b2ca52c174139b6abae8021f31b1b9" Sep 11 04:09:35.153426 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:35.153408 2568 scope.go:117] "RemoveContainer" containerID="3ef37dd8880e3b3e48852574f4fa82ef20b2ca52c174139b6abae8021f31b1b9" Sep 11 04:09:35.153710 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:09:35.153691 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ef37dd8880e3b3e48852574f4fa82ef20b2ca52c174139b6abae8021f31b1b9\": container with ID starting with 3ef37dd8880e3b3e48852574f4fa82ef20b2ca52c174139b6abae8021f31b1b9 not found: ID does not exist" containerID="3ef37dd8880e3b3e48852574f4fa82ef20b2ca52c174139b6abae8021f31b1b9" Sep 11 04:09:35.153780 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:35.153721 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ef37dd8880e3b3e48852574f4fa82ef20b2ca52c174139b6abae8021f31b1b9"} err="failed to get container status \"3ef37dd8880e3b3e48852574f4fa82ef20b2ca52c174139b6abae8021f31b1b9\": rpc error: code = NotFound desc = could not find container \"3ef37dd8880e3b3e48852574f4fa82ef20b2ca52c174139b6abae8021f31b1b9\": container with ID starting with 3ef37dd8880e3b3e48852574f4fa82ef20b2ca52c174139b6abae8021f31b1b9 not found: ID does not exist" Sep 11 04:09:35.168171 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:35.168146 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-26814939-6b4cdc4d6b-gbx9h"] Sep 11 04:09:35.171434 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:35.171413 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-26814939-6b4cdc4d6b-gbx9h"] Sep 11 04:09:35.237596 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:35.237510 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e69d38a-9530-45be-a3b7-8a10d99bd851-plugins-config-volume\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:09:35.237596 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:35.237542 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzv6x\" (UniqueName: \"kubernetes.io/projected/5e69d38a-9530-45be-a3b7-8a10d99bd851-kube-api-access-lzv6x\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:09:35.768299 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:35.768268 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e69d38a-9530-45be-a3b7-8a10d99bd851" path="/var/lib/kubelet/pods/5e69d38a-9530-45be-a3b7-8a10d99bd851/volumes" Sep 11 04:09:35.768676 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:35.768618 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d106487b-0bed-43e9-baaa-6bae4ba9a6d7" path="/var/lib/kubelet/pods/d106487b-0bed-43e9-baaa-6bae4ba9a6d7/volumes" Sep 11 04:09:36.150742 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:36.150643 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:09:36.626634 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:36.626591 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:09:36.941978 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:36.941947 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp"] Sep 11 04:09:36.942574 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:36.942556 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d106487b-0bed-43e9-baaa-6bae4ba9a6d7" containerName="revoke-keys" Sep 11 04:09:36.942632 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:36.942578 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d106487b-0bed-43e9-baaa-6bae4ba9a6d7" containerName="revoke-keys" Sep 11 04:09:36.942632 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:36.942621 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e69d38a-9530-45be-a3b7-8a10d99bd851" containerName="payload-pre-processing" Sep 11 04:09:36.942704 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:36.942633 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e69d38a-9530-45be-a3b7-8a10d99bd851" containerName="payload-pre-processing" Sep 11 04:09:36.942736 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:36.942714 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d106487b-0bed-43e9-baaa-6bae4ba9a6d7" containerName="revoke-keys" Sep 11 04:09:36.942736 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:36.942727 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e69d38a-9530-45be-a3b7-8a10d99bd851" containerName="payload-pre-processing" Sep 11 04:09:36.947493 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:36.947472 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" Sep 11 04:09:36.957905 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:36.957883 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-285c6871-openshift-default-dockercfg-mgtzb\"" Sep 11 04:09:36.984791 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:36.984344 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp"] Sep 11 04:09:36.990291 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:36.990263 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:09:37.052413 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:37.052372 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-istio-token\") pod \"e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp\" (UID: \"14c3b9b4-c179-42fe-a4d9-cdd1162678ef\") " pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" Sep 11 04:09:37.052598 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:37.052438 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg6vj\" (UniqueName: \"kubernetes.io/projected/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-kube-api-access-xg6vj\") pod \"e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp\" (UID: \"14c3b9b4-c179-42fe-a4d9-cdd1162678ef\") " pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" Sep 11 04:09:37.052598 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:37.052476 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-workload-certs\") pod \"e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp\" (UID: \"14c3b9b4-c179-42fe-a4d9-cdd1162678ef\") " pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" Sep 11 04:09:37.052598 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:37.052499 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-istio-envoy\") pod \"e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp\" (UID: \"14c3b9b4-c179-42fe-a4d9-cdd1162678ef\") " pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" Sep 11 04:09:37.052598 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:37.052546 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-istio-podinfo\") pod \"e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp\" (UID: \"14c3b9b4-c179-42fe-a4d9-cdd1162678ef\") " pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" Sep 11 04:09:37.052790 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:37.052638 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-istio-data\") pod \"e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp\" (UID: \"14c3b9b4-c179-42fe-a4d9-cdd1162678ef\") " pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" Sep 11 04:09:37.052790 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:37.052663 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-istiod-ca-cert\") pod \"e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp\" (UID: \"14c3b9b4-c179-42fe-a4d9-cdd1162678ef\") " pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" Sep 11 04:09:37.052790 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:37.052691 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-workload-socket\") pod \"e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp\" (UID: \"14c3b9b4-c179-42fe-a4d9-cdd1162678ef\") " pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" Sep 11 04:09:37.052790 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:37.052719 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-credential-socket\") pod \"e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp\" (UID: \"14c3b9b4-c179-42fe-a4d9-cdd1162678ef\") " pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" Sep 11 04:09:37.153200 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:37.153168 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-workload-socket\") pod \"e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp\" (UID: \"14c3b9b4-c179-42fe-a4d9-cdd1162678ef\") " pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" Sep 11 04:09:37.153200 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:37.153204 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-credential-socket\") pod \"e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp\" (UID: \"14c3b9b4-c179-42fe-a4d9-cdd1162678ef\") " pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" Sep 11 04:09:37.153478 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:37.153234 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-istio-token\") pod \"e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp\" (UID: \"14c3b9b4-c179-42fe-a4d9-cdd1162678ef\") " pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" Sep 11 04:09:37.153478 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:37.153277 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg6vj\" (UniqueName: \"kubernetes.io/projected/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-kube-api-access-xg6vj\") pod \"e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp\" (UID: \"14c3b9b4-c179-42fe-a4d9-cdd1162678ef\") " pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" Sep 11 04:09:37.153478 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:37.153304 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-workload-certs\") pod \"e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp\" (UID: \"14c3b9b4-c179-42fe-a4d9-cdd1162678ef\") " pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" Sep 11 04:09:37.153478 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:37.153326 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-istio-envoy\") pod \"e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp\" (UID: \"14c3b9b4-c179-42fe-a4d9-cdd1162678ef\") " pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" Sep 11 04:09:37.153478 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:37.153348 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-istio-podinfo\") pod \"e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp\" (UID: \"14c3b9b4-c179-42fe-a4d9-cdd1162678ef\") " pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" Sep 11 04:09:37.153478 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:37.153412 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-istio-data\") pod \"e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp\" (UID: \"14c3b9b4-c179-42fe-a4d9-cdd1162678ef\") " pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" Sep 11 04:09:37.153478 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:37.153433 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-istiod-ca-cert\") pod \"e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp\" (UID: \"14c3b9b4-c179-42fe-a4d9-cdd1162678ef\") " pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" Sep 11 04:09:37.153837 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:37.153617 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-workload-socket\") pod \"e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp\" (UID: \"14c3b9b4-c179-42fe-a4d9-cdd1162678ef\") " pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" Sep 11 04:09:37.153837 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:37.153767 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-credential-socket\") pod \"e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp\" (UID: \"14c3b9b4-c179-42fe-a4d9-cdd1162678ef\") " pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" Sep 11 04:09:37.153947 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:37.153899 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-istio-data\") pod \"e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp\" (UID: \"14c3b9b4-c179-42fe-a4d9-cdd1162678ef\") " pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" Sep 11 04:09:37.154001 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:37.153943 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-workload-certs\") pod \"e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp\" (UID: \"14c3b9b4-c179-42fe-a4d9-cdd1162678ef\") " pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" Sep 11 04:09:37.154277 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:37.154250 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-istiod-ca-cert\") pod \"e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp\" (UID: \"14c3b9b4-c179-42fe-a4d9-cdd1162678ef\") " pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" Sep 11 04:09:37.155875 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:37.155850 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-istio-envoy\") pod \"e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp\" (UID: \"14c3b9b4-c179-42fe-a4d9-cdd1162678ef\") " pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" Sep 11 04:09:37.156511 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:37.156488 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-istio-podinfo\") pod \"e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp\" (UID: \"14c3b9b4-c179-42fe-a4d9-cdd1162678ef\") " pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" Sep 11 04:09:37.163124 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:37.163094 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-istio-token\") pod \"e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp\" (UID: \"14c3b9b4-c179-42fe-a4d9-cdd1162678ef\") " pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" Sep 11 04:09:37.163483 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:37.163465 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg6vj\" (UniqueName: \"kubernetes.io/projected/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-kube-api-access-xg6vj\") pod \"e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp\" (UID: \"14c3b9b4-c179-42fe-a4d9-cdd1162678ef\") " pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" Sep 11 04:09:37.257552 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:37.257458 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" Sep 11 04:09:37.289894 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:37.289849 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:09:37.548754 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:37.548651 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp"] Sep 11 04:09:37.551780 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:09:37.551743 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14c3b9b4_c179_42fe_a4d9_cdd1162678ef.slice/crio-4bc4f0e9377cba457c6ccd78bdbb72a691fc4d46a1d653f70274916286aa0559 WatchSource:0}: Error finding container 4bc4f0e9377cba457c6ccd78bdbb72a691fc4d46a1d653f70274916286aa0559: Status 404 returned error can't find the container with id 4bc4f0e9377cba457c6ccd78bdbb72a691fc4d46a1d653f70274916286aa0559 Sep 11 04:09:37.554360 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:37.554312 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 04:09:37.554451 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:37.554406 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 04:09:37.554493 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:37.554449 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 04:09:38.166566 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:38.166527 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" event={"ID":"14c3b9b4-c179-42fe-a4d9-cdd1162678ef","Type":"ContainerStarted","Data":"0abf6720136fb6cfe6143d473901469d90554d962386e000a23b76e84311983a"} Sep 11 04:09:38.166986 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:38.166579 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" event={"ID":"14c3b9b4-c179-42fe-a4d9-cdd1162678ef","Type":"ContainerStarted","Data":"4bc4f0e9377cba457c6ccd78bdbb72a691fc4d46a1d653f70274916286aa0559"} Sep 11 04:09:38.210232 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:38.210149 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" podStartSLOduration=2.210132411 podStartE2EDuration="2.210132411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:09:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:09:38.209342917 +0000 UTC m=+992.962974319" watchObservedRunningTime="2026-09-11 04:09:38.210132411 +0000 UTC m=+992.963763841" Sep 11 04:09:38.258472 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:38.258434 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" Sep 11 04:09:38.263377 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:38.263352 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" Sep 11 04:09:39.173544 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:39.173469 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" Sep 11 04:09:39.174803 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:39.174780 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" Sep 11 04:09:39.663077 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:39.663025 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:09:39.784949 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:39.784915 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:09:41.179835 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:41.179798 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:09:41.324490 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:41.324455 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:09:41.486792 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:41.486702 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:09:42.020060 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:42.020008 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:09:42.142758 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:42.142723 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:09:42.213129 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:42.213092 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-285c6871-fb984d6cd-7j9lz"] Sep 11 04:09:42.216637 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:42.216619 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-285c6871-fb984d6cd-7j9lz" Sep 11 04:09:42.220349 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:42.220328 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-285c6871\"" Sep 11 04:09:42.235887 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:42.235860 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-285c6871-fb984d6cd-7j9lz"] Sep 11 04:09:42.332602 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:42.332505 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-285c6871-99659647c-s8p22"] Sep 11 04:09:42.336592 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:42.336565 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-285c6871-99659647c-s8p22" Sep 11 04:09:42.340784 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:42.340752 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-285c6871-dockercfg-k9f6s\"" Sep 11 04:09:42.368554 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:42.368504 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-285c6871-99659647c-s8p22"] Sep 11 04:09:42.399453 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:42.399373 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3c4b220-3bed-49b6-ad2a-fe455fd1a28a-plugins-config-volume\") pod \"payload-processing-e2e-derive-285c6871-99659647c-s8p22\" (UID: \"b3c4b220-3bed-49b6-ad2a-fe455fd1a28a\") " pod="openshift-ingress/payload-processing-e2e-derive-285c6871-99659647c-s8p22" Sep 11 04:09:42.399453 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:42.399428 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4fkw\" (UniqueName: \"kubernetes.io/projected/cc26c083-4e3b-414b-8530-7319206a5cb8-kube-api-access-g4fkw\") pod \"payload-pre-processing-e2e-derive-285c6871-fb984d6cd-7j9lz\" (UID: \"cc26c083-4e3b-414b-8530-7319206a5cb8\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-285c6871-fb984d6cd-7j9lz" Sep 11 04:09:42.399721 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:42.399487 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkj52\" (UniqueName: \"kubernetes.io/projected/b3c4b220-3bed-49b6-ad2a-fe455fd1a28a-kube-api-access-rkj52\") pod \"payload-processing-e2e-derive-285c6871-99659647c-s8p22\" (UID: \"b3c4b220-3bed-49b6-ad2a-fe455fd1a28a\") " pod="openshift-ingress/payload-processing-e2e-derive-285c6871-99659647c-s8p22" Sep 11 04:09:42.399721 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:42.399539 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc26c083-4e3b-414b-8530-7319206a5cb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-285c6871-fb984d6cd-7j9lz\" (UID: \"cc26c083-4e3b-414b-8530-7319206a5cb8\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-285c6871-fb984d6cd-7j9lz" Sep 11 04:09:42.500212 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:42.500174 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkj52\" (UniqueName: \"kubernetes.io/projected/b3c4b220-3bed-49b6-ad2a-fe455fd1a28a-kube-api-access-rkj52\") pod \"payload-processing-e2e-derive-285c6871-99659647c-s8p22\" (UID: \"b3c4b220-3bed-49b6-ad2a-fe455fd1a28a\") " pod="openshift-ingress/payload-processing-e2e-derive-285c6871-99659647c-s8p22" Sep 11 04:09:42.500401 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:42.500240 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc26c083-4e3b-414b-8530-7319206a5cb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-285c6871-fb984d6cd-7j9lz\" (UID: \"cc26c083-4e3b-414b-8530-7319206a5cb8\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-285c6871-fb984d6cd-7j9lz" Sep 11 04:09:42.500401 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:42.500338 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3c4b220-3bed-49b6-ad2a-fe455fd1a28a-plugins-config-volume\") pod \"payload-processing-e2e-derive-285c6871-99659647c-s8p22\" (UID: \"b3c4b220-3bed-49b6-ad2a-fe455fd1a28a\") " pod="openshift-ingress/payload-processing-e2e-derive-285c6871-99659647c-s8p22" Sep 11 04:09:42.500401 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:42.500373 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4fkw\" (UniqueName: \"kubernetes.io/projected/cc26c083-4e3b-414b-8530-7319206a5cb8-kube-api-access-g4fkw\") pod \"payload-pre-processing-e2e-derive-285c6871-fb984d6cd-7j9lz\" (UID: \"cc26c083-4e3b-414b-8530-7319206a5cb8\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-285c6871-fb984d6cd-7j9lz" Sep 11 04:09:42.501064 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:42.501028 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc26c083-4e3b-414b-8530-7319206a5cb8-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-285c6871-fb984d6cd-7j9lz\" (UID: \"cc26c083-4e3b-414b-8530-7319206a5cb8\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-285c6871-fb984d6cd-7j9lz" Sep 11 04:09:42.501275 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:42.501257 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3c4b220-3bed-49b6-ad2a-fe455fd1a28a-plugins-config-volume\") pod \"payload-processing-e2e-derive-285c6871-99659647c-s8p22\" (UID: \"b3c4b220-3bed-49b6-ad2a-fe455fd1a28a\") " pod="openshift-ingress/payload-processing-e2e-derive-285c6871-99659647c-s8p22" Sep 11 04:09:42.511633 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:42.511608 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4fkw\" (UniqueName: \"kubernetes.io/projected/cc26c083-4e3b-414b-8530-7319206a5cb8-kube-api-access-g4fkw\") pod \"payload-pre-processing-e2e-derive-285c6871-fb984d6cd-7j9lz\" (UID: \"cc26c083-4e3b-414b-8530-7319206a5cb8\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-285c6871-fb984d6cd-7j9lz" Sep 11 04:09:42.511739 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:42.511659 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkj52\" (UniqueName: \"kubernetes.io/projected/b3c4b220-3bed-49b6-ad2a-fe455fd1a28a-kube-api-access-rkj52\") pod \"payload-processing-e2e-derive-285c6871-99659647c-s8p22\" (UID: \"b3c4b220-3bed-49b6-ad2a-fe455fd1a28a\") " pod="openshift-ingress/payload-processing-e2e-derive-285c6871-99659647c-s8p22" Sep 11 04:09:42.531526 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:42.531494 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-285c6871-fb984d6cd-7j9lz" Sep 11 04:09:42.651707 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:42.651664 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-285c6871-99659647c-s8p22" Sep 11 04:09:42.822974 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:42.822948 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-285c6871-fb984d6cd-7j9lz"] Sep 11 04:09:42.825829 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:09:42.825798 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc26c083_4e3b_414b_8530_7319206a5cb8.slice/crio-4334b6058323ba1daabe33bf0de00f46a1c8588d5bdb3395cc3231b4a143be1c WatchSource:0}: Error finding container 4334b6058323ba1daabe33bf0de00f46a1c8588d5bdb3395cc3231b4a143be1c: Status 404 returned error can't find the container with id 4334b6058323ba1daabe33bf0de00f46a1c8588d5bdb3395cc3231b4a143be1c Sep 11 04:09:42.949775 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:42.949739 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-285c6871-99659647c-s8p22"] Sep 11 04:09:42.953492 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:09:42.953460 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3c4b220_3bed_49b6_ad2a_fe455fd1a28a.slice/crio-636e0c86e93cac77c792c2f6d5a921cf5fda0fcf075a489f5f08986e162859da WatchSource:0}: Error finding container 636e0c86e93cac77c792c2f6d5a921cf5fda0fcf075a489f5f08986e162859da: Status 404 returned error can't find the container with id 636e0c86e93cac77c792c2f6d5a921cf5fda0fcf075a489f5f08986e162859da Sep 11 04:09:43.195092 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:43.195030 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-285c6871-99659647c-s8p22" event={"ID":"b3c4b220-3bed-49b6-ad2a-fe455fd1a28a","Type":"ContainerStarted","Data":"138e49c4a99607c2e213ba7ec8b8e67156680ea6c19baebf3e4286c8b59d1bce"} Sep 11 04:09:43.195298 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:43.195117 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-285c6871-99659647c-s8p22" Sep 11 04:09:43.195298 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:43.195142 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-285c6871-99659647c-s8p22" event={"ID":"b3c4b220-3bed-49b6-ad2a-fe455fd1a28a","Type":"ContainerStarted","Data":"636e0c86e93cac77c792c2f6d5a921cf5fda0fcf075a489f5f08986e162859da"} Sep 11 04:09:43.196771 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:43.196736 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-285c6871-fb984d6cd-7j9lz" event={"ID":"cc26c083-4e3b-414b-8530-7319206a5cb8","Type":"ContainerStarted","Data":"65402d681bc2605fbfd81c2dd46f3579651c739c1b4ddfc2f9e0962f7b78f9ce"} Sep 11 04:09:43.196876 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:43.196788 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-285c6871-fb984d6cd-7j9lz" event={"ID":"cc26c083-4e3b-414b-8530-7319206a5cb8","Type":"ContainerStarted","Data":"4334b6058323ba1daabe33bf0de00f46a1c8588d5bdb3395cc3231b4a143be1c"} Sep 11 04:09:43.196876 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:43.196817 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-285c6871-fb984d6cd-7j9lz" Sep 11 04:09:43.224409 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:43.224362 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-285c6871-99659647c-s8p22" podStartSLOduration=1.224346245 podStartE2EDuration="1.224346245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:09:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:09:43.222895988 +0000 UTC m=+997.976527390" watchObservedRunningTime="2026-09-11 04:09:43.224346245 +0000 UTC m=+997.977977674" Sep 11 04:09:43.249811 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:43.249758 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-285c6871-fb984d6cd-7j9lz" podStartSLOduration=1.249741871 podStartE2EDuration="1.249741871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:09:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:09:43.247285349 +0000 UTC m=+998.000916752" watchObservedRunningTime="2026-09-11 04:09:43.249741871 +0000 UTC m=+998.003373310" Sep 11 04:09:48.218004 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:48.217912 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:09:54.205137 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:54.205106 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-285c6871-99659647c-s8p22" Sep 11 04:09:54.205638 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:54.205187 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-285c6871-fb984d6cd-7j9lz" Sep 11 04:09:56.473326 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:56.472004 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:09:58.628506 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:58.628467 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:09:59.313130 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:59.313101 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:09:59.537535 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:09:59.537497 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:10:00.300897 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:00.300865 2568 generic.go:356] "Generic (PLEG): container finished" podID="14c3b9b4-c179-42fe-a4d9-cdd1162678ef" containerID="0abf6720136fb6cfe6143d473901469d90554d962386e000a23b76e84311983a" exitCode=0 Sep 11 04:10:00.301347 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:00.300948 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" event={"ID":"14c3b9b4-c179-42fe-a4d9-cdd1162678ef","Type":"ContainerDied","Data":"0abf6720136fb6cfe6143d473901469d90554d962386e000a23b76e84311983a"} Sep 11 04:10:00.301399 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:00.301356 2568 scope.go:117] "RemoveContainer" containerID="0abf6720136fb6cfe6143d473901469d90554d962386e000a23b76e84311983a" Sep 11 04:10:00.302437 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:00.302404 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 04:10:00.302540 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:00.302473 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 04:10:00.302540 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:00.302504 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 04:10:01.307657 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:01.307619 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" event={"ID":"14c3b9b4-c179-42fe-a4d9-cdd1162678ef","Type":"ContainerStarted","Data":"d843a007961f5ca9545581f7a699949996015a2b22d4b91455f0e886eef07d94"} Sep 11 04:10:02.258920 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:02.258876 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" Sep 11 04:10:02.259256 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:02.259012 2568 patch_prober.go:28] interesting pod/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Sep 11 04:10:02.259256 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:02.259076 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" podUID="14c3b9b4-c179-42fe-a4d9-cdd1162678ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Sep 11 04:10:02.695787 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:02.695756 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:10:03.258205 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:03.258150 2568 patch_prober.go:28] interesting pod/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Sep 11 04:10:03.258423 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:03.258224 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" podUID="14c3b9b4-c179-42fe-a4d9-cdd1162678ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Sep 11 04:10:04.258451 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:04.258411 2568 patch_prober.go:28] interesting pod/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Sep 11 04:10:04.259103 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:04.259064 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" podUID="14c3b9b4-c179-42fe-a4d9-cdd1162678ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Sep 11 04:10:05.258846 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:05.258800 2568 patch_prober.go:28] interesting pod/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Sep 11 04:10:05.259416 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:05.258878 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" podUID="14c3b9b4-c179-42fe-a4d9-cdd1162678ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Sep 11 04:10:06.258871 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:06.258825 2568 patch_prober.go:28] interesting pod/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Sep 11 04:10:06.259265 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:06.258903 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" podUID="14c3b9b4-c179-42fe-a4d9-cdd1162678ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Sep 11 04:10:07.262608 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:07.262573 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" Sep 11 04:10:07.263009 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:07.262860 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" Sep 11 04:10:07.263452 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:07.263434 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" Sep 11 04:10:08.920604 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:08.920543 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:10:09.020932 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:09.020902 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:10:09.592454 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:09.592419 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:10:10.209272 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:10.209238 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:10:10.240626 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:10.240596 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-eda9e4-695967bd58-t2g26"] Sep 11 04:10:10.241091 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:10.241039 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-eda9e4-695967bd58-t2g26" podUID="20581566-083b-489a-9d5f-62ccbb4bb771" containerName="payload-pre-processing" containerID="cri-o://6b0e6e744cd432e7ced08e3550f2636f5e800463ef0a0c84e4d4ca77cb313f22" gracePeriod=30 Sep 11 04:10:10.290689 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:10.290657 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:10:10.352842 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:10.352805 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:10:10.422268 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:10.422234 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:10:10.491320 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:10.491298 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-eda9e4-695967bd58-t2g26" Sep 11 04:10:10.566223 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:10.566191 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqmtm\" (UniqueName: \"kubernetes.io/projected/20581566-083b-489a-9d5f-62ccbb4bb771-kube-api-access-lqmtm\") pod \"20581566-083b-489a-9d5f-62ccbb4bb771\" (UID: \"20581566-083b-489a-9d5f-62ccbb4bb771\") " Sep 11 04:10:10.566412 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:10.566240 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20581566-083b-489a-9d5f-62ccbb4bb771-plugins-config-volume\") pod \"20581566-083b-489a-9d5f-62ccbb4bb771\" (UID: \"20581566-083b-489a-9d5f-62ccbb4bb771\") " Sep 11 04:10:10.566842 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:10.566810 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20581566-083b-489a-9d5f-62ccbb4bb771-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "20581566-083b-489a-9d5f-62ccbb4bb771" (UID: "20581566-083b-489a-9d5f-62ccbb4bb771"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 04:10:10.568805 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:10.568545 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20581566-083b-489a-9d5f-62ccbb4bb771-kube-api-access-lqmtm" (OuterVolumeSpecName: "kube-api-access-lqmtm") pod "20581566-083b-489a-9d5f-62ccbb4bb771" (UID: "20581566-083b-489a-9d5f-62ccbb4bb771"). InnerVolumeSpecName "kube-api-access-lqmtm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:10:10.667570 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:10.667532 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqmtm\" (UniqueName: \"kubernetes.io/projected/20581566-083b-489a-9d5f-62ccbb4bb771-kube-api-access-lqmtm\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:10:10.667570 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:10.667565 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20581566-083b-489a-9d5f-62ccbb4bb771-plugins-config-volume\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:10:10.772717 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:10.772639 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:10:11.116787 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:11.116696 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:10:11.363682 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:11.363643 2568 generic.go:356] "Generic (PLEG): container finished" podID="20581566-083b-489a-9d5f-62ccbb4bb771" containerID="6b0e6e744cd432e7ced08e3550f2636f5e800463ef0a0c84e4d4ca77cb313f22" exitCode=0 Sep 11 04:10:11.364126 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:11.363691 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-eda9e4-695967bd58-t2g26" Sep 11 04:10:11.364126 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:11.363734 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-eda9e4-695967bd58-t2g26" event={"ID":"20581566-083b-489a-9d5f-62ccbb4bb771","Type":"ContainerDied","Data":"6b0e6e744cd432e7ced08e3550f2636f5e800463ef0a0c84e4d4ca77cb313f22"} Sep 11 04:10:11.364126 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:11.363786 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-eda9e4-695967bd58-t2g26" event={"ID":"20581566-083b-489a-9d5f-62ccbb4bb771","Type":"ContainerDied","Data":"f32e4c38d886225a2230e4f7fc258e9fbc34ed90097180e88846d248e6d387e4"} Sep 11 04:10:11.364126 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:11.363810 2568 scope.go:117] "RemoveContainer" containerID="6b0e6e744cd432e7ced08e3550f2636f5e800463ef0a0c84e4d4ca77cb313f22" Sep 11 04:10:11.372591 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:11.372574 2568 scope.go:117] "RemoveContainer" containerID="6b0e6e744cd432e7ced08e3550f2636f5e800463ef0a0c84e4d4ca77cb313f22" Sep 11 04:10:11.372862 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:10:11.372844 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b0e6e744cd432e7ced08e3550f2636f5e800463ef0a0c84e4d4ca77cb313f22\": container with ID starting with 6b0e6e744cd432e7ced08e3550f2636f5e800463ef0a0c84e4d4ca77cb313f22 not found: ID does not exist" containerID="6b0e6e744cd432e7ced08e3550f2636f5e800463ef0a0c84e4d4ca77cb313f22" Sep 11 04:10:11.372910 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:11.372870 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b0e6e744cd432e7ced08e3550f2636f5e800463ef0a0c84e4d4ca77cb313f22"} err="failed to get container status \"6b0e6e744cd432e7ced08e3550f2636f5e800463ef0a0c84e4d4ca77cb313f22\": rpc error: code = NotFound desc = could not find container \"6b0e6e744cd432e7ced08e3550f2636f5e800463ef0a0c84e4d4ca77cb313f22\": container with ID starting with 6b0e6e744cd432e7ced08e3550f2636f5e800463ef0a0c84e4d4ca77cb313f22 not found: ID does not exist" Sep 11 04:10:11.384234 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:11.384210 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-eda9e4-695967bd58-t2g26"] Sep 11 04:10:11.390169 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:11.390150 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-eda9e4-695967bd58-t2g26"] Sep 11 04:10:11.768501 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:11.768467 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20581566-083b-489a-9d5f-62ccbb4bb771" path="/var/lib/kubelet/pods/20581566-083b-489a-9d5f-62ccbb4bb771/volumes" Sep 11 04:10:11.854970 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:11.854937 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:10:11.891278 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:11.891248 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-285c6871-99659647c-s8p22"] Sep 11 04:10:11.891487 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:11.891462 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-285c6871-99659647c-s8p22" podUID="b3c4b220-3bed-49b6-ad2a-fe455fd1a28a" containerName="payload-processing" containerID="cri-o://138e49c4a99607c2e213ba7ec8b8e67156680ea6c19baebf3e4286c8b59d1bce" gracePeriod=30 Sep 11 04:10:11.894867 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:11.894837 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-285c6871-fb984d6cd-7j9lz"] Sep 11 04:10:11.895164 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:11.895134 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-285c6871-fb984d6cd-7j9lz" podUID="cc26c083-4e3b-414b-8530-7319206a5cb8" containerName="payload-pre-processing" containerID="cri-o://65402d681bc2605fbfd81c2dd46f3579651c739c1b4ddfc2f9e0962f7b78f9ce" gracePeriod=30 Sep 11 04:10:11.935848 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:11.935812 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:10:12.002772 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:12.002742 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:10:12.070377 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:12.070327 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:10:12.145133 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:12.145105 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-285c6871-99659647c-s8p22" Sep 11 04:10:12.177332 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:12.177306 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-285c6871-fb984d6cd-7j9lz" Sep 11 04:10:12.207513 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:12.203213 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp"] Sep 11 04:10:12.207513 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:12.203538 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" podUID="14c3b9b4-c179-42fe-a4d9-cdd1162678ef" containerName="istio-proxy" containerID="cri-o://d843a007961f5ca9545581f7a699949996015a2b22d4b91455f0e886eef07d94" gracePeriod=30 Sep 11 04:10:12.214253 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:12.214161 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:10:12.282016 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:12.281982 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4fkw\" (UniqueName: \"kubernetes.io/projected/cc26c083-4e3b-414b-8530-7319206a5cb8-kube-api-access-g4fkw\") pod \"cc26c083-4e3b-414b-8530-7319206a5cb8\" (UID: \"cc26c083-4e3b-414b-8530-7319206a5cb8\") " Sep 11 04:10:12.282169 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:12.282083 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkj52\" (UniqueName: \"kubernetes.io/projected/b3c4b220-3bed-49b6-ad2a-fe455fd1a28a-kube-api-access-rkj52\") pod \"b3c4b220-3bed-49b6-ad2a-fe455fd1a28a\" (UID: \"b3c4b220-3bed-49b6-ad2a-fe455fd1a28a\") " Sep 11 04:10:12.282169 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:12.282115 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3c4b220-3bed-49b6-ad2a-fe455fd1a28a-plugins-config-volume\") pod \"b3c4b220-3bed-49b6-ad2a-fe455fd1a28a\" (UID: \"b3c4b220-3bed-49b6-ad2a-fe455fd1a28a\") " Sep 11 04:10:12.282249 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:12.282174 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc26c083-4e3b-414b-8530-7319206a5cb8-plugins-config-volume\") pod \"cc26c083-4e3b-414b-8530-7319206a5cb8\" (UID: \"cc26c083-4e3b-414b-8530-7319206a5cb8\") " Sep 11 04:10:12.282556 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:12.282527 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3c4b220-3bed-49b6-ad2a-fe455fd1a28a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b3c4b220-3bed-49b6-ad2a-fe455fd1a28a" (UID: "b3c4b220-3bed-49b6-ad2a-fe455fd1a28a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 04:10:12.282662 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:12.282570 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc26c083-4e3b-414b-8530-7319206a5cb8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cc26c083-4e3b-414b-8530-7319206a5cb8" (UID: "cc26c083-4e3b-414b-8530-7319206a5cb8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 04:10:12.284070 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:12.284018 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc26c083-4e3b-414b-8530-7319206a5cb8-kube-api-access-g4fkw" (OuterVolumeSpecName: "kube-api-access-g4fkw") pod "cc26c083-4e3b-414b-8530-7319206a5cb8" (UID: "cc26c083-4e3b-414b-8530-7319206a5cb8"). InnerVolumeSpecName "kube-api-access-g4fkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:10:12.284177 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:12.284124 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3c4b220-3bed-49b6-ad2a-fe455fd1a28a-kube-api-access-rkj52" (OuterVolumeSpecName: "kube-api-access-rkj52") pod "b3c4b220-3bed-49b6-ad2a-fe455fd1a28a" (UID: "b3c4b220-3bed-49b6-ad2a-fe455fd1a28a"). InnerVolumeSpecName "kube-api-access-rkj52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:10:12.371607 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:12.371515 2568 generic.go:356] "Generic (PLEG): container finished" podID="cc26c083-4e3b-414b-8530-7319206a5cb8" containerID="65402d681bc2605fbfd81c2dd46f3579651c739c1b4ddfc2f9e0962f7b78f9ce" exitCode=0 Sep 11 04:10:12.371607 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:12.371579 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-285c6871-fb984d6cd-7j9lz" Sep 11 04:10:12.372139 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:12.371610 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-285c6871-fb984d6cd-7j9lz" event={"ID":"cc26c083-4e3b-414b-8530-7319206a5cb8","Type":"ContainerDied","Data":"65402d681bc2605fbfd81c2dd46f3579651c739c1b4ddfc2f9e0962f7b78f9ce"} Sep 11 04:10:12.372139 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:12.371675 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-285c6871-fb984d6cd-7j9lz" event={"ID":"cc26c083-4e3b-414b-8530-7319206a5cb8","Type":"ContainerDied","Data":"4334b6058323ba1daabe33bf0de00f46a1c8588d5bdb3395cc3231b4a143be1c"} Sep 11 04:10:12.372139 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:12.371707 2568 scope.go:117] "RemoveContainer" containerID="65402d681bc2605fbfd81c2dd46f3579651c739c1b4ddfc2f9e0962f7b78f9ce" Sep 11 04:10:12.374206 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:12.374180 2568 generic.go:356] "Generic (PLEG): container finished" podID="b3c4b220-3bed-49b6-ad2a-fe455fd1a28a" containerID="138e49c4a99607c2e213ba7ec8b8e67156680ea6c19baebf3e4286c8b59d1bce" exitCode=0 Sep 11 04:10:12.374325 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:12.374260 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-285c6871-99659647c-s8p22" Sep 11 04:10:12.374325 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:12.374262 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-285c6871-99659647c-s8p22" event={"ID":"b3c4b220-3bed-49b6-ad2a-fe455fd1a28a","Type":"ContainerDied","Data":"138e49c4a99607c2e213ba7ec8b8e67156680ea6c19baebf3e4286c8b59d1bce"} Sep 11 04:10:12.374325 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:12.374316 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-285c6871-99659647c-s8p22" event={"ID":"b3c4b220-3bed-49b6-ad2a-fe455fd1a28a","Type":"ContainerDied","Data":"636e0c86e93cac77c792c2f6d5a921cf5fda0fcf075a489f5f08986e162859da"} Sep 11 04:10:12.381238 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:12.381185 2568 scope.go:117] "RemoveContainer" containerID="65402d681bc2605fbfd81c2dd46f3579651c739c1b4ddfc2f9e0962f7b78f9ce" Sep 11 04:10:12.381831 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:10:12.381806 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65402d681bc2605fbfd81c2dd46f3579651c739c1b4ddfc2f9e0962f7b78f9ce\": container with ID starting with 65402d681bc2605fbfd81c2dd46f3579651c739c1b4ddfc2f9e0962f7b78f9ce not found: ID does not exist" containerID="65402d681bc2605fbfd81c2dd46f3579651c739c1b4ddfc2f9e0962f7b78f9ce" Sep 11 04:10:12.382425 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:12.382030 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65402d681bc2605fbfd81c2dd46f3579651c739c1b4ddfc2f9e0962f7b78f9ce"} err="failed to get container status \"65402d681bc2605fbfd81c2dd46f3579651c739c1b4ddfc2f9e0962f7b78f9ce\": rpc error: code = NotFound desc = could not find container \"65402d681bc2605fbfd81c2dd46f3579651c739c1b4ddfc2f9e0962f7b78f9ce\": container with ID starting with 65402d681bc2605fbfd81c2dd46f3579651c739c1b4ddfc2f9e0962f7b78f9ce not found: ID does not exist" Sep 11 04:10:12.382554 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:12.382531 2568 scope.go:117] "RemoveContainer" containerID="138e49c4a99607c2e213ba7ec8b8e67156680ea6c19baebf3e4286c8b59d1bce" Sep 11 04:10:12.382969 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:12.382948 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4fkw\" (UniqueName: \"kubernetes.io/projected/cc26c083-4e3b-414b-8530-7319206a5cb8-kube-api-access-g4fkw\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:10:12.383085 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:12.382976 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkj52\" (UniqueName: \"kubernetes.io/projected/b3c4b220-3bed-49b6-ad2a-fe455fd1a28a-kube-api-access-rkj52\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:10:12.383085 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:12.382990 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3c4b220-3bed-49b6-ad2a-fe455fd1a28a-plugins-config-volume\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:10:12.383085 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:12.383020 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc26c083-4e3b-414b-8530-7319206a5cb8-plugins-config-volume\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:10:12.391559 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:12.391541 2568 scope.go:117] "RemoveContainer" containerID="138e49c4a99607c2e213ba7ec8b8e67156680ea6c19baebf3e4286c8b59d1bce" Sep 11 04:10:12.391799 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:10:12.391771 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"138e49c4a99607c2e213ba7ec8b8e67156680ea6c19baebf3e4286c8b59d1bce\": container with ID starting with 138e49c4a99607c2e213ba7ec8b8e67156680ea6c19baebf3e4286c8b59d1bce not found: ID does not exist" containerID="138e49c4a99607c2e213ba7ec8b8e67156680ea6c19baebf3e4286c8b59d1bce" Sep 11 04:10:12.391859 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:12.391799 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"138e49c4a99607c2e213ba7ec8b8e67156680ea6c19baebf3e4286c8b59d1bce"} err="failed to get container status \"138e49c4a99607c2e213ba7ec8b8e67156680ea6c19baebf3e4286c8b59d1bce\": rpc error: code = NotFound desc = could not find container \"138e49c4a99607c2e213ba7ec8b8e67156680ea6c19baebf3e4286c8b59d1bce\": container with ID starting with 138e49c4a99607c2e213ba7ec8b8e67156680ea6c19baebf3e4286c8b59d1bce not found: ID does not exist" Sep 11 04:10:12.395522 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:12.395502 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-285c6871-fb984d6cd-7j9lz"] Sep 11 04:10:12.399495 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:12.399475 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-285c6871-fb984d6cd-7j9lz"] Sep 11 04:10:12.409199 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:12.409180 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-285c6871-99659647c-s8p22"] Sep 11 04:10:12.419034 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:12.419011 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-285c6871-99659647c-s8p22"] Sep 11 04:10:12.925516 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:12.925485 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:10:13.033241 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:13.033205 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:10:13.175533 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:13.175440 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:10:13.767864 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:13.767829 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3c4b220-3bed-49b6-ad2a-fe455fd1a28a" path="/var/lib/kubelet/pods/b3c4b220-3bed-49b6-ad2a-fe455fd1a28a/volumes" Sep 11 04:10:13.768256 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:13.768209 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc26c083-4e3b-414b-8530-7319206a5cb8" path="/var/lib/kubelet/pods/cc26c083-4e3b-414b-8530-7319206a5cb8/volumes" Sep 11 04:10:14.567026 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:14.566990 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:10:16.698214 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:16.698182 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:10:16.803972 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:16.803940 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:10:16.986800 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:16.986721 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5"] Sep 11 04:10:16.987003 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:16.986979 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" podUID="82fa5c18-c332-45ab-8bea-ac64f4b2caf3" containerName="istio-proxy" containerID="cri-o://d341be0b05ce8b48489cb0bd1e8fef8a12319c7df9084d7723118544eabae99d" gracePeriod=30 Sep 11 04:10:16.997338 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:16.997312 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:10:17.405937 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:17.405898 2568 generic.go:356] "Generic (PLEG): container finished" podID="14c3b9b4-c179-42fe-a4d9-cdd1162678ef" containerID="d843a007961f5ca9545581f7a699949996015a2b22d4b91455f0e886eef07d94" exitCode=0 Sep 11 04:10:17.406111 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:17.406000 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" event={"ID":"14c3b9b4-c179-42fe-a4d9-cdd1162678ef","Type":"ContainerDied","Data":"d843a007961f5ca9545581f7a699949996015a2b22d4b91455f0e886eef07d94"} Sep 11 04:10:17.406170 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:17.406107 2568 scope.go:117] "RemoveContainer" containerID="0abf6720136fb6cfe6143d473901469d90554d962386e000a23b76e84311983a" Sep 11 04:10:17.464453 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:17.464424 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:10:17.466312 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:17.466295 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" Sep 11 04:10:17.630903 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:17.630811 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-istio-envoy\") pod \"14c3b9b4-c179-42fe-a4d9-cdd1162678ef\" (UID: \"14c3b9b4-c179-42fe-a4d9-cdd1162678ef\") " Sep 11 04:10:17.630903 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:17.630867 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-istiod-ca-cert\") pod \"14c3b9b4-c179-42fe-a4d9-cdd1162678ef\" (UID: \"14c3b9b4-c179-42fe-a4d9-cdd1162678ef\") " Sep 11 04:10:17.630903 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:17.630906 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-istio-token\") pod \"14c3b9b4-c179-42fe-a4d9-cdd1162678ef\" (UID: \"14c3b9b4-c179-42fe-a4d9-cdd1162678ef\") " Sep 11 04:10:17.631207 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:17.630935 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-workload-certs\") pod \"14c3b9b4-c179-42fe-a4d9-cdd1162678ef\" (UID: \"14c3b9b4-c179-42fe-a4d9-cdd1162678ef\") " Sep 11 04:10:17.631207 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:17.630959 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xg6vj\" (UniqueName: \"kubernetes.io/projected/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-kube-api-access-xg6vj\") pod \"14c3b9b4-c179-42fe-a4d9-cdd1162678ef\" (UID: \"14c3b9b4-c179-42fe-a4d9-cdd1162678ef\") " Sep 11 04:10:17.631207 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:17.631075 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-workload-socket\") pod \"14c3b9b4-c179-42fe-a4d9-cdd1162678ef\" (UID: \"14c3b9b4-c179-42fe-a4d9-cdd1162678ef\") " Sep 11 04:10:17.631207 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:17.631200 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-istio-podinfo\") pod \"14c3b9b4-c179-42fe-a4d9-cdd1162678ef\" (UID: \"14c3b9b4-c179-42fe-a4d9-cdd1162678ef\") " Sep 11 04:10:17.631409 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:17.631228 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-credential-socket\") pod \"14c3b9b4-c179-42fe-a4d9-cdd1162678ef\" (UID: \"14c3b9b4-c179-42fe-a4d9-cdd1162678ef\") " Sep 11 04:10:17.631409 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:17.631258 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-istio-data\") pod \"14c3b9b4-c179-42fe-a4d9-cdd1162678ef\" (UID: \"14c3b9b4-c179-42fe-a4d9-cdd1162678ef\") " Sep 11 04:10:17.631409 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:17.631263 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "14c3b9b4-c179-42fe-a4d9-cdd1162678ef" (UID: "14c3b9b4-c179-42fe-a4d9-cdd1162678ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:10:17.631409 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:17.631384 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "14c3b9b4-c179-42fe-a4d9-cdd1162678ef" (UID: "14c3b9b4-c179-42fe-a4d9-cdd1162678ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 04:10:17.631607 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:17.631418 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "14c3b9b4-c179-42fe-a4d9-cdd1162678ef" (UID: "14c3b9b4-c179-42fe-a4d9-cdd1162678ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:10:17.631658 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:17.631633 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-istiod-ca-cert\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:10:17.631658 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:17.631654 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-workload-certs\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:10:17.631749 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:17.631658 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-istio-data" (OuterVolumeSpecName: "istio-data") pod "14c3b9b4-c179-42fe-a4d9-cdd1162678ef" (UID: "14c3b9b4-c179-42fe-a4d9-cdd1162678ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:10:17.631749 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:17.631668 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-workload-socket\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:10:17.631749 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:17.631674 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "14c3b9b4-c179-42fe-a4d9-cdd1162678ef" (UID: "14c3b9b4-c179-42fe-a4d9-cdd1162678ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:10:17.633170 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:17.633146 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "14c3b9b4-c179-42fe-a4d9-cdd1162678ef" (UID: "14c3b9b4-c179-42fe-a4d9-cdd1162678ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:10:17.633290 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:17.633185 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-istio-token" (OuterVolumeSpecName: "istio-token") pod "14c3b9b4-c179-42fe-a4d9-cdd1162678ef" (UID: "14c3b9b4-c179-42fe-a4d9-cdd1162678ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:10:17.633601 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:17.633583 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "14c3b9b4-c179-42fe-a4d9-cdd1162678ef" (UID: "14c3b9b4-c179-42fe-a4d9-cdd1162678ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 04:10:17.633671 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:17.633587 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-kube-api-access-xg6vj" (OuterVolumeSpecName: "kube-api-access-xg6vj") pod "14c3b9b4-c179-42fe-a4d9-cdd1162678ef" (UID: "14c3b9b4-c179-42fe-a4d9-cdd1162678ef"). InnerVolumeSpecName "kube-api-access-xg6vj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:10:17.733130 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:17.733090 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-istio-token\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:10:17.733130 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:17.733126 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg6vj\" (UniqueName: \"kubernetes.io/projected/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-kube-api-access-xg6vj\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:10:17.733665 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:17.733142 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-istio-podinfo\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:10:17.733665 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:17.733156 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-credential-socket\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:10:17.733665 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:17.733169 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-istio-data\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:10:17.733665 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:17.733182 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14c3b9b4-c179-42fe-a4d9-cdd1162678ef-istio-envoy\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:10:17.777708 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:17.777671 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:10:18.184730 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:18.184690 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:10:18.411855 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:18.411815 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" event={"ID":"14c3b9b4-c179-42fe-a4d9-cdd1162678ef","Type":"ContainerDied","Data":"4bc4f0e9377cba457c6ccd78bdbb72a691fc4d46a1d653f70274916286aa0559"} Sep 11 04:10:18.412032 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:18.411878 2568 scope.go:117] "RemoveContainer" containerID="d843a007961f5ca9545581f7a699949996015a2b22d4b91455f0e886eef07d94" Sep 11 04:10:18.412032 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:18.411837 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp" Sep 11 04:10:18.433579 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:18.433551 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp"] Sep 11 04:10:18.439435 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:18.439371 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-285c6871-openshift-default-78bdc7c7cb-74rhp"] Sep 11 04:10:19.773222 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:19.773188 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14c3b9b4-c179-42fe-a4d9-cdd1162678ef" path="/var/lib/kubelet/pods/14c3b9b4-c179-42fe-a4d9-cdd1162678ef/volumes" Sep 11 04:10:20.803407 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:20.803373 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:10:20.923001 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:20.922949 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:10:21.013625 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:21.013583 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:10:21.106522 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:21.106433 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:10:21.569832 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:21.569800 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:10:22.117421 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.117389 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:10:22.206835 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.206801 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:10:22.251199 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.251134 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" Sep 11 04:10:22.375581 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.375536 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-credential-socket\") pod \"82fa5c18-c332-45ab-8bea-ac64f4b2caf3\" (UID: \"82fa5c18-c332-45ab-8bea-ac64f4b2caf3\") " Sep 11 04:10:22.375752 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.375598 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-workload-certs\") pod \"82fa5c18-c332-45ab-8bea-ac64f4b2caf3\" (UID: \"82fa5c18-c332-45ab-8bea-ac64f4b2caf3\") " Sep 11 04:10:22.375752 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.375644 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-istio-data\") pod \"82fa5c18-c332-45ab-8bea-ac64f4b2caf3\" (UID: \"82fa5c18-c332-45ab-8bea-ac64f4b2caf3\") " Sep 11 04:10:22.375752 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.375666 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-istio-token\") pod \"82fa5c18-c332-45ab-8bea-ac64f4b2caf3\" (UID: \"82fa5c18-c332-45ab-8bea-ac64f4b2caf3\") " Sep 11 04:10:22.375752 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.375692 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-istio-podinfo\") pod \"82fa5c18-c332-45ab-8bea-ac64f4b2caf3\" (UID: \"82fa5c18-c332-45ab-8bea-ac64f4b2caf3\") " Sep 11 04:10:22.375752 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.375737 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-workload-socket\") pod \"82fa5c18-c332-45ab-8bea-ac64f4b2caf3\" (UID: \"82fa5c18-c332-45ab-8bea-ac64f4b2caf3\") " Sep 11 04:10:22.376022 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.375776 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-istio-envoy\") pod \"82fa5c18-c332-45ab-8bea-ac64f4b2caf3\" (UID: \"82fa5c18-c332-45ab-8bea-ac64f4b2caf3\") " Sep 11 04:10:22.376022 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.375803 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jb5p\" (UniqueName: \"kubernetes.io/projected/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-kube-api-access-7jb5p\") pod \"82fa5c18-c332-45ab-8bea-ac64f4b2caf3\" (UID: \"82fa5c18-c332-45ab-8bea-ac64f4b2caf3\") " Sep 11 04:10:22.376022 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.375837 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-istiod-ca-cert\") pod \"82fa5c18-c332-45ab-8bea-ac64f4b2caf3\" (UID: \"82fa5c18-c332-45ab-8bea-ac64f4b2caf3\") " Sep 11 04:10:22.376022 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.375838 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "82fa5c18-c332-45ab-8bea-ac64f4b2caf3" (UID: "82fa5c18-c332-45ab-8bea-ac64f4b2caf3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:10:22.376022 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.375933 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "82fa5c18-c332-45ab-8bea-ac64f4b2caf3" (UID: "82fa5c18-c332-45ab-8bea-ac64f4b2caf3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:10:22.376022 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.375965 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-istio-data" (OuterVolumeSpecName: "istio-data") pod "82fa5c18-c332-45ab-8bea-ac64f4b2caf3" (UID: "82fa5c18-c332-45ab-8bea-ac64f4b2caf3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:10:22.376364 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.376143 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-workload-certs\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:10:22.376364 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.376164 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-istio-data\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:10:22.376364 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.376178 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-credential-socket\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:10:22.376364 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.376184 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "82fa5c18-c332-45ab-8bea-ac64f4b2caf3" (UID: "82fa5c18-c332-45ab-8bea-ac64f4b2caf3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:10:22.376562 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.376432 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "82fa5c18-c332-45ab-8bea-ac64f4b2caf3" (UID: "82fa5c18-c332-45ab-8bea-ac64f4b2caf3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 04:10:22.378017 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.377992 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-kube-api-access-7jb5p" (OuterVolumeSpecName: "kube-api-access-7jb5p") pod "82fa5c18-c332-45ab-8bea-ac64f4b2caf3" (UID: "82fa5c18-c332-45ab-8bea-ac64f4b2caf3"). InnerVolumeSpecName "kube-api-access-7jb5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:10:22.378137 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.378071 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "82fa5c18-c332-45ab-8bea-ac64f4b2caf3" (UID: "82fa5c18-c332-45ab-8bea-ac64f4b2caf3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 04:10:22.378137 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.378076 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "82fa5c18-c332-45ab-8bea-ac64f4b2caf3" (UID: "82fa5c18-c332-45ab-8bea-ac64f4b2caf3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:10:22.378304 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.378289 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-istio-token" (OuterVolumeSpecName: "istio-token") pod "82fa5c18-c332-45ab-8bea-ac64f4b2caf3" (UID: "82fa5c18-c332-45ab-8bea-ac64f4b2caf3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:10:22.436978 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.436940 2568 generic.go:356] "Generic (PLEG): container finished" podID="82fa5c18-c332-45ab-8bea-ac64f4b2caf3" containerID="d341be0b05ce8b48489cb0bd1e8fef8a12319c7df9084d7723118544eabae99d" exitCode=0 Sep 11 04:10:22.437166 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.436991 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" Sep 11 04:10:22.437166 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.437031 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" event={"ID":"82fa5c18-c332-45ab-8bea-ac64f4b2caf3","Type":"ContainerDied","Data":"d341be0b05ce8b48489cb0bd1e8fef8a12319c7df9084d7723118544eabae99d"} Sep 11 04:10:22.437166 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.437103 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5" event={"ID":"82fa5c18-c332-45ab-8bea-ac64f4b2caf3","Type":"ContainerDied","Data":"3f1cd958a54b0dbaa5b8434a1e3d1d33178579bdf8680cda19e81133a77f401c"} Sep 11 04:10:22.437166 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.437131 2568 scope.go:117] "RemoveContainer" containerID="d341be0b05ce8b48489cb0bd1e8fef8a12319c7df9084d7723118544eabae99d" Sep 11 04:10:22.447504 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.447484 2568 scope.go:117] "RemoveContainer" containerID="2b901d7d94ac2c752e7edec8edd0e1290da11947f049812a3c5179ad1233910c" Sep 11 04:10:22.455291 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.455271 2568 scope.go:117] "RemoveContainer" containerID="d341be0b05ce8b48489cb0bd1e8fef8a12319c7df9084d7723118544eabae99d" Sep 11 04:10:22.455560 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:10:22.455540 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d341be0b05ce8b48489cb0bd1e8fef8a12319c7df9084d7723118544eabae99d\": container with ID starting with d341be0b05ce8b48489cb0bd1e8fef8a12319c7df9084d7723118544eabae99d not found: ID does not exist" containerID="d341be0b05ce8b48489cb0bd1e8fef8a12319c7df9084d7723118544eabae99d" Sep 11 04:10:22.455615 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.455569 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d341be0b05ce8b48489cb0bd1e8fef8a12319c7df9084d7723118544eabae99d"} err="failed to get container status \"d341be0b05ce8b48489cb0bd1e8fef8a12319c7df9084d7723118544eabae99d\": rpc error: code = NotFound desc = could not find container \"d341be0b05ce8b48489cb0bd1e8fef8a12319c7df9084d7723118544eabae99d\": container with ID starting with d341be0b05ce8b48489cb0bd1e8fef8a12319c7df9084d7723118544eabae99d not found: ID does not exist" Sep 11 04:10:22.455615 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.455586 2568 scope.go:117] "RemoveContainer" containerID="2b901d7d94ac2c752e7edec8edd0e1290da11947f049812a3c5179ad1233910c" Sep 11 04:10:22.455819 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:10:22.455802 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b901d7d94ac2c752e7edec8edd0e1290da11947f049812a3c5179ad1233910c\": container with ID starting with 2b901d7d94ac2c752e7edec8edd0e1290da11947f049812a3c5179ad1233910c not found: ID does not exist" containerID="2b901d7d94ac2c752e7edec8edd0e1290da11947f049812a3c5179ad1233910c" Sep 11 04:10:22.455872 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.455824 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b901d7d94ac2c752e7edec8edd0e1290da11947f049812a3c5179ad1233910c"} err="failed to get container status \"2b901d7d94ac2c752e7edec8edd0e1290da11947f049812a3c5179ad1233910c\": rpc error: code = NotFound desc = could not find container \"2b901d7d94ac2c752e7edec8edd0e1290da11947f049812a3c5179ad1233910c\": container with ID starting with 2b901d7d94ac2c752e7edec8edd0e1290da11947f049812a3c5179ad1233910c not found: ID does not exist" Sep 11 04:10:22.464929 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.464907 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5"] Sep 11 04:10:22.469161 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.469139 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-eda9e4-openshift-default-686577ddb-sz7v5"] Sep 11 04:10:22.471378 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.471354 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:10:22.476542 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.476526 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-istiod-ca-cert\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:10:22.476606 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.476544 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-istio-token\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:10:22.476606 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.476553 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-istio-podinfo\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:10:22.476606 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.476561 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-workload-socket\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:10:22.476606 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.476570 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-istio-envoy\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:10:22.476606 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.476580 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jb5p\" (UniqueName: \"kubernetes.io/projected/82fa5c18-c332-45ab-8bea-ac64f4b2caf3-kube-api-access-7jb5p\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:10:22.543871 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.543785 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:10:22.644897 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.644851 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:10:22.753170 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.753129 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:10:22.776560 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.776532 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-57c530-7765788dfd-hwwtn"] Sep 11 04:10:22.776967 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.776949 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc26c083-4e3b-414b-8530-7319206a5cb8" containerName="payload-pre-processing" Sep 11 04:10:22.777024 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.776971 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc26c083-4e3b-414b-8530-7319206a5cb8" containerName="payload-pre-processing" Sep 11 04:10:22.777024 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.776989 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20581566-083b-489a-9d5f-62ccbb4bb771" containerName="payload-pre-processing" Sep 11 04:10:22.777024 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.776998 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="20581566-083b-489a-9d5f-62ccbb4bb771" containerName="payload-pre-processing" Sep 11 04:10:22.777024 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.777014 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3c4b220-3bed-49b6-ad2a-fe455fd1a28a" containerName="payload-processing" Sep 11 04:10:22.777024 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.777023 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3c4b220-3bed-49b6-ad2a-fe455fd1a28a" containerName="payload-processing" Sep 11 04:10:22.777222 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.777034 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14c3b9b4-c179-42fe-a4d9-cdd1162678ef" containerName="istio-proxy" Sep 11 04:10:22.777222 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.777042 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="14c3b9b4-c179-42fe-a4d9-cdd1162678ef" containerName="istio-proxy" Sep 11 04:10:22.777222 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.777084 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14c3b9b4-c179-42fe-a4d9-cdd1162678ef" containerName="istio-proxy" Sep 11 04:10:22.777222 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.777092 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="14c3b9b4-c179-42fe-a4d9-cdd1162678ef" containerName="istio-proxy" Sep 11 04:10:22.777222 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.777102 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82fa5c18-c332-45ab-8bea-ac64f4b2caf3" containerName="istio-proxy" Sep 11 04:10:22.777222 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.777110 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="82fa5c18-c332-45ab-8bea-ac64f4b2caf3" containerName="istio-proxy" Sep 11 04:10:22.777222 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.777121 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82fa5c18-c332-45ab-8bea-ac64f4b2caf3" containerName="istio-proxy" Sep 11 04:10:22.777222 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.777128 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="82fa5c18-c332-45ab-8bea-ac64f4b2caf3" containerName="istio-proxy" Sep 11 04:10:22.777222 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.777214 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc26c083-4e3b-414b-8530-7319206a5cb8" containerName="payload-pre-processing" Sep 11 04:10:22.777222 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.777225 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="20581566-083b-489a-9d5f-62ccbb4bb771" containerName="payload-pre-processing" Sep 11 04:10:22.777546 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.777236 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3c4b220-3bed-49b6-ad2a-fe455fd1a28a" containerName="payload-processing" Sep 11 04:10:22.777546 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.777246 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="14c3b9b4-c179-42fe-a4d9-cdd1162678ef" containerName="istio-proxy" Sep 11 04:10:22.777546 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.777255 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="82fa5c18-c332-45ab-8bea-ac64f4b2caf3" containerName="istio-proxy" Sep 11 04:10:22.777546 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.777264 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="82fa5c18-c332-45ab-8bea-ac64f4b2caf3" containerName="istio-proxy" Sep 11 04:10:22.781772 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.781754 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-57c530-7765788dfd-hwwtn" Sep 11 04:10:22.784655 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.784621 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-57c530-dockercfg-2q7s2\"" Sep 11 04:10:22.784772 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.784663 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-57c530\"" Sep 11 04:10:22.805428 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.805363 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-57c530-7765788dfd-hwwtn"] Sep 11 04:10:22.880588 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.880554 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kfsp\" (UniqueName: \"kubernetes.io/projected/89f51db2-f9e8-43a1-becb-4c2046df5dbc-kube-api-access-6kfsp\") pod \"payload-pre-processing-e2e-explicit-57c530-7765788dfd-hwwtn\" (UID: \"89f51db2-f9e8-43a1-becb-4c2046df5dbc\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-57c530-7765788dfd-hwwtn" Sep 11 04:10:22.880764 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.880617 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89f51db2-f9e8-43a1-becb-4c2046df5dbc-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-57c530-7765788dfd-hwwtn\" (UID: \"89f51db2-f9e8-43a1-becb-4c2046df5dbc\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-57c530-7765788dfd-hwwtn" Sep 11 04:10:22.981315 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.981280 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89f51db2-f9e8-43a1-becb-4c2046df5dbc-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-57c530-7765788dfd-hwwtn\" (UID: \"89f51db2-f9e8-43a1-becb-4c2046df5dbc\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-57c530-7765788dfd-hwwtn" Sep 11 04:10:22.981494 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.981361 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kfsp\" (UniqueName: \"kubernetes.io/projected/89f51db2-f9e8-43a1-becb-4c2046df5dbc-kube-api-access-6kfsp\") pod \"payload-pre-processing-e2e-explicit-57c530-7765788dfd-hwwtn\" (UID: \"89f51db2-f9e8-43a1-becb-4c2046df5dbc\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-57c530-7765788dfd-hwwtn" Sep 11 04:10:22.982031 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.982012 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89f51db2-f9e8-43a1-becb-4c2046df5dbc-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-57c530-7765788dfd-hwwtn\" (UID: \"89f51db2-f9e8-43a1-becb-4c2046df5dbc\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-57c530-7765788dfd-hwwtn" Sep 11 04:10:22.991966 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:22.991943 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kfsp\" (UniqueName: \"kubernetes.io/projected/89f51db2-f9e8-43a1-becb-4c2046df5dbc-kube-api-access-6kfsp\") pod \"payload-pre-processing-e2e-explicit-57c530-7765788dfd-hwwtn\" (UID: \"89f51db2-f9e8-43a1-becb-4c2046df5dbc\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-57c530-7765788dfd-hwwtn" Sep 11 04:10:23.092746 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:23.092675 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-57c530-7765788dfd-hwwtn" Sep 11 04:10:23.375440 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:23.375414 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-57c530-7765788dfd-hwwtn"] Sep 11 04:10:23.377632 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:10:23.377605 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89f51db2_f9e8_43a1_becb_4c2046df5dbc.slice/crio-bf57c904caed71d3acdbf209622245e456af7eabc8ac8b1eac5fc2e5a764e05e WatchSource:0}: Error finding container bf57c904caed71d3acdbf209622245e456af7eabc8ac8b1eac5fc2e5a764e05e: Status 404 returned error can't find the container with id bf57c904caed71d3acdbf209622245e456af7eabc8ac8b1eac5fc2e5a764e05e Sep 11 04:10:23.446563 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:23.446531 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-57c530-7765788dfd-hwwtn" event={"ID":"89f51db2-f9e8-43a1-becb-4c2046df5dbc","Type":"ContainerStarted","Data":"55128a01fa53428ecf7c629cc4261df0181960dc47a9490f9c6c2ff3a5f4dace"} Sep 11 04:10:23.446687 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:23.446589 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-57c530-7765788dfd-hwwtn" event={"ID":"89f51db2-f9e8-43a1-becb-4c2046df5dbc","Type":"ContainerStarted","Data":"bf57c904caed71d3acdbf209622245e456af7eabc8ac8b1eac5fc2e5a764e05e"} Sep 11 04:10:23.446687 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:23.446673 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-57c530-7765788dfd-hwwtn" Sep 11 04:10:23.468458 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:23.468410 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-57c530-7765788dfd-hwwtn" podStartSLOduration=1.468394416 podStartE2EDuration="1.468394416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:10:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:10:23.467369969 +0000 UTC m=+1038.221001382" watchObservedRunningTime="2026-09-11 04:10:23.468394416 +0000 UTC m=+1038.222025817" Sep 11 04:10:23.768275 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:23.768245 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82fa5c18-c332-45ab-8bea-ac64f4b2caf3" path="/var/lib/kubelet/pods/82fa5c18-c332-45ab-8bea-ac64f4b2caf3/volumes" Sep 11 04:10:27.137979 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:27.137944 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:10:34.454250 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:34.454214 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-57c530-7765788dfd-hwwtn" Sep 11 04:10:45.957828 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:45.957791 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:10:47.556817 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:47.556752 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:10:53.534552 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:53.534518 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:10:53.642549 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:53.642505 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:10:53.734750 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:53.734716 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:10:58.554764 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:10:58.554687 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:09.625621 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:09.625579 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:14.549506 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:14.549461 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:15.451376 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:15.451342 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:15.554984 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:15.554939 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:15.651963 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:15.651913 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:22.197582 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:22.197545 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:22.344563 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:22.344530 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-57c530-jmsz2"] Sep 11 04:11:22.344963 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:22.344948 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="14c3b9b4-c179-42fe-a4d9-cdd1162678ef" containerName="istio-proxy" Sep 11 04:11:22.349652 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:22.349630 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-57c530-jmsz2" Sep 11 04:11:22.352298 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:22.352276 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 04:11:22.352823 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:22.352808 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6brsr\"" Sep 11 04:11:22.392649 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:22.392617 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85ba0342-5887-4cb0-85f8-0331e7a565e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-57c530-jmsz2\" (UID: \"85ba0342-5887-4cb0-85f8-0331e7a565e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-57c530-jmsz2" Sep 11 04:11:22.411209 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:22.411175 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-57c530-jmsz2"] Sep 11 04:11:22.493462 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:22.493390 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85ba0342-5887-4cb0-85f8-0331e7a565e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-57c530-jmsz2\" (UID: \"85ba0342-5887-4cb0-85f8-0331e7a565e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-57c530-jmsz2" Sep 11 04:11:22.493988 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:22.493968 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85ba0342-5887-4cb0-85f8-0331e7a565e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-57c530-jmsz2\" (UID: \"85ba0342-5887-4cb0-85f8-0331e7a565e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-57c530-jmsz2" Sep 11 04:11:22.659702 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:22.659664 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-57c530-jmsz2" Sep 11 04:11:22.933648 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:22.933626 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-57c530-jmsz2"] Sep 11 04:11:22.935891 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:11:22.935864 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85ba0342_5887_4cb0_85f8_0331e7a565e0.slice/crio-31ee70d77f5cb95ea7f956149bd575b453a7f6e7886da1e2a8f1269567b4c609 WatchSource:0}: Error finding container 31ee70d77f5cb95ea7f956149bd575b453a7f6e7886da1e2a8f1269567b4c609: Status 404 returned error can't find the container with id 31ee70d77f5cb95ea7f956149bd575b453a7f6e7886da1e2a8f1269567b4c609 Sep 11 04:11:23.754329 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:23.754288 2568 generic.go:356] "Generic (PLEG): container finished" podID="85ba0342-5887-4cb0-85f8-0331e7a565e0" containerID="8ac3f9d22b0b8a9902cf38c31ef3db5f16010a47716ddea00547bc84a1e04076" exitCode=0 Sep 11 04:11:23.754742 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:23.754376 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-57c530-jmsz2" event={"ID":"85ba0342-5887-4cb0-85f8-0331e7a565e0","Type":"ContainerDied","Data":"8ac3f9d22b0b8a9902cf38c31ef3db5f16010a47716ddea00547bc84a1e04076"} Sep 11 04:11:23.754742 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:23.754490 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-57c530-jmsz2" event={"ID":"85ba0342-5887-4cb0-85f8-0331e7a565e0","Type":"ContainerStarted","Data":"31ee70d77f5cb95ea7f956149bd575b453a7f6e7886da1e2a8f1269567b4c609"} Sep 11 04:11:24.894022 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:24.893985 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-57c530-jmsz2" Sep 11 04:11:24.915939 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:24.915744 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85ba0342-5887-4cb0-85f8-0331e7a565e0-maas-api-service-ca\") pod \"85ba0342-5887-4cb0-85f8-0331e7a565e0\" (UID: \"85ba0342-5887-4cb0-85f8-0331e7a565e0\") " Sep 11 04:11:24.916186 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:24.916158 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85ba0342-5887-4cb0-85f8-0331e7a565e0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "85ba0342-5887-4cb0-85f8-0331e7a565e0" (UID: "85ba0342-5887-4cb0-85f8-0331e7a565e0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 04:11:25.017177 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:25.017098 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85ba0342-5887-4cb0-85f8-0331e7a565e0-maas-api-service-ca\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:11:25.766071 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:25.766023 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-57c530-jmsz2" Sep 11 04:11:25.768878 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:25.768845 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-57c530-jmsz2" event={"ID":"85ba0342-5887-4cb0-85f8-0331e7a565e0","Type":"ContainerDied","Data":"31ee70d77f5cb95ea7f956149bd575b453a7f6e7886da1e2a8f1269567b4c609"} Sep 11 04:11:25.769016 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:25.768902 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31ee70d77f5cb95ea7f956149bd575b453a7f6e7886da1e2a8f1269567b4c609" Sep 11 04:11:26.171236 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:26.171205 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:27.064940 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:27.064905 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:32.857365 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:32.857324 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:32.889266 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:32.889229 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-57c530-7765788dfd-hwwtn"] Sep 11 04:11:32.889538 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:32.889511 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-57c530-7765788dfd-hwwtn" podUID="89f51db2-f9e8-43a1-becb-4c2046df5dbc" containerName="payload-pre-processing" containerID="cri-o://55128a01fa53428ecf7c629cc4261df0181960dc47a9490f9c6c2ff3a5f4dace" gracePeriod=30 Sep 11 04:11:32.952647 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:32.952613 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:33.072003 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:33.071971 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-57c530-jmsz2"] Sep 11 04:11:33.077267 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:33.077236 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-57c530-jmsz2"] Sep 11 04:11:33.086364 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:33.086336 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:33.154570 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:33.154543 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-57c530-7765788dfd-hwwtn" Sep 11 04:11:33.167835 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:33.167798 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:33.186554 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:33.186528 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6kfsp\" (UniqueName: \"kubernetes.io/projected/89f51db2-f9e8-43a1-becb-4c2046df5dbc-kube-api-access-6kfsp\") pod \"89f51db2-f9e8-43a1-becb-4c2046df5dbc\" (UID: \"89f51db2-f9e8-43a1-becb-4c2046df5dbc\") " Sep 11 04:11:33.186720 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:33.186661 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89f51db2-f9e8-43a1-becb-4c2046df5dbc-plugins-config-volume\") pod \"89f51db2-f9e8-43a1-becb-4c2046df5dbc\" (UID: \"89f51db2-f9e8-43a1-becb-4c2046df5dbc\") " Sep 11 04:11:33.187033 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:33.187011 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89f51db2-f9e8-43a1-becb-4c2046df5dbc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "89f51db2-f9e8-43a1-becb-4c2046df5dbc" (UID: "89f51db2-f9e8-43a1-becb-4c2046df5dbc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 04:11:33.188962 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:33.188939 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89f51db2-f9e8-43a1-becb-4c2046df5dbc-kube-api-access-6kfsp" (OuterVolumeSpecName: "kube-api-access-6kfsp") pod "89f51db2-f9e8-43a1-becb-4c2046df5dbc" (UID: "89f51db2-f9e8-43a1-becb-4c2046df5dbc"). InnerVolumeSpecName "kube-api-access-6kfsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:11:33.253623 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:33.253585 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:33.287821 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:33.287781 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89f51db2-f9e8-43a1-becb-4c2046df5dbc-plugins-config-volume\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:11:33.287821 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:33.287812 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kfsp\" (UniqueName: \"kubernetes.io/projected/89f51db2-f9e8-43a1-becb-4c2046df5dbc-kube-api-access-6kfsp\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:11:33.353484 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:33.353447 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:33.426682 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:33.426649 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:33.768346 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:33.768263 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85ba0342-5887-4cb0-85f8-0331e7a565e0" path="/var/lib/kubelet/pods/85ba0342-5887-4cb0-85f8-0331e7a565e0/volumes" Sep 11 04:11:33.807095 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:33.807059 2568 generic.go:356] "Generic (PLEG): container finished" podID="89f51db2-f9e8-43a1-becb-4c2046df5dbc" containerID="55128a01fa53428ecf7c629cc4261df0181960dc47a9490f9c6c2ff3a5f4dace" exitCode=0 Sep 11 04:11:33.807246 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:33.807117 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-57c530-7765788dfd-hwwtn" Sep 11 04:11:33.807246 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:33.807134 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-57c530-7765788dfd-hwwtn" event={"ID":"89f51db2-f9e8-43a1-becb-4c2046df5dbc","Type":"ContainerDied","Data":"55128a01fa53428ecf7c629cc4261df0181960dc47a9490f9c6c2ff3a5f4dace"} Sep 11 04:11:33.807246 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:33.807201 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-57c530-7765788dfd-hwwtn" event={"ID":"89f51db2-f9e8-43a1-becb-4c2046df5dbc","Type":"ContainerDied","Data":"bf57c904caed71d3acdbf209622245e456af7eabc8ac8b1eac5fc2e5a764e05e"} Sep 11 04:11:33.807246 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:33.807231 2568 scope.go:117] "RemoveContainer" containerID="55128a01fa53428ecf7c629cc4261df0181960dc47a9490f9c6c2ff3a5f4dace" Sep 11 04:11:33.815786 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:33.815771 2568 scope.go:117] "RemoveContainer" containerID="55128a01fa53428ecf7c629cc4261df0181960dc47a9490f9c6c2ff3a5f4dace" Sep 11 04:11:33.816054 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:11:33.816025 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55128a01fa53428ecf7c629cc4261df0181960dc47a9490f9c6c2ff3a5f4dace\": container with ID starting with 55128a01fa53428ecf7c629cc4261df0181960dc47a9490f9c6c2ff3a5f4dace not found: ID does not exist" containerID="55128a01fa53428ecf7c629cc4261df0181960dc47a9490f9c6c2ff3a5f4dace" Sep 11 04:11:33.816122 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:33.816073 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55128a01fa53428ecf7c629cc4261df0181960dc47a9490f9c6c2ff3a5f4dace"} err="failed to get container status \"55128a01fa53428ecf7c629cc4261df0181960dc47a9490f9c6c2ff3a5f4dace\": rpc error: code = NotFound desc = could not find container \"55128a01fa53428ecf7c629cc4261df0181960dc47a9490f9c6c2ff3a5f4dace\": container with ID starting with 55128a01fa53428ecf7c629cc4261df0181960dc47a9490f9c6c2ff3a5f4dace not found: ID does not exist" Sep 11 04:11:33.823317 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:33.823295 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-57c530-7765788dfd-hwwtn"] Sep 11 04:11:33.827931 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:33.827905 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-57c530-7765788dfd-hwwtn"] Sep 11 04:11:34.142683 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:34.142591 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:34.314918 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:34.314874 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:35.767683 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:35.767649 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89f51db2-f9e8-43a1-becb-4c2046df5dbc" path="/var/lib/kubelet/pods/89f51db2-f9e8-43a1-becb-4c2046df5dbc/volumes" Sep 11 04:11:35.819262 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:35.819227 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:39.923531 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:39.923496 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:39.992261 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:39.992226 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:40.185385 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:40.185269 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:40.659978 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:40.659939 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:40.760700 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:40.760659 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:40.966423 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:40.966346 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq"] Sep 11 04:11:40.966821 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:40.966704 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85ba0342-5887-4cb0-85f8-0331e7a565e0" containerName="revoke-keys" Sep 11 04:11:40.966821 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:40.966715 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="85ba0342-5887-4cb0-85f8-0331e7a565e0" containerName="revoke-keys" Sep 11 04:11:40.966821 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:40.966742 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89f51db2-f9e8-43a1-becb-4c2046df5dbc" containerName="payload-pre-processing" Sep 11 04:11:40.966821 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:40.966747 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="89f51db2-f9e8-43a1-becb-4c2046df5dbc" containerName="payload-pre-processing" Sep 11 04:11:40.966821 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:40.966799 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="85ba0342-5887-4cb0-85f8-0331e7a565e0" containerName="revoke-keys" Sep 11 04:11:40.966821 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:40.966810 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="89f51db2-f9e8-43a1-becb-4c2046df5dbc" containerName="payload-pre-processing" Sep 11 04:11:40.975636 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:40.975610 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq" Sep 11 04:11:40.979554 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:40.979534 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-9cca98-openshift-default-dockercfg-8d7tm\"" Sep 11 04:11:40.995310 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:40.995289 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq"] Sep 11 04:11:41.014204 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:41.014157 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:41.056158 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:41.056107 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c24c91df-de1a-4a81-a096-8323f832d3d4-istio-envoy\") pod \"e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq\" (UID: \"c24c91df-de1a-4a81-a096-8323f832d3d4\") " pod="openshift-ingress/e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq" Sep 11 04:11:41.056323 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:41.056182 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c24c91df-de1a-4a81-a096-8323f832d3d4-credential-socket\") pod \"e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq\" (UID: \"c24c91df-de1a-4a81-a096-8323f832d3d4\") " pod="openshift-ingress/e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq" Sep 11 04:11:41.056323 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:41.056235 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c24c91df-de1a-4a81-a096-8323f832d3d4-workload-certs\") pod \"e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq\" (UID: \"c24c91df-de1a-4a81-a096-8323f832d3d4\") " pod="openshift-ingress/e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq" Sep 11 04:11:41.056323 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:41.056278 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c24c91df-de1a-4a81-a096-8323f832d3d4-workload-socket\") pod \"e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq\" (UID: \"c24c91df-de1a-4a81-a096-8323f832d3d4\") " pod="openshift-ingress/e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq" Sep 11 04:11:41.056323 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:41.056306 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c24c91df-de1a-4a81-a096-8323f832d3d4-istiod-ca-cert\") pod \"e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq\" (UID: \"c24c91df-de1a-4a81-a096-8323f832d3d4\") " pod="openshift-ingress/e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq" Sep 11 04:11:41.056549 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:41.056393 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c24c91df-de1a-4a81-a096-8323f832d3d4-istio-token\") pod \"e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq\" (UID: \"c24c91df-de1a-4a81-a096-8323f832d3d4\") " pod="openshift-ingress/e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq" Sep 11 04:11:41.056549 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:41.056466 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c24c91df-de1a-4a81-a096-8323f832d3d4-istio-data\") pod \"e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq\" (UID: \"c24c91df-de1a-4a81-a096-8323f832d3d4\") " pod="openshift-ingress/e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq" Sep 11 04:11:41.056549 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:41.056499 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87jkc\" (UniqueName: \"kubernetes.io/projected/c24c91df-de1a-4a81-a096-8323f832d3d4-kube-api-access-87jkc\") pod \"e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq\" (UID: \"c24c91df-de1a-4a81-a096-8323f832d3d4\") " pod="openshift-ingress/e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq" Sep 11 04:11:41.056717 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:41.056554 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c24c91df-de1a-4a81-a096-8323f832d3d4-istio-podinfo\") pod \"e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq\" (UID: \"c24c91df-de1a-4a81-a096-8323f832d3d4\") " pod="openshift-ingress/e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq" Sep 11 04:11:41.157606 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:41.157543 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c24c91df-de1a-4a81-a096-8323f832d3d4-istio-envoy\") pod \"e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq\" (UID: \"c24c91df-de1a-4a81-a096-8323f832d3d4\") " pod="openshift-ingress/e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq" Sep 11 04:11:41.158037 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:41.158007 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c24c91df-de1a-4a81-a096-8323f832d3d4-credential-socket\") pod \"e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq\" (UID: \"c24c91df-de1a-4a81-a096-8323f832d3d4\") " pod="openshift-ingress/e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq" Sep 11 04:11:41.158205 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:41.158089 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c24c91df-de1a-4a81-a096-8323f832d3d4-workload-certs\") pod \"e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq\" (UID: \"c24c91df-de1a-4a81-a096-8323f832d3d4\") " pod="openshift-ingress/e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq" Sep 11 04:11:41.158205 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:41.158138 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c24c91df-de1a-4a81-a096-8323f832d3d4-workload-socket\") pod \"e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq\" (UID: \"c24c91df-de1a-4a81-a096-8323f832d3d4\") " pod="openshift-ingress/e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq" Sep 11 04:11:41.158205 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:41.158169 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c24c91df-de1a-4a81-a096-8323f832d3d4-istiod-ca-cert\") pod \"e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq\" (UID: \"c24c91df-de1a-4a81-a096-8323f832d3d4\") " pod="openshift-ingress/e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq" Sep 11 04:11:41.158378 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:41.158224 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c24c91df-de1a-4a81-a096-8323f832d3d4-istio-token\") pod \"e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq\" (UID: \"c24c91df-de1a-4a81-a096-8323f832d3d4\") " pod="openshift-ingress/e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq" Sep 11 04:11:41.158378 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:41.158258 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c24c91df-de1a-4a81-a096-8323f832d3d4-istio-data\") pod \"e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq\" (UID: \"c24c91df-de1a-4a81-a096-8323f832d3d4\") " pod="openshift-ingress/e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq" Sep 11 04:11:41.158378 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:41.158288 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87jkc\" (UniqueName: \"kubernetes.io/projected/c24c91df-de1a-4a81-a096-8323f832d3d4-kube-api-access-87jkc\") pod \"e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq\" (UID: \"c24c91df-de1a-4a81-a096-8323f832d3d4\") " pod="openshift-ingress/e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq" Sep 11 04:11:41.158378 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:41.158346 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c24c91df-de1a-4a81-a096-8323f832d3d4-istio-podinfo\") pod \"e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq\" (UID: \"c24c91df-de1a-4a81-a096-8323f832d3d4\") " pod="openshift-ingress/e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq" Sep 11 04:11:41.159095 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:41.159021 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c24c91df-de1a-4a81-a096-8323f832d3d4-workload-socket\") pod \"e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq\" (UID: \"c24c91df-de1a-4a81-a096-8323f832d3d4\") " pod="openshift-ingress/e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq" Sep 11 04:11:41.159199 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:41.159087 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c24c91df-de1a-4a81-a096-8323f832d3d4-istio-data\") pod \"e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq\" (UID: \"c24c91df-de1a-4a81-a096-8323f832d3d4\") " pod="openshift-ingress/e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq" Sep 11 04:11:41.159199 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:41.159169 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c24c91df-de1a-4a81-a096-8323f832d3d4-credential-socket\") pod \"e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq\" (UID: \"c24c91df-de1a-4a81-a096-8323f832d3d4\") " pod="openshift-ingress/e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq" Sep 11 04:11:41.159542 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:41.159519 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c24c91df-de1a-4a81-a096-8323f832d3d4-istiod-ca-cert\") pod \"e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq\" (UID: \"c24c91df-de1a-4a81-a096-8323f832d3d4\") " pod="openshift-ingress/e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq" Sep 11 04:11:41.159685 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:41.159660 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c24c91df-de1a-4a81-a096-8323f832d3d4-workload-certs\") pod \"e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq\" (UID: \"c24c91df-de1a-4a81-a096-8323f832d3d4\") " pod="openshift-ingress/e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq" Sep 11 04:11:41.160548 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:41.160513 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c24c91df-de1a-4a81-a096-8323f832d3d4-istio-envoy\") pod \"e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq\" (UID: \"c24c91df-de1a-4a81-a096-8323f832d3d4\") " pod="openshift-ingress/e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq" Sep 11 04:11:41.161216 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:41.161194 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c24c91df-de1a-4a81-a096-8323f832d3d4-istio-podinfo\") pod \"e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq\" (UID: \"c24c91df-de1a-4a81-a096-8323f832d3d4\") " pod="openshift-ingress/e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq" Sep 11 04:11:41.167796 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:41.167762 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87jkc\" (UniqueName: \"kubernetes.io/projected/c24c91df-de1a-4a81-a096-8323f832d3d4-kube-api-access-87jkc\") pod \"e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq\" (UID: \"c24c91df-de1a-4a81-a096-8323f832d3d4\") " pod="openshift-ingress/e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq" Sep 11 04:11:41.167970 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:41.167949 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c24c91df-de1a-4a81-a096-8323f832d3d4-istio-token\") pod \"e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq\" (UID: \"c24c91df-de1a-4a81-a096-8323f832d3d4\") " pod="openshift-ingress/e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq" Sep 11 04:11:41.216871 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:41.216790 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:41.288235 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:41.288200 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq" Sep 11 04:11:41.568946 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:41.568921 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq"] Sep 11 04:11:41.571104 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:11:41.571073 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc24c91df_de1a_4a81_a096_8323f832d3d4.slice/crio-0db7b07f44387704091d314e2613d2b6e62b9c68cb6accb85608e4d7e20071fd WatchSource:0}: Error finding container 0db7b07f44387704091d314e2613d2b6e62b9c68cb6accb85608e4d7e20071fd: Status 404 returned error can't find the container with id 0db7b07f44387704091d314e2613d2b6e62b9c68cb6accb85608e4d7e20071fd Sep 11 04:11:41.573399 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:41.573362 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 04:11:41.573502 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:41.573432 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 04:11:41.573502 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:41.573461 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 04:11:41.848071 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:41.847962 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq" event={"ID":"c24c91df-de1a-4a81-a096-8323f832d3d4","Type":"ContainerStarted","Data":"2c4939cc3e66b726f70bc7c6f95349a3b12e6d7ab41485d9b9244f583df4b6d6"} Sep 11 04:11:41.848071 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:41.848017 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq" event={"ID":"c24c91df-de1a-4a81-a096-8323f832d3d4","Type":"ContainerStarted","Data":"0db7b07f44387704091d314e2613d2b6e62b9c68cb6accb85608e4d7e20071fd"} Sep 11 04:11:41.872225 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:41.872167 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq" podStartSLOduration=1.8721467999999999 podStartE2EDuration="1.8721468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:11:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:11:41.869299203 +0000 UTC m=+1116.622930605" watchObservedRunningTime="2026-09-11 04:11:41.8721468 +0000 UTC m=+1116.625778202" Sep 11 04:11:42.206067 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:42.206022 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:42.288816 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:42.288785 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq" Sep 11 04:11:42.293671 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:42.293647 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq" Sep 11 04:11:42.374983 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:42.374949 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:42.455566 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:42.455529 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:42.853908 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:42.853878 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq" Sep 11 04:11:42.854797 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:42.854773 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq" Sep 11 04:11:43.024311 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:43.024280 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:43.126360 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:43.126275 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:43.722359 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:43.722325 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:49.631624 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:49.631579 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:49.732611 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:49.732575 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:49.873167 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:49.873131 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:50.201362 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:50.201315 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:50.520072 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:50.519953 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:52.793581 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:52.793545 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:52.820998 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:52.820964 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a28c9f-5b6c74bcbc-j7hwv"] Sep 11 04:11:52.821231 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:52.821202 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-a28c9f-5b6c74bcbc-j7hwv" podUID="3818a5e1-5ed9-4a12-8f89-7c7b58ccb0ec" containerName="payload-processing" containerID="cri-o://4c09006dfc68014aa0b63f96f4e3d3c0c83c82fd1bd233e160668f760a8d7690" gracePeriod=30 Sep 11 04:11:52.894477 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:52.894435 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:52.991776 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:52.991729 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:53.055492 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:53.055457 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:53.074844 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:53.074822 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a28c9f-5b6c74bcbc-j7hwv" Sep 11 04:11:53.165695 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:53.165654 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k8xgb\" (UniqueName: \"kubernetes.io/projected/3818a5e1-5ed9-4a12-8f89-7c7b58ccb0ec-kube-api-access-k8xgb\") pod \"3818a5e1-5ed9-4a12-8f89-7c7b58ccb0ec\" (UID: \"3818a5e1-5ed9-4a12-8f89-7c7b58ccb0ec\") " Sep 11 04:11:53.165897 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:53.165768 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3818a5e1-5ed9-4a12-8f89-7c7b58ccb0ec-plugins-config-volume\") pod \"3818a5e1-5ed9-4a12-8f89-7c7b58ccb0ec\" (UID: \"3818a5e1-5ed9-4a12-8f89-7c7b58ccb0ec\") " Sep 11 04:11:53.166368 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:53.166336 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3818a5e1-5ed9-4a12-8f89-7c7b58ccb0ec-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3818a5e1-5ed9-4a12-8f89-7c7b58ccb0ec" (UID: "3818a5e1-5ed9-4a12-8f89-7c7b58ccb0ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 04:11:53.168773 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:53.168732 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3818a5e1-5ed9-4a12-8f89-7c7b58ccb0ec-kube-api-access-k8xgb" (OuterVolumeSpecName: "kube-api-access-k8xgb") pod "3818a5e1-5ed9-4a12-8f89-7c7b58ccb0ec" (UID: "3818a5e1-5ed9-4a12-8f89-7c7b58ccb0ec"). InnerVolumeSpecName "kube-api-access-k8xgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:11:53.267446 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:53.267407 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3818a5e1-5ed9-4a12-8f89-7c7b58ccb0ec-plugins-config-volume\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:11:53.267446 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:53.267442 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8xgb\" (UniqueName: \"kubernetes.io/projected/3818a5e1-5ed9-4a12-8f89-7c7b58ccb0ec-kube-api-access-k8xgb\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:11:53.402849 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:53.402813 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:53.503900 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:53.503863 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:53.605366 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:53.605336 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:53.911903 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:53.911870 2568 generic.go:356] "Generic (PLEG): container finished" podID="3818a5e1-5ed9-4a12-8f89-7c7b58ccb0ec" containerID="4c09006dfc68014aa0b63f96f4e3d3c0c83c82fd1bd233e160668f760a8d7690" exitCode=0 Sep 11 04:11:53.912428 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:53.911929 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a28c9f-5b6c74bcbc-j7hwv" Sep 11 04:11:53.912428 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:53.911963 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a28c9f-5b6c74bcbc-j7hwv" event={"ID":"3818a5e1-5ed9-4a12-8f89-7c7b58ccb0ec","Type":"ContainerDied","Data":"4c09006dfc68014aa0b63f96f4e3d3c0c83c82fd1bd233e160668f760a8d7690"} Sep 11 04:11:53.912428 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:53.912027 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a28c9f-5b6c74bcbc-j7hwv" event={"ID":"3818a5e1-5ed9-4a12-8f89-7c7b58ccb0ec","Type":"ContainerDied","Data":"455a114ae498fd5023367e51d3aefa290370915e8149e44dd462fda00c1e94f2"} Sep 11 04:11:53.912428 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:53.912072 2568 scope.go:117] "RemoveContainer" containerID="4c09006dfc68014aa0b63f96f4e3d3c0c83c82fd1bd233e160668f760a8d7690" Sep 11 04:11:53.920670 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:53.920639 2568 scope.go:117] "RemoveContainer" containerID="4c09006dfc68014aa0b63f96f4e3d3c0c83c82fd1bd233e160668f760a8d7690" Sep 11 04:11:53.920932 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:11:53.920913 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c09006dfc68014aa0b63f96f4e3d3c0c83c82fd1bd233e160668f760a8d7690\": container with ID starting with 4c09006dfc68014aa0b63f96f4e3d3c0c83c82fd1bd233e160668f760a8d7690 not found: ID does not exist" containerID="4c09006dfc68014aa0b63f96f4e3d3c0c83c82fd1bd233e160668f760a8d7690" Sep 11 04:11:53.920995 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:53.920940 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c09006dfc68014aa0b63f96f4e3d3c0c83c82fd1bd233e160668f760a8d7690"} err="failed to get container status \"4c09006dfc68014aa0b63f96f4e3d3c0c83c82fd1bd233e160668f760a8d7690\": rpc error: code = NotFound desc = could not find container \"4c09006dfc68014aa0b63f96f4e3d3c0c83c82fd1bd233e160668f760a8d7690\": container with ID starting with 4c09006dfc68014aa0b63f96f4e3d3c0c83c82fd1bd233e160668f760a8d7690 not found: ID does not exist" Sep 11 04:11:53.929574 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:53.929550 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a28c9f-5b6c74bcbc-j7hwv"] Sep 11 04:11:53.934038 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:53.934017 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a28c9f-5b6c74bcbc-j7hwv"] Sep 11 04:11:54.610338 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:54.610308 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:54.710161 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:54.710112 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:55.004619 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:55.004581 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:55.192932 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:55.192901 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:55.288113 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:55.288008 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:55.299095 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:55.299065 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-34523cde-85746cb8b-z54nv"] Sep 11 04:11:55.299427 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:55.299414 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3818a5e1-5ed9-4a12-8f89-7c7b58ccb0ec" containerName="payload-processing" Sep 11 04:11:55.299478 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:55.299428 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3818a5e1-5ed9-4a12-8f89-7c7b58ccb0ec" containerName="payload-processing" Sep 11 04:11:55.299519 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:55.299487 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="3818a5e1-5ed9-4a12-8f89-7c7b58ccb0ec" containerName="payload-processing" Sep 11 04:11:55.303744 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:55.303723 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-34523cde-85746cb8b-z54nv" Sep 11 04:11:55.306373 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:55.306350 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-34523cde-dockercfg-7bngv\"" Sep 11 04:11:55.306486 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:55.306437 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-34523cde\"" Sep 11 04:11:55.311961 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:55.311936 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-34523cde-85746cb8b-z54nv"] Sep 11 04:11:55.386424 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:55.386392 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2280354-4bbe-429e-baea-2c5c34e4297b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-34523cde-85746cb8b-z54nv\" (UID: \"d2280354-4bbe-429e-baea-2c5c34e4297b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-34523cde-85746cb8b-z54nv" Sep 11 04:11:55.386595 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:55.386457 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvfjw\" (UniqueName: \"kubernetes.io/projected/d2280354-4bbe-429e-baea-2c5c34e4297b-kube-api-access-hvfjw\") pod \"payload-pre-processing-e2e-mt-34523cde-85746cb8b-z54nv\" (UID: \"d2280354-4bbe-429e-baea-2c5c34e4297b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-34523cde-85746cb8b-z54nv" Sep 11 04:11:55.487661 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:55.487620 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvfjw\" (UniqueName: \"kubernetes.io/projected/d2280354-4bbe-429e-baea-2c5c34e4297b-kube-api-access-hvfjw\") pod \"payload-pre-processing-e2e-mt-34523cde-85746cb8b-z54nv\" (UID: \"d2280354-4bbe-429e-baea-2c5c34e4297b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-34523cde-85746cb8b-z54nv" Sep 11 04:11:55.487849 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:55.487687 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2280354-4bbe-429e-baea-2c5c34e4297b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-34523cde-85746cb8b-z54nv\" (UID: \"d2280354-4bbe-429e-baea-2c5c34e4297b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-34523cde-85746cb8b-z54nv" Sep 11 04:11:55.488397 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:55.488382 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2280354-4bbe-429e-baea-2c5c34e4297b-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-34523cde-85746cb8b-z54nv\" (UID: \"d2280354-4bbe-429e-baea-2c5c34e4297b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-34523cde-85746cb8b-z54nv" Sep 11 04:11:55.496673 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:55.496647 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvfjw\" (UniqueName: \"kubernetes.io/projected/d2280354-4bbe-429e-baea-2c5c34e4297b-kube-api-access-hvfjw\") pod \"payload-pre-processing-e2e-mt-34523cde-85746cb8b-z54nv\" (UID: \"d2280354-4bbe-429e-baea-2c5c34e4297b\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-34523cde-85746cb8b-z54nv" Sep 11 04:11:55.614824 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:55.614737 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-34523cde-85746cb8b-z54nv" Sep 11 04:11:55.768728 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:55.768693 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3818a5e1-5ed9-4a12-8f89-7c7b58ccb0ec" path="/var/lib/kubelet/pods/3818a5e1-5ed9-4a12-8f89-7c7b58ccb0ec/volumes" Sep 11 04:11:55.886074 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:55.885988 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-34523cde-85746cb8b-z54nv"] Sep 11 04:11:55.889021 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:11:55.888989 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2280354_4bbe_429e_baea_2c5c34e4297b.slice/crio-609ab2448776aedd50496a26b576aa042ffc668ad154b2833e319249b1d6f778 WatchSource:0}: Error finding container 609ab2448776aedd50496a26b576aa042ffc668ad154b2833e319249b1d6f778: Status 404 returned error can't find the container with id 609ab2448776aedd50496a26b576aa042ffc668ad154b2833e319249b1d6f778 Sep 11 04:11:55.923182 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:55.923154 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-34523cde-85746cb8b-z54nv" event={"ID":"d2280354-4bbe-429e-baea-2c5c34e4297b","Type":"ContainerStarted","Data":"609ab2448776aedd50496a26b576aa042ffc668ad154b2833e319249b1d6f778"} Sep 11 04:11:56.931358 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:56.931311 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-34523cde-85746cb8b-z54nv" event={"ID":"d2280354-4bbe-429e-baea-2c5c34e4297b","Type":"ContainerStarted","Data":"d24f2d424cb2780e48c253c851ccda23cf901521dceb5da65c140c6315296f6b"} Sep 11 04:11:56.931735 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:56.931408 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-34523cde-85746cb8b-z54nv" Sep 11 04:11:56.949766 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:56.949706 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-34523cde-85746cb8b-z54nv" podStartSLOduration=1.949687167 podStartE2EDuration="1.949687167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:11:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:11:56.946805025 +0000 UTC m=+1131.700436430" watchObservedRunningTime="2026-09-11 04:11:56.949687167 +0000 UTC m=+1131.703318570" Sep 11 04:11:59.534270 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:59.534226 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:59.635043 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:59.634994 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:11:59.782521 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:59.782489 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s"] Sep 11 04:11:59.782766 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:59.782740 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" podUID="136cff62-ad79-4976-81dd-ac786e4d4cec" containerName="istio-proxy" containerID="cri-o://97d702943b0b252e3faccfa9c9c412d3cce055fb30a5c6d437351f01592afe5f" gracePeriod=30 Sep 11 04:11:59.802615 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:11:59.802531 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:00.188033 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:00.188001 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fca08f-jgpvj"] Sep 11 04:12:00.191567 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:00.191550 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fca08f-jgpvj" Sep 11 04:12:00.196110 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:00.196087 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 04:12:00.196262 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:00.196118 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6brsr\"" Sep 11 04:12:00.206671 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:00.206644 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fca08f-jgpvj"] Sep 11 04:12:00.302130 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:00.302093 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:00.330417 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:00.330384 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/248091f3-1915-4236-8751-751ca75facfb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fca08f-jgpvj\" (UID: \"248091f3-1915-4236-8751-751ca75facfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fca08f-jgpvj" Sep 11 04:12:00.431533 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:00.431497 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/248091f3-1915-4236-8751-751ca75facfb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fca08f-jgpvj\" (UID: \"248091f3-1915-4236-8751-751ca75facfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fca08f-jgpvj" Sep 11 04:12:00.432216 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:00.432191 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/248091f3-1915-4236-8751-751ca75facfb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fca08f-jgpvj\" (UID: \"248091f3-1915-4236-8751-751ca75facfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fca08f-jgpvj" Sep 11 04:12:00.501896 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:00.501795 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fca08f-jgpvj" Sep 11 04:12:00.780107 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:00.779999 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fca08f-jgpvj"] Sep 11 04:12:00.783293 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:12:00.783252 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod248091f3_1915_4236_8751_751ca75facfb.slice/crio-228fc622498506767674cbddeaff63927612eb7500590c894932b8d885e27ee7 WatchSource:0}: Error finding container 228fc622498506767674cbddeaff63927612eb7500590c894932b8d885e27ee7: Status 404 returned error can't find the container with id 228fc622498506767674cbddeaff63927612eb7500590c894932b8d885e27ee7 Sep 11 04:12:00.953396 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:00.953356 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fca08f-jgpvj" event={"ID":"248091f3-1915-4236-8751-751ca75facfb","Type":"ContainerStarted","Data":"228fc622498506767674cbddeaff63927612eb7500590c894932b8d885e27ee7"} Sep 11 04:12:01.965806 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:01.965767 2568 generic.go:356] "Generic (PLEG): container finished" podID="248091f3-1915-4236-8751-751ca75facfb" containerID="abae39621b38677eefd5228be3e1b7932853a2ffa01b8d7d39b1534ebb3779dc" exitCode=0 Sep 11 04:12:01.966201 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:01.965845 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fca08f-jgpvj" event={"ID":"248091f3-1915-4236-8751-751ca75facfb","Type":"ContainerDied","Data":"abae39621b38677eefd5228be3e1b7932853a2ffa01b8d7d39b1534ebb3779dc"} Sep 11 04:12:02.371677 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:02.371586 2568 patch_prober.go:28] interesting pod/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 04:12:02.371835 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:02.371668 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" podUID="136cff62-ad79-4976-81dd-ac786e4d4cec" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 04:12:03.101004 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:03.100983 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fca08f-jgpvj" Sep 11 04:12:03.255325 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:03.255236 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/248091f3-1915-4236-8751-751ca75facfb-maas-api-service-ca\") pod \"248091f3-1915-4236-8751-751ca75facfb\" (UID: \"248091f3-1915-4236-8751-751ca75facfb\") " Sep 11 04:12:03.255608 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:03.255581 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/248091f3-1915-4236-8751-751ca75facfb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "248091f3-1915-4236-8751-751ca75facfb" (UID: "248091f3-1915-4236-8751-751ca75facfb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 04:12:03.356100 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:03.356031 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/248091f3-1915-4236-8751-751ca75facfb-maas-api-service-ca\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:12:03.979727 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:03.979692 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fca08f-jgpvj" event={"ID":"248091f3-1915-4236-8751-751ca75facfb","Type":"ContainerDied","Data":"228fc622498506767674cbddeaff63927612eb7500590c894932b8d885e27ee7"} Sep 11 04:12:03.979881 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:03.979748 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="228fc622498506767674cbddeaff63927612eb7500590c894932b8d885e27ee7" Sep 11 04:12:03.979881 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:03.979707 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fca08f-jgpvj" Sep 11 04:12:04.987477 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:04.987446 2568 generic.go:356] "Generic (PLEG): container finished" podID="136cff62-ad79-4976-81dd-ac786e4d4cec" containerID="97d702943b0b252e3faccfa9c9c412d3cce055fb30a5c6d437351f01592afe5f" exitCode=0 Sep 11 04:12:04.987863 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:04.987494 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" event={"ID":"136cff62-ad79-4976-81dd-ac786e4d4cec","Type":"ContainerDied","Data":"97d702943b0b252e3faccfa9c9c412d3cce055fb30a5c6d437351f01592afe5f"} Sep 11 04:12:04.987863 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:04.987557 2568 scope.go:117] "RemoveContainer" containerID="2d01931adcd8522c7f52f6336c29b07b3a6ca37a8e83d26653caf9834e041989" Sep 11 04:12:05.037219 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:05.037197 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" Sep 11 04:12:05.071734 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:05.071678 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/136cff62-ad79-4976-81dd-ac786e4d4cec-istiod-ca-cert\") pod \"136cff62-ad79-4976-81dd-ac786e4d4cec\" (UID: \"136cff62-ad79-4976-81dd-ac786e4d4cec\") " Sep 11 04:12:05.071734 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:05.071723 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gdnpl\" (UniqueName: \"kubernetes.io/projected/136cff62-ad79-4976-81dd-ac786e4d4cec-kube-api-access-gdnpl\") pod \"136cff62-ad79-4976-81dd-ac786e4d4cec\" (UID: \"136cff62-ad79-4976-81dd-ac786e4d4cec\") " Sep 11 04:12:05.071913 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:05.071756 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/136cff62-ad79-4976-81dd-ac786e4d4cec-istio-podinfo\") pod \"136cff62-ad79-4976-81dd-ac786e4d4cec\" (UID: \"136cff62-ad79-4976-81dd-ac786e4d4cec\") " Sep 11 04:12:05.071913 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:05.071773 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/136cff62-ad79-4976-81dd-ac786e4d4cec-istio-data\") pod \"136cff62-ad79-4976-81dd-ac786e4d4cec\" (UID: \"136cff62-ad79-4976-81dd-ac786e4d4cec\") " Sep 11 04:12:05.071913 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:05.071800 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/136cff62-ad79-4976-81dd-ac786e4d4cec-workload-socket\") pod \"136cff62-ad79-4976-81dd-ac786e4d4cec\" (UID: \"136cff62-ad79-4976-81dd-ac786e4d4cec\") " Sep 11 04:12:05.071913 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:05.071816 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/136cff62-ad79-4976-81dd-ac786e4d4cec-workload-certs\") pod \"136cff62-ad79-4976-81dd-ac786e4d4cec\" (UID: \"136cff62-ad79-4976-81dd-ac786e4d4cec\") " Sep 11 04:12:05.071913 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:05.071835 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/136cff62-ad79-4976-81dd-ac786e4d4cec-credential-socket\") pod \"136cff62-ad79-4976-81dd-ac786e4d4cec\" (UID: \"136cff62-ad79-4976-81dd-ac786e4d4cec\") " Sep 11 04:12:05.071913 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:05.071861 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/136cff62-ad79-4976-81dd-ac786e4d4cec-istio-token\") pod \"136cff62-ad79-4976-81dd-ac786e4d4cec\" (UID: \"136cff62-ad79-4976-81dd-ac786e4d4cec\") " Sep 11 04:12:05.071913 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:05.071911 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/136cff62-ad79-4976-81dd-ac786e4d4cec-istio-envoy\") pod \"136cff62-ad79-4976-81dd-ac786e4d4cec\" (UID: \"136cff62-ad79-4976-81dd-ac786e4d4cec\") " Sep 11 04:12:05.072309 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:05.072144 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/136cff62-ad79-4976-81dd-ac786e4d4cec-istio-data" (OuterVolumeSpecName: "istio-data") pod "136cff62-ad79-4976-81dd-ac786e4d4cec" (UID: "136cff62-ad79-4976-81dd-ac786e4d4cec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:12:05.072364 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:05.072300 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/136cff62-ad79-4976-81dd-ac786e4d4cec-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "136cff62-ad79-4976-81dd-ac786e4d4cec" (UID: "136cff62-ad79-4976-81dd-ac786e4d4cec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:12:05.072428 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:05.072373 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/136cff62-ad79-4976-81dd-ac786e4d4cec-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "136cff62-ad79-4976-81dd-ac786e4d4cec" (UID: "136cff62-ad79-4976-81dd-ac786e4d4cec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:12:05.072525 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:05.072502 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/136cff62-ad79-4976-81dd-ac786e4d4cec-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "136cff62-ad79-4976-81dd-ac786e4d4cec" (UID: "136cff62-ad79-4976-81dd-ac786e4d4cec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:12:05.072617 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:05.072595 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/136cff62-ad79-4976-81dd-ac786e4d4cec-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "136cff62-ad79-4976-81dd-ac786e4d4cec" (UID: "136cff62-ad79-4976-81dd-ac786e4d4cec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 04:12:05.074222 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:05.074196 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/136cff62-ad79-4976-81dd-ac786e4d4cec-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "136cff62-ad79-4976-81dd-ac786e4d4cec" (UID: "136cff62-ad79-4976-81dd-ac786e4d4cec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 04:12:05.074222 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:05.074208 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/136cff62-ad79-4976-81dd-ac786e4d4cec-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "136cff62-ad79-4976-81dd-ac786e4d4cec" (UID: "136cff62-ad79-4976-81dd-ac786e4d4cec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:12:05.074363 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:05.074219 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/136cff62-ad79-4976-81dd-ac786e4d4cec-kube-api-access-gdnpl" (OuterVolumeSpecName: "kube-api-access-gdnpl") pod "136cff62-ad79-4976-81dd-ac786e4d4cec" (UID: "136cff62-ad79-4976-81dd-ac786e4d4cec"). InnerVolumeSpecName "kube-api-access-gdnpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:12:05.074703 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:05.074676 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/136cff62-ad79-4976-81dd-ac786e4d4cec-istio-token" (OuterVolumeSpecName: "istio-token") pod "136cff62-ad79-4976-81dd-ac786e4d4cec" (UID: "136cff62-ad79-4976-81dd-ac786e4d4cec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:12:05.173201 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:05.173172 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/136cff62-ad79-4976-81dd-ac786e4d4cec-istio-podinfo\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:12:05.173201 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:05.173198 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/136cff62-ad79-4976-81dd-ac786e4d4cec-istio-data\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:12:05.173201 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:05.173206 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/136cff62-ad79-4976-81dd-ac786e4d4cec-workload-socket\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:12:05.173201 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:05.173215 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/136cff62-ad79-4976-81dd-ac786e4d4cec-workload-certs\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:12:05.173460 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:05.173223 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/136cff62-ad79-4976-81dd-ac786e4d4cec-credential-socket\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:12:05.173460 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:05.173232 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/136cff62-ad79-4976-81dd-ac786e4d4cec-istio-token\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:12:05.173460 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:05.173244 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/136cff62-ad79-4976-81dd-ac786e4d4cec-istio-envoy\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:12:05.173460 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:05.173257 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/136cff62-ad79-4976-81dd-ac786e4d4cec-istiod-ca-cert\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:12:05.173460 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:05.173271 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdnpl\" (UniqueName: \"kubernetes.io/projected/136cff62-ad79-4976-81dd-ac786e4d4cec-kube-api-access-gdnpl\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:12:05.993788 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:05.993756 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" Sep 11 04:12:05.994245 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:05.993806 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s" event={"ID":"136cff62-ad79-4976-81dd-ac786e4d4cec","Type":"ContainerDied","Data":"78b2467b2a72285920c7f8234c12f0c225e650cb00adaf87c5cffcd215d1864b"} Sep 11 04:12:05.994245 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:05.993866 2568 scope.go:117] "RemoveContainer" containerID="97d702943b0b252e3faccfa9c9c412d3cce055fb30a5c6d437351f01592afe5f" Sep 11 04:12:06.011616 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:06.011590 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s"] Sep 11 04:12:06.015373 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:06.015351 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a28c9f-openshift-default-6686cbc66d-xd54s"] Sep 11 04:12:07.768095 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:07.768039 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="136cff62-ad79-4976-81dd-ac786e4d4cec" path="/var/lib/kubelet/pods/136cff62-ad79-4976-81dd-ac786e4d4cec/volumes" Sep 11 04:12:07.938793 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:07.938764 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-34523cde-85746cb8b-z54nv" Sep 11 04:12:08.129266 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:08.129172 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:09.975579 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:09.975530 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:10.075171 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:10.075135 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq"] Sep 11 04:12:10.075394 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:10.075353 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq" podUID="c24c91df-de1a-4a81-a096-8323f832d3d4" containerName="istio-proxy" containerID="cri-o://2c4939cc3e66b726f70bc7c6f95349a3b12e6d7ab41485d9b9244f583df4b6d6" gracePeriod=30 Sep 11 04:12:10.099970 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:10.099940 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:10.329240 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:10.329149 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:10.449127 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:10.449097 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:10.826142 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:10.826082 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:10.914968 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:10.914905 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:10.992215 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:10.992173 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:11.056404 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:11.056374 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fca08f-jgpvj"] Sep 11 04:12:11.058776 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:11.058748 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fca08f-jgpvj"] Sep 11 04:12:11.059584 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:11.059558 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:11.124452 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:11.124370 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:11.450915 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:11.450872 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:11.770441 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:11.770359 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="248091f3-1915-4236-8751-751ca75facfb" path="/var/lib/kubelet/pods/248091f3-1915-4236-8751-751ca75facfb/volumes" Sep 11 04:12:12.856803 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:12.856766 2568 patch_prober.go:28] interesting pod/e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 04:12:12.857308 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:12.856834 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq" podUID="c24c91df-de1a-4a81-a096-8323f832d3d4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 04:12:15.329493 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:15.329469 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq" Sep 11 04:12:15.363962 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:15.363889 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c24c91df-de1a-4a81-a096-8323f832d3d4-workload-certs\") pod \"c24c91df-de1a-4a81-a096-8323f832d3d4\" (UID: \"c24c91df-de1a-4a81-a096-8323f832d3d4\") " Sep 11 04:12:15.363962 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:15.363936 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c24c91df-de1a-4a81-a096-8323f832d3d4-istio-data\") pod \"c24c91df-de1a-4a81-a096-8323f832d3d4\" (UID: \"c24c91df-de1a-4a81-a096-8323f832d3d4\") " Sep 11 04:12:15.364182 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:15.363969 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87jkc\" (UniqueName: \"kubernetes.io/projected/c24c91df-de1a-4a81-a096-8323f832d3d4-kube-api-access-87jkc\") pod \"c24c91df-de1a-4a81-a096-8323f832d3d4\" (UID: \"c24c91df-de1a-4a81-a096-8323f832d3d4\") " Sep 11 04:12:15.364182 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:15.364004 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c24c91df-de1a-4a81-a096-8323f832d3d4-istiod-ca-cert\") pod \"c24c91df-de1a-4a81-a096-8323f832d3d4\" (UID: \"c24c91df-de1a-4a81-a096-8323f832d3d4\") " Sep 11 04:12:15.364182 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:15.364068 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c24c91df-de1a-4a81-a096-8323f832d3d4-credential-socket\") pod \"c24c91df-de1a-4a81-a096-8323f832d3d4\" (UID: \"c24c91df-de1a-4a81-a096-8323f832d3d4\") " Sep 11 04:12:15.364182 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:15.364122 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c24c91df-de1a-4a81-a096-8323f832d3d4-istio-podinfo\") pod \"c24c91df-de1a-4a81-a096-8323f832d3d4\" (UID: \"c24c91df-de1a-4a81-a096-8323f832d3d4\") " Sep 11 04:12:15.364182 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:15.364157 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c24c91df-de1a-4a81-a096-8323f832d3d4-istio-token\") pod \"c24c91df-de1a-4a81-a096-8323f832d3d4\" (UID: \"c24c91df-de1a-4a81-a096-8323f832d3d4\") " Sep 11 04:12:15.364457 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:15.364206 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c24c91df-de1a-4a81-a096-8323f832d3d4-istio-envoy\") pod \"c24c91df-de1a-4a81-a096-8323f832d3d4\" (UID: \"c24c91df-de1a-4a81-a096-8323f832d3d4\") " Sep 11 04:12:15.364457 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:15.364209 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c24c91df-de1a-4a81-a096-8323f832d3d4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c24c91df-de1a-4a81-a096-8323f832d3d4" (UID: "c24c91df-de1a-4a81-a096-8323f832d3d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:12:15.364457 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:15.364253 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c24c91df-de1a-4a81-a096-8323f832d3d4-workload-socket\") pod \"c24c91df-de1a-4a81-a096-8323f832d3d4\" (UID: \"c24c91df-de1a-4a81-a096-8323f832d3d4\") " Sep 11 04:12:15.364457 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:15.364262 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c24c91df-de1a-4a81-a096-8323f832d3d4-istio-data" (OuterVolumeSpecName: "istio-data") pod "c24c91df-de1a-4a81-a096-8323f832d3d4" (UID: "c24c91df-de1a-4a81-a096-8323f832d3d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:12:15.364457 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:15.364399 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c24c91df-de1a-4a81-a096-8323f832d3d4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c24c91df-de1a-4a81-a096-8323f832d3d4" (UID: "c24c91df-de1a-4a81-a096-8323f832d3d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:12:15.364724 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:15.364592 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c24c91df-de1a-4a81-a096-8323f832d3d4-credential-socket\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:12:15.364724 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:15.364612 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c24c91df-de1a-4a81-a096-8323f832d3d4-workload-certs\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:12:15.364724 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:15.364612 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c24c91df-de1a-4a81-a096-8323f832d3d4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c24c91df-de1a-4a81-a096-8323f832d3d4" (UID: "c24c91df-de1a-4a81-a096-8323f832d3d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:12:15.364724 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:15.364625 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c24c91df-de1a-4a81-a096-8323f832d3d4-istio-data\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:12:15.364936 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:15.364773 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c24c91df-de1a-4a81-a096-8323f832d3d4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c24c91df-de1a-4a81-a096-8323f832d3d4" (UID: "c24c91df-de1a-4a81-a096-8323f832d3d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 04:12:15.366439 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:15.366411 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c24c91df-de1a-4a81-a096-8323f832d3d4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c24c91df-de1a-4a81-a096-8323f832d3d4" (UID: "c24c91df-de1a-4a81-a096-8323f832d3d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:12:15.366551 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:15.366468 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c24c91df-de1a-4a81-a096-8323f832d3d4-istio-token" (OuterVolumeSpecName: "istio-token") pod "c24c91df-de1a-4a81-a096-8323f832d3d4" (UID: "c24c91df-de1a-4a81-a096-8323f832d3d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:12:15.366747 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:15.366728 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c24c91df-de1a-4a81-a096-8323f832d3d4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c24c91df-de1a-4a81-a096-8323f832d3d4" (UID: "c24c91df-de1a-4a81-a096-8323f832d3d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 04:12:15.366820 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:15.366777 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c24c91df-de1a-4a81-a096-8323f832d3d4-kube-api-access-87jkc" (OuterVolumeSpecName: "kube-api-access-87jkc") pod "c24c91df-de1a-4a81-a096-8323f832d3d4" (UID: "c24c91df-de1a-4a81-a096-8323f832d3d4"). InnerVolumeSpecName "kube-api-access-87jkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:12:15.461386 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:15.461348 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:15.465398 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:15.465376 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87jkc\" (UniqueName: \"kubernetes.io/projected/c24c91df-de1a-4a81-a096-8323f832d3d4-kube-api-access-87jkc\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:12:15.465466 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:15.465402 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c24c91df-de1a-4a81-a096-8323f832d3d4-istiod-ca-cert\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:12:15.465466 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:15.465412 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c24c91df-de1a-4a81-a096-8323f832d3d4-istio-podinfo\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:12:15.465466 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:15.465421 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c24c91df-de1a-4a81-a096-8323f832d3d4-istio-token\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:12:15.465466 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:15.465428 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c24c91df-de1a-4a81-a096-8323f832d3d4-istio-envoy\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:12:15.465466 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:15.465436 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c24c91df-de1a-4a81-a096-8323f832d3d4-workload-socket\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:12:16.049648 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:16.049551 2568 generic.go:356] "Generic (PLEG): container finished" podID="c24c91df-de1a-4a81-a096-8323f832d3d4" containerID="2c4939cc3e66b726f70bc7c6f95349a3b12e6d7ab41485d9b9244f583df4b6d6" exitCode=0 Sep 11 04:12:16.049648 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:16.049596 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq" Sep 11 04:12:16.049937 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:16.049643 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq" event={"ID":"c24c91df-de1a-4a81-a096-8323f832d3d4","Type":"ContainerDied","Data":"2c4939cc3e66b726f70bc7c6f95349a3b12e6d7ab41485d9b9244f583df4b6d6"} Sep 11 04:12:16.049937 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:16.049693 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq" event={"ID":"c24c91df-de1a-4a81-a096-8323f832d3d4","Type":"ContainerDied","Data":"0db7b07f44387704091d314e2613d2b6e62b9c68cb6accb85608e4d7e20071fd"} Sep 11 04:12:16.049937 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:16.049720 2568 scope.go:117] "RemoveContainer" containerID="2c4939cc3e66b726f70bc7c6f95349a3b12e6d7ab41485d9b9244f583df4b6d6" Sep 11 04:12:16.058456 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:16.058441 2568 scope.go:117] "RemoveContainer" containerID="2c4939cc3e66b726f70bc7c6f95349a3b12e6d7ab41485d9b9244f583df4b6d6" Sep 11 04:12:16.058697 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:12:16.058681 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c4939cc3e66b726f70bc7c6f95349a3b12e6d7ab41485d9b9244f583df4b6d6\": container with ID starting with 2c4939cc3e66b726f70bc7c6f95349a3b12e6d7ab41485d9b9244f583df4b6d6 not found: ID does not exist" containerID="2c4939cc3e66b726f70bc7c6f95349a3b12e6d7ab41485d9b9244f583df4b6d6" Sep 11 04:12:16.058745 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:16.058703 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c4939cc3e66b726f70bc7c6f95349a3b12e6d7ab41485d9b9244f583df4b6d6"} err="failed to get container status \"2c4939cc3e66b726f70bc7c6f95349a3b12e6d7ab41485d9b9244f583df4b6d6\": rpc error: code = NotFound desc = could not find container \"2c4939cc3e66b726f70bc7c6f95349a3b12e6d7ab41485d9b9244f583df4b6d6\": container with ID starting with 2c4939cc3e66b726f70bc7c6f95349a3b12e6d7ab41485d9b9244f583df4b6d6 not found: ID does not exist" Sep 11 04:12:16.067526 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:16.067505 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq"] Sep 11 04:12:16.075218 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:16.075199 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-9cca98-openshift-default-564bf4ccd-2s6xq"] Sep 11 04:12:16.648165 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:16.648130 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:17.550576 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:17.550540 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:17.634397 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:17.634366 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:17.768692 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:17.768658 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c24c91df-de1a-4a81-a096-8323f832d3d4" path="/var/lib/kubelet/pods/c24c91df-de1a-4a81-a096-8323f832d3d4/volumes" Sep 11 04:12:17.807930 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:17.807845 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt"] Sep 11 04:12:17.808090 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:17.808064 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" podUID="7e0046d7-abc0-4be9-857d-b1e9d93baeee" containerName="istio-proxy" containerID="cri-o://6d0170a66dcd7ebb2cabe82421c3aa1c6cb5bf324e27babfe301eac68a6ac24c" gracePeriod=30 Sep 11 04:12:17.877452 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:17.877413 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:18.384741 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:18.384702 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:18.494928 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:18.494888 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:18.678749 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:18.678713 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w"] Sep 11 04:12:18.679322 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:18.679305 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="136cff62-ad79-4976-81dd-ac786e4d4cec" containerName="istio-proxy" Sep 11 04:12:18.679385 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:18.679326 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="136cff62-ad79-4976-81dd-ac786e4d4cec" containerName="istio-proxy" Sep 11 04:12:18.679385 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:18.679337 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="248091f3-1915-4236-8751-751ca75facfb" containerName="revoke-keys" Sep 11 04:12:18.679385 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:18.679346 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="248091f3-1915-4236-8751-751ca75facfb" containerName="revoke-keys" Sep 11 04:12:18.679385 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:18.679360 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c24c91df-de1a-4a81-a096-8323f832d3d4" containerName="istio-proxy" Sep 11 04:12:18.679385 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:18.679369 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c24c91df-de1a-4a81-a096-8323f832d3d4" containerName="istio-proxy" Sep 11 04:12:18.679385 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:18.679385 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="136cff62-ad79-4976-81dd-ac786e4d4cec" containerName="istio-proxy" Sep 11 04:12:18.679595 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:18.679394 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="136cff62-ad79-4976-81dd-ac786e4d4cec" containerName="istio-proxy" Sep 11 04:12:18.679595 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:18.679474 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="136cff62-ad79-4976-81dd-ac786e4d4cec" containerName="istio-proxy" Sep 11 04:12:18.679595 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:18.679485 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="136cff62-ad79-4976-81dd-ac786e4d4cec" containerName="istio-proxy" Sep 11 04:12:18.679595 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:18.679497 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="c24c91df-de1a-4a81-a096-8323f832d3d4" containerName="istio-proxy" Sep 11 04:12:18.679595 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:18.679506 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="248091f3-1915-4236-8751-751ca75facfb" containerName="revoke-keys" Sep 11 04:12:18.684289 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:18.684265 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" Sep 11 04:12:18.687117 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:18.687097 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-cc6708-openshift-default-dockercfg-7zbps\"" Sep 11 04:12:18.695362 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:18.695334 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w"] Sep 11 04:12:18.708820 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:18.708788 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:18.795546 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:18.795514 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8795982f-8f14-4d43-948c-f1966fc1be21-istio-envoy\") pod \"e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w\" (UID: \"8795982f-8f14-4d43-948c-f1966fc1be21\") " pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" Sep 11 04:12:18.795970 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:18.795552 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8795982f-8f14-4d43-948c-f1966fc1be21-istio-data\") pod \"e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w\" (UID: \"8795982f-8f14-4d43-948c-f1966fc1be21\") " pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" Sep 11 04:12:18.795970 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:18.795578 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8795982f-8f14-4d43-948c-f1966fc1be21-credential-socket\") pod \"e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w\" (UID: \"8795982f-8f14-4d43-948c-f1966fc1be21\") " pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" Sep 11 04:12:18.795970 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:18.795595 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8795982f-8f14-4d43-948c-f1966fc1be21-istio-podinfo\") pod \"e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w\" (UID: \"8795982f-8f14-4d43-948c-f1966fc1be21\") " pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" Sep 11 04:12:18.795970 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:18.795626 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8795982f-8f14-4d43-948c-f1966fc1be21-workload-certs\") pod \"e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w\" (UID: \"8795982f-8f14-4d43-948c-f1966fc1be21\") " pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" Sep 11 04:12:18.795970 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:18.795641 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8795982f-8f14-4d43-948c-f1966fc1be21-istiod-ca-cert\") pod \"e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w\" (UID: \"8795982f-8f14-4d43-948c-f1966fc1be21\") " pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" Sep 11 04:12:18.795970 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:18.795663 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8795982f-8f14-4d43-948c-f1966fc1be21-workload-socket\") pod \"e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w\" (UID: \"8795982f-8f14-4d43-948c-f1966fc1be21\") " pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" Sep 11 04:12:18.795970 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:18.795683 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8795982f-8f14-4d43-948c-f1966fc1be21-istio-token\") pod \"e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w\" (UID: \"8795982f-8f14-4d43-948c-f1966fc1be21\") " pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" Sep 11 04:12:18.795970 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:18.795806 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjnt5\" (UniqueName: \"kubernetes.io/projected/8795982f-8f14-4d43-948c-f1966fc1be21-kube-api-access-rjnt5\") pod \"e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w\" (UID: \"8795982f-8f14-4d43-948c-f1966fc1be21\") " pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" Sep 11 04:12:18.896842 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:18.896790 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8795982f-8f14-4d43-948c-f1966fc1be21-istio-token\") pod \"e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w\" (UID: \"8795982f-8f14-4d43-948c-f1966fc1be21\") " pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" Sep 11 04:12:18.897071 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:18.896854 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjnt5\" (UniqueName: \"kubernetes.io/projected/8795982f-8f14-4d43-948c-f1966fc1be21-kube-api-access-rjnt5\") pod \"e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w\" (UID: \"8795982f-8f14-4d43-948c-f1966fc1be21\") " pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" Sep 11 04:12:18.897071 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:18.896916 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8795982f-8f14-4d43-948c-f1966fc1be21-istio-envoy\") pod \"e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w\" (UID: \"8795982f-8f14-4d43-948c-f1966fc1be21\") " pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" Sep 11 04:12:18.897071 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:18.896949 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8795982f-8f14-4d43-948c-f1966fc1be21-istio-data\") pod \"e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w\" (UID: \"8795982f-8f14-4d43-948c-f1966fc1be21\") " pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" Sep 11 04:12:18.897071 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:18.896991 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8795982f-8f14-4d43-948c-f1966fc1be21-credential-socket\") pod \"e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w\" (UID: \"8795982f-8f14-4d43-948c-f1966fc1be21\") " pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" Sep 11 04:12:18.897071 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:18.897016 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8795982f-8f14-4d43-948c-f1966fc1be21-istio-podinfo\") pod \"e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w\" (UID: \"8795982f-8f14-4d43-948c-f1966fc1be21\") " pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" Sep 11 04:12:18.897343 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:18.897072 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8795982f-8f14-4d43-948c-f1966fc1be21-workload-certs\") pod \"e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w\" (UID: \"8795982f-8f14-4d43-948c-f1966fc1be21\") " pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" Sep 11 04:12:18.897343 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:18.897100 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8795982f-8f14-4d43-948c-f1966fc1be21-istiod-ca-cert\") pod \"e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w\" (UID: \"8795982f-8f14-4d43-948c-f1966fc1be21\") " pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" Sep 11 04:12:18.897343 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:18.897136 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8795982f-8f14-4d43-948c-f1966fc1be21-workload-socket\") pod \"e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w\" (UID: \"8795982f-8f14-4d43-948c-f1966fc1be21\") " pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" Sep 11 04:12:18.897498 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:18.897481 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8795982f-8f14-4d43-948c-f1966fc1be21-credential-socket\") pod \"e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w\" (UID: \"8795982f-8f14-4d43-948c-f1966fc1be21\") " pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" Sep 11 04:12:18.897556 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:18.897517 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8795982f-8f14-4d43-948c-f1966fc1be21-workload-socket\") pod \"e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w\" (UID: \"8795982f-8f14-4d43-948c-f1966fc1be21\") " pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" Sep 11 04:12:18.897652 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:18.897630 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8795982f-8f14-4d43-948c-f1966fc1be21-workload-certs\") pod \"e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w\" (UID: \"8795982f-8f14-4d43-948c-f1966fc1be21\") " pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" Sep 11 04:12:18.897742 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:18.897720 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8795982f-8f14-4d43-948c-f1966fc1be21-istio-data\") pod \"e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w\" (UID: \"8795982f-8f14-4d43-948c-f1966fc1be21\") " pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" Sep 11 04:12:18.897928 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:18.897908 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8795982f-8f14-4d43-948c-f1966fc1be21-istiod-ca-cert\") pod \"e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w\" (UID: \"8795982f-8f14-4d43-948c-f1966fc1be21\") " pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" Sep 11 04:12:18.899376 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:18.899354 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8795982f-8f14-4d43-948c-f1966fc1be21-istio-envoy\") pod \"e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w\" (UID: \"8795982f-8f14-4d43-948c-f1966fc1be21\") " pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" Sep 11 04:12:18.899656 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:18.899636 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8795982f-8f14-4d43-948c-f1966fc1be21-istio-podinfo\") pod \"e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w\" (UID: \"8795982f-8f14-4d43-948c-f1966fc1be21\") " pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" Sep 11 04:12:18.904195 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:18.904165 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8795982f-8f14-4d43-948c-f1966fc1be21-istio-token\") pod \"e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w\" (UID: \"8795982f-8f14-4d43-948c-f1966fc1be21\") " pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" Sep 11 04:12:18.904480 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:18.904460 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjnt5\" (UniqueName: \"kubernetes.io/projected/8795982f-8f14-4d43-948c-f1966fc1be21-kube-api-access-rjnt5\") pod \"e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w\" (UID: \"8795982f-8f14-4d43-948c-f1966fc1be21\") " pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" Sep 11 04:12:18.986460 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:18.986387 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:18.995223 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:18.995189 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" Sep 11 04:12:19.160358 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:19.160266 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:19.253824 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:19.253674 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:19.282877 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:19.282852 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w"] Sep 11 04:12:19.284945 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:12:19.284918 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8795982f_8f14_4d43_948c_f1966fc1be21.slice/crio-f19c98cba04478039a824e1872105fe0bb58e8069b43ab28998cbb1d68624512 WatchSource:0}: Error finding container f19c98cba04478039a824e1872105fe0bb58e8069b43ab28998cbb1d68624512: Status 404 returned error can't find the container with id f19c98cba04478039a824e1872105fe0bb58e8069b43ab28998cbb1d68624512 Sep 11 04:12:19.286888 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:19.286857 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 04:12:19.286961 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:19.286934 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 04:12:19.286996 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:19.286963 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 04:12:19.599558 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:19.599461 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:20.073516 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:20.073476 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" event={"ID":"8795982f-8f14-4d43-948c-f1966fc1be21","Type":"ContainerStarted","Data":"bd3f2e2a4696ac80c356e3dd8bb01319919b012d3302dcb41c50719c8cc9742c"} Sep 11 04:12:20.073894 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:20.073536 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" event={"ID":"8795982f-8f14-4d43-948c-f1966fc1be21","Type":"ContainerStarted","Data":"f19c98cba04478039a824e1872105fe0bb58e8069b43ab28998cbb1d68624512"} Sep 11 04:12:20.093307 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:20.093257 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" podStartSLOduration=2.093243525 podStartE2EDuration="2.093243525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:12:20.092168139 +0000 UTC m=+1154.845799543" watchObservedRunningTime="2026-09-11 04:12:20.093243525 +0000 UTC m=+1154.846874927" Sep 11 04:12:20.995571 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:20.995540 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" Sep 11 04:12:21.000517 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:21.000492 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" Sep 11 04:12:21.078731 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:21.078700 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" Sep 11 04:12:21.079760 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:21.079736 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" Sep 11 04:12:22.010077 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:22.010000 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:22.107335 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:22.107291 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:23.059938 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.059914 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" Sep 11 04:12:23.091300 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.091227 2568 generic.go:356] "Generic (PLEG): container finished" podID="7e0046d7-abc0-4be9-857d-b1e9d93baeee" containerID="6d0170a66dcd7ebb2cabe82421c3aa1c6cb5bf324e27babfe301eac68a6ac24c" exitCode=0 Sep 11 04:12:23.091300 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.091272 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" Sep 11 04:12:23.091485 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.091303 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" event={"ID":"7e0046d7-abc0-4be9-857d-b1e9d93baeee","Type":"ContainerDied","Data":"6d0170a66dcd7ebb2cabe82421c3aa1c6cb5bf324e27babfe301eac68a6ac24c"} Sep 11 04:12:23.091485 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.091365 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt" event={"ID":"7e0046d7-abc0-4be9-857d-b1e9d93baeee","Type":"ContainerDied","Data":"fd3232e5a27182f2d2d637db7dcf59768cf979c489cbca0acb39025140a7d58f"} Sep 11 04:12:23.091485 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.091399 2568 scope.go:117] "RemoveContainer" containerID="6d0170a66dcd7ebb2cabe82421c3aa1c6cb5bf324e27babfe301eac68a6ac24c" Sep 11 04:12:23.100655 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.100634 2568 scope.go:117] "RemoveContainer" containerID="ae092688161795ed7c50a17cd3ef8cb77b52281e9a945e1a8859d44966b068e0" Sep 11 04:12:23.108770 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.108751 2568 scope.go:117] "RemoveContainer" containerID="6d0170a66dcd7ebb2cabe82421c3aa1c6cb5bf324e27babfe301eac68a6ac24c" Sep 11 04:12:23.109121 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:12:23.109042 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d0170a66dcd7ebb2cabe82421c3aa1c6cb5bf324e27babfe301eac68a6ac24c\": container with ID starting with 6d0170a66dcd7ebb2cabe82421c3aa1c6cb5bf324e27babfe301eac68a6ac24c not found: ID does not exist" containerID="6d0170a66dcd7ebb2cabe82421c3aa1c6cb5bf324e27babfe301eac68a6ac24c" Sep 11 04:12:23.109121 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.109081 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d0170a66dcd7ebb2cabe82421c3aa1c6cb5bf324e27babfe301eac68a6ac24c"} err="failed to get container status \"6d0170a66dcd7ebb2cabe82421c3aa1c6cb5bf324e27babfe301eac68a6ac24c\": rpc error: code = NotFound desc = could not find container \"6d0170a66dcd7ebb2cabe82421c3aa1c6cb5bf324e27babfe301eac68a6ac24c\": container with ID starting with 6d0170a66dcd7ebb2cabe82421c3aa1c6cb5bf324e27babfe301eac68a6ac24c not found: ID does not exist" Sep 11 04:12:23.109121 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.109101 2568 scope.go:117] "RemoveContainer" containerID="ae092688161795ed7c50a17cd3ef8cb77b52281e9a945e1a8859d44966b068e0" Sep 11 04:12:23.109438 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:12:23.109420 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae092688161795ed7c50a17cd3ef8cb77b52281e9a945e1a8859d44966b068e0\": container with ID starting with ae092688161795ed7c50a17cd3ef8cb77b52281e9a945e1a8859d44966b068e0 not found: ID does not exist" containerID="ae092688161795ed7c50a17cd3ef8cb77b52281e9a945e1a8859d44966b068e0" Sep 11 04:12:23.109476 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.109444 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae092688161795ed7c50a17cd3ef8cb77b52281e9a945e1a8859d44966b068e0"} err="failed to get container status \"ae092688161795ed7c50a17cd3ef8cb77b52281e9a945e1a8859d44966b068e0\": rpc error: code = NotFound desc = could not find container \"ae092688161795ed7c50a17cd3ef8cb77b52281e9a945e1a8859d44966b068e0\": container with ID starting with ae092688161795ed7c50a17cd3ef8cb77b52281e9a945e1a8859d44966b068e0 not found: ID does not exist" Sep 11 04:12:23.133799 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.133760 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cc6708-56f88d9f5c-mjgd9"] Sep 11 04:12:23.133954 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.133909 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e0046d7-abc0-4be9-857d-b1e9d93baeee-istiod-ca-cert\") pod \"7e0046d7-abc0-4be9-857d-b1e9d93baeee\" (UID: \"7e0046d7-abc0-4be9-857d-b1e9d93baeee\") " Sep 11 04:12:23.133954 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.133947 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e0046d7-abc0-4be9-857d-b1e9d93baeee-istio-podinfo\") pod \"7e0046d7-abc0-4be9-857d-b1e9d93baeee\" (UID: \"7e0046d7-abc0-4be9-857d-b1e9d93baeee\") " Sep 11 04:12:23.134115 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.133978 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e0046d7-abc0-4be9-857d-b1e9d93baeee-istio-data\") pod \"7e0046d7-abc0-4be9-857d-b1e9d93baeee\" (UID: \"7e0046d7-abc0-4be9-857d-b1e9d93baeee\") " Sep 11 04:12:23.134115 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.134037 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0046d7-abc0-4be9-857d-b1e9d93baeee-credential-socket\") pod \"7e0046d7-abc0-4be9-857d-b1e9d93baeee\" (UID: \"7e0046d7-abc0-4be9-857d-b1e9d93baeee\") " Sep 11 04:12:23.134115 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.134090 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52pbs\" (UniqueName: \"kubernetes.io/projected/7e0046d7-abc0-4be9-857d-b1e9d93baeee-kube-api-access-52pbs\") pod \"7e0046d7-abc0-4be9-857d-b1e9d93baeee\" (UID: \"7e0046d7-abc0-4be9-857d-b1e9d93baeee\") " Sep 11 04:12:23.134287 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.134119 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0046d7-abc0-4be9-857d-b1e9d93baeee-workload-socket\") pod \"7e0046d7-abc0-4be9-857d-b1e9d93baeee\" (UID: \"7e0046d7-abc0-4be9-857d-b1e9d93baeee\") " Sep 11 04:12:23.134287 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.134146 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e0046d7-abc0-4be9-857d-b1e9d93baeee-istio-token\") pod \"7e0046d7-abc0-4be9-857d-b1e9d93baeee\" (UID: \"7e0046d7-abc0-4be9-857d-b1e9d93baeee\") " Sep 11 04:12:23.134287 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.134179 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e0046d7-abc0-4be9-857d-b1e9d93baeee-istio-envoy\") pod \"7e0046d7-abc0-4be9-857d-b1e9d93baeee\" (UID: \"7e0046d7-abc0-4be9-857d-b1e9d93baeee\") " Sep 11 04:12:23.134287 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.134218 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e0046d7-abc0-4be9-857d-b1e9d93baeee-workload-certs\") pod \"7e0046d7-abc0-4be9-857d-b1e9d93baeee\" (UID: \"7e0046d7-abc0-4be9-857d-b1e9d93baeee\") " Sep 11 04:12:23.134501 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.134395 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e0046d7-abc0-4be9-857d-b1e9d93baeee-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7e0046d7-abc0-4be9-857d-b1e9d93baeee" (UID: "7e0046d7-abc0-4be9-857d-b1e9d93baeee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 04:12:23.134501 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.134406 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0046d7-abc0-4be9-857d-b1e9d93baeee-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7e0046d7-abc0-4be9-857d-b1e9d93baeee" (UID: "7e0046d7-abc0-4be9-857d-b1e9d93baeee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:12:23.134501 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.134220 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e0046d7-abc0-4be9-857d-b1e9d93baeee" containerName="istio-proxy" Sep 11 04:12:23.134942 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.134515 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e0046d7-abc0-4be9-857d-b1e9d93baeee" containerName="istio-proxy" Sep 11 04:12:23.134942 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.134590 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0046d7-abc0-4be9-857d-b1e9d93baeee-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7e0046d7-abc0-4be9-857d-b1e9d93baeee" (UID: "7e0046d7-abc0-4be9-857d-b1e9d93baeee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:12:23.134942 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.134677 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e0046d7-abc0-4be9-857d-b1e9d93baeee" containerName="istio-proxy" Sep 11 04:12:23.134942 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.134693 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e0046d7-abc0-4be9-857d-b1e9d93baeee" containerName="istio-proxy" Sep 11 04:12:23.134942 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.134803 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0046d7-abc0-4be9-857d-b1e9d93baeee-workload-socket\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:12:23.134942 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.134838 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e0046d7-abc0-4be9-857d-b1e9d93baeee-workload-certs\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:12:23.134942 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.134943 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e0046d7-abc0-4be9-857d-b1e9d93baeee-istiod-ca-cert\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:12:23.135957 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.135918 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0046d7-abc0-4be9-857d-b1e9d93baeee-istio-data" (OuterVolumeSpecName: "istio-data") pod "7e0046d7-abc0-4be9-857d-b1e9d93baeee" (UID: "7e0046d7-abc0-4be9-857d-b1e9d93baeee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:12:23.136140 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.135989 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0046d7-abc0-4be9-857d-b1e9d93baeee-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7e0046d7-abc0-4be9-857d-b1e9d93baeee" (UID: "7e0046d7-abc0-4be9-857d-b1e9d93baeee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:12:23.137151 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.137128 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e0046d7-abc0-4be9-857d-b1e9d93baeee-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7e0046d7-abc0-4be9-857d-b1e9d93baeee" (UID: "7e0046d7-abc0-4be9-857d-b1e9d93baeee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:12:23.137232 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.137156 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e0046d7-abc0-4be9-857d-b1e9d93baeee-kube-api-access-52pbs" (OuterVolumeSpecName: "kube-api-access-52pbs") pod "7e0046d7-abc0-4be9-857d-b1e9d93baeee" (UID: "7e0046d7-abc0-4be9-857d-b1e9d93baeee"). InnerVolumeSpecName "kube-api-access-52pbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:12:23.138461 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.138430 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e0046d7-abc0-4be9-857d-b1e9d93baeee-istio-token" (OuterVolumeSpecName: "istio-token") pod "7e0046d7-abc0-4be9-857d-b1e9d93baeee" (UID: "7e0046d7-abc0-4be9-857d-b1e9d93baeee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:12:23.138801 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.138779 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e0046d7-abc0-4be9-857d-b1e9d93baeee-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7e0046d7-abc0-4be9-857d-b1e9d93baeee" (UID: "7e0046d7-abc0-4be9-857d-b1e9d93baeee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 04:12:23.141666 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.141255 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cc6708-56f88d9f5c-mjgd9" Sep 11 04:12:23.143556 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.143532 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-cc6708-serving-cert\"" Sep 11 04:12:23.143844 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.143795 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 04:12:23.143997 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.143889 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 11 04:12:23.144400 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.144129 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 11 04:12:23.148113 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.146570 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-b8w8q\"" Sep 11 04:12:23.148113 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.148033 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cc6708-56f88d9f5c-mjgd9"] Sep 11 04:12:23.187189 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.187160 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:23.235806 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.235771 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/957ae4ff-94c3-4f2c-b224-852d4e742bfc-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-cc6708-56f88d9f5c-mjgd9\" (UID: \"957ae4ff-94c3-4f2c-b224-852d4e742bfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cc6708-56f88d9f5c-mjgd9" Sep 11 04:12:23.235980 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.235807 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/957ae4ff-94c3-4f2c-b224-852d4e742bfc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-cc6708-56f88d9f5c-mjgd9\" (UID: \"957ae4ff-94c3-4f2c-b224-852d4e742bfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cc6708-56f88d9f5c-mjgd9" Sep 11 04:12:23.235980 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.235852 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbbjf\" (UniqueName: \"kubernetes.io/projected/957ae4ff-94c3-4f2c-b224-852d4e742bfc-kube-api-access-lbbjf\") pod \"maas-api-e2e-shared-a-w3-cc6708-56f88d9f5c-mjgd9\" (UID: \"957ae4ff-94c3-4f2c-b224-852d4e742bfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cc6708-56f88d9f5c-mjgd9" Sep 11 04:12:23.235980 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.235905 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/957ae4ff-94c3-4f2c-b224-852d4e742bfc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-cc6708-56f88d9f5c-mjgd9\" (UID: \"957ae4ff-94c3-4f2c-b224-852d4e742bfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cc6708-56f88d9f5c-mjgd9" Sep 11 04:12:23.236114 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.236018 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e0046d7-abc0-4be9-857d-b1e9d93baeee-istio-podinfo\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:12:23.236114 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.236071 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e0046d7-abc0-4be9-857d-b1e9d93baeee-istio-data\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:12:23.236114 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.236086 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e0046d7-abc0-4be9-857d-b1e9d93baeee-credential-socket\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:12:23.236114 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.236100 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52pbs\" (UniqueName: \"kubernetes.io/projected/7e0046d7-abc0-4be9-857d-b1e9d93baeee-kube-api-access-52pbs\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:12:23.236244 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.236114 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e0046d7-abc0-4be9-857d-b1e9d93baeee-istio-token\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:12:23.236244 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.236128 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e0046d7-abc0-4be9-857d-b1e9d93baeee-istio-envoy\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:12:23.287478 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.287445 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:23.337361 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.337331 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbbjf\" (UniqueName: \"kubernetes.io/projected/957ae4ff-94c3-4f2c-b224-852d4e742bfc-kube-api-access-lbbjf\") pod \"maas-api-e2e-shared-a-w3-cc6708-56f88d9f5c-mjgd9\" (UID: \"957ae4ff-94c3-4f2c-b224-852d4e742bfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cc6708-56f88d9f5c-mjgd9" Sep 11 04:12:23.337524 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.337392 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/957ae4ff-94c3-4f2c-b224-852d4e742bfc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-cc6708-56f88d9f5c-mjgd9\" (UID: \"957ae4ff-94c3-4f2c-b224-852d4e742bfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cc6708-56f88d9f5c-mjgd9" Sep 11 04:12:23.337524 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.337427 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/957ae4ff-94c3-4f2c-b224-852d4e742bfc-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-cc6708-56f88d9f5c-mjgd9\" (UID: \"957ae4ff-94c3-4f2c-b224-852d4e742bfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cc6708-56f88d9f5c-mjgd9" Sep 11 04:12:23.337524 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.337445 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/957ae4ff-94c3-4f2c-b224-852d4e742bfc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-cc6708-56f88d9f5c-mjgd9\" (UID: \"957ae4ff-94c3-4f2c-b224-852d4e742bfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cc6708-56f88d9f5c-mjgd9" Sep 11 04:12:23.338116 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.338088 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/957ae4ff-94c3-4f2c-b224-852d4e742bfc-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-cc6708-56f88d9f5c-mjgd9\" (UID: \"957ae4ff-94c3-4f2c-b224-852d4e742bfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cc6708-56f88d9f5c-mjgd9" Sep 11 04:12:23.339983 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.339963 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/957ae4ff-94c3-4f2c-b224-852d4e742bfc-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-cc6708-56f88d9f5c-mjgd9\" (UID: \"957ae4ff-94c3-4f2c-b224-852d4e742bfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cc6708-56f88d9f5c-mjgd9" Sep 11 04:12:23.340107 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.340020 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/957ae4ff-94c3-4f2c-b224-852d4e742bfc-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-cc6708-56f88d9f5c-mjgd9\" (UID: \"957ae4ff-94c3-4f2c-b224-852d4e742bfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cc6708-56f88d9f5c-mjgd9" Sep 11 04:12:23.345927 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.345869 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbbjf\" (UniqueName: \"kubernetes.io/projected/957ae4ff-94c3-4f2c-b224-852d4e742bfc-kube-api-access-lbbjf\") pod \"maas-api-e2e-shared-a-w3-cc6708-56f88d9f5c-mjgd9\" (UID: \"957ae4ff-94c3-4f2c-b224-852d4e742bfc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cc6708-56f88d9f5c-mjgd9" Sep 11 04:12:23.415704 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.415678 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt"] Sep 11 04:12:23.420669 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.420646 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-fca08f-openshift-default-55695f6bd8-j49nt"] Sep 11 04:12:23.469083 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.469055 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cc6708-56f88d9f5c-mjgd9" Sep 11 04:12:23.746731 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.746683 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cc6708-56f88d9f5c-mjgd9"] Sep 11 04:12:23.750554 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:12:23.750526 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod957ae4ff_94c3_4f2c_b224_852d4e742bfc.slice/crio-602e7f2d76de1510be1e1f66c16f57da3b24daf1181fc72c70f56fc8367c8ab7 WatchSource:0}: Error finding container 602e7f2d76de1510be1e1f66c16f57da3b24daf1181fc72c70f56fc8367c8ab7: Status 404 returned error can't find the container with id 602e7f2d76de1510be1e1f66c16f57da3b24daf1181fc72c70f56fc8367c8ab7 Sep 11 04:12:23.752098 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.752036 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 04:12:23.769030 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.769002 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e0046d7-abc0-4be9-857d-b1e9d93baeee" path="/var/lib/kubelet/pods/7e0046d7-abc0-4be9-857d-b1e9d93baeee/volumes" Sep 11 04:12:23.886486 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.886445 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:23.994309 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:23.994278 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:24.098663 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:24.098572 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cc6708-56f88d9f5c-mjgd9" event={"ID":"957ae4ff-94c3-4f2c-b224-852d4e742bfc","Type":"ContainerStarted","Data":"602e7f2d76de1510be1e1f66c16f57da3b24daf1181fc72c70f56fc8367c8ab7"} Sep 11 04:12:24.106072 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:24.106035 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cc6708-7679d85c98-nz6rx"] Sep 11 04:12:24.106410 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:24.106398 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e0046d7-abc0-4be9-857d-b1e9d93baeee" containerName="istio-proxy" Sep 11 04:12:24.106464 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:24.106412 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e0046d7-abc0-4be9-857d-b1e9d93baeee" containerName="istio-proxy" Sep 11 04:12:24.110674 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:24.110656 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cc6708-7679d85c98-nz6rx" Sep 11 04:12:24.113654 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:24.113633 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-cc6708-dockercfg-fkj92\"" Sep 11 04:12:24.113798 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:24.113655 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-cc6708\"" Sep 11 04:12:24.119612 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:24.119589 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cc6708-7679d85c98-nz6rx"] Sep 11 04:12:24.249163 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:24.249091 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f09610e1-0ccd-4a16-8c9b-b670eaecb1e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-cc6708-7679d85c98-nz6rx\" (UID: \"f09610e1-0ccd-4a16-8c9b-b670eaecb1e4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cc6708-7679d85c98-nz6rx" Sep 11 04:12:24.249359 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:24.249203 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42lfc\" (UniqueName: \"kubernetes.io/projected/f09610e1-0ccd-4a16-8c9b-b670eaecb1e4-kube-api-access-42lfc\") pod \"payload-pre-processing-e2e-shared-a-w3-cc6708-7679d85c98-nz6rx\" (UID: \"f09610e1-0ccd-4a16-8c9b-b670eaecb1e4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cc6708-7679d85c98-nz6rx" Sep 11 04:12:24.350762 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:24.350673 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f09610e1-0ccd-4a16-8c9b-b670eaecb1e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-cc6708-7679d85c98-nz6rx\" (UID: \"f09610e1-0ccd-4a16-8c9b-b670eaecb1e4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cc6708-7679d85c98-nz6rx" Sep 11 04:12:24.350928 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:24.350759 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42lfc\" (UniqueName: \"kubernetes.io/projected/f09610e1-0ccd-4a16-8c9b-b670eaecb1e4-kube-api-access-42lfc\") pod \"payload-pre-processing-e2e-shared-a-w3-cc6708-7679d85c98-nz6rx\" (UID: \"f09610e1-0ccd-4a16-8c9b-b670eaecb1e4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cc6708-7679d85c98-nz6rx" Sep 11 04:12:24.351661 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:24.351621 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f09610e1-0ccd-4a16-8c9b-b670eaecb1e4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-cc6708-7679d85c98-nz6rx\" (UID: \"f09610e1-0ccd-4a16-8c9b-b670eaecb1e4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cc6708-7679d85c98-nz6rx" Sep 11 04:12:24.359060 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:24.359018 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42lfc\" (UniqueName: \"kubernetes.io/projected/f09610e1-0ccd-4a16-8c9b-b670eaecb1e4-kube-api-access-42lfc\") pod \"payload-pre-processing-e2e-shared-a-w3-cc6708-7679d85c98-nz6rx\" (UID: \"f09610e1-0ccd-4a16-8c9b-b670eaecb1e4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cc6708-7679d85c98-nz6rx" Sep 11 04:12:24.423308 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:24.423261 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cc6708-7679d85c98-nz6rx" Sep 11 04:12:24.725645 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:24.725612 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cc6708-7679d85c98-nz6rx"] Sep 11 04:12:24.727696 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:12:24.727665 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf09610e1_0ccd_4a16_8c9b_b670eaecb1e4.slice/crio-11c5fe8a48643202afb8bf119320d39fdd52feaf3fef6ab3fad9b2834faf3f8b WatchSource:0}: Error finding container 11c5fe8a48643202afb8bf119320d39fdd52feaf3fef6ab3fad9b2834faf3f8b: Status 404 returned error can't find the container with id 11c5fe8a48643202afb8bf119320d39fdd52feaf3fef6ab3fad9b2834faf3f8b Sep 11 04:12:25.106393 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:25.106299 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cc6708-7679d85c98-nz6rx" event={"ID":"f09610e1-0ccd-4a16-8c9b-b670eaecb1e4","Type":"ContainerStarted","Data":"cc81f2115b68cf065a4dad0b8d38ec99a0c53287d5c3f2a58f5b1410b9813d51"} Sep 11 04:12:25.106393 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:25.106363 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cc6708-7679d85c98-nz6rx" event={"ID":"f09610e1-0ccd-4a16-8c9b-b670eaecb1e4","Type":"ContainerStarted","Data":"11c5fe8a48643202afb8bf119320d39fdd52feaf3fef6ab3fad9b2834faf3f8b"} Sep 11 04:12:25.106591 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:25.106424 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cc6708-7679d85c98-nz6rx" Sep 11 04:12:25.137896 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:25.137839 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cc6708-7679d85c98-nz6rx" podStartSLOduration=1.13782363 podStartE2EDuration="1.13782363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:12:25.137466637 +0000 UTC m=+1159.891098038" watchObservedRunningTime="2026-09-11 04:12:25.13782363 +0000 UTC m=+1159.891455032" Sep 11 04:12:26.244967 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:26.244929 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:26.345476 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:26.345391 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:26.723694 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:26.723659 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:27.124826 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:27.124733 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cc6708-56f88d9f5c-mjgd9" event={"ID":"957ae4ff-94c3-4f2c-b224-852d4e742bfc","Type":"ContainerStarted","Data":"44ec23d777b901313243b30e472dc3a8fd9ccc168b2e955900b6fd5844f758b6"} Sep 11 04:12:27.125000 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:27.124958 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cc6708-56f88d9f5c-mjgd9" Sep 11 04:12:27.146510 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:27.146451 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cc6708-56f88d9f5c-mjgd9" podStartSLOduration=1.8522230880000001 podStartE2EDuration="4.146414742s" totalImagesPullingTime="2.294191654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:12:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:12:27.146247218 +0000 UTC m=+1161.899878621" watchObservedRunningTime="2026-09-11 04:12:27.146414742 +0000 UTC m=+1161.900046145" Sep 11 04:12:33.135915 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:33.135835 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cc6708-56f88d9f5c-mjgd9" Sep 11 04:12:36.117183 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:36.117147 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cc6708-7679d85c98-nz6rx" Sep 11 04:12:36.656117 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:36.656085 2568 patch_prober.go:28] interesting pod/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": read tcp 10.132.0.2:36392->10.132.0.56:15021: read: connection reset by peer" start-of-body= Sep 11 04:12:36.656269 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:36.656168 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" podUID="8795982f-8f14-4d43-948c-f1966fc1be21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": read tcp 10.132.0.2:36392->10.132.0.56:15021: read: connection reset by peer" Sep 11 04:12:37.186268 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:37.186235 2568 generic.go:356] "Generic (PLEG): container finished" podID="8795982f-8f14-4d43-948c-f1966fc1be21" containerID="bd3f2e2a4696ac80c356e3dd8bb01319919b012d3302dcb41c50719c8cc9742c" exitCode=0 Sep 11 04:12:37.186657 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:37.186313 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" event={"ID":"8795982f-8f14-4d43-948c-f1966fc1be21","Type":"ContainerDied","Data":"bd3f2e2a4696ac80c356e3dd8bb01319919b012d3302dcb41c50719c8cc9742c"} Sep 11 04:12:37.186731 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:37.186715 2568 scope.go:117] "RemoveContainer" containerID="bd3f2e2a4696ac80c356e3dd8bb01319919b012d3302dcb41c50719c8cc9742c" Sep 11 04:12:37.187824 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:37.187791 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 04:12:37.187894 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:37.187879 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 04:12:37.187931 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:37.187918 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 04:12:37.269303 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:37.269275 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:37.313269 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:37.313231 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-34523cde-85746cb8b-z54nv"] Sep 11 04:12:37.313498 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:37.313470 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-34523cde-85746cb8b-z54nv" podUID="d2280354-4bbe-429e-baea-2c5c34e4297b" containerName="payload-pre-processing" containerID="cri-o://d24f2d424cb2780e48c253c851ccda23cf901521dceb5da65c140c6315296f6b" gracePeriod=30 Sep 11 04:12:37.369727 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:37.369671 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:37.486025 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:37.485923 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:37.574643 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:37.574612 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:37.575145 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:37.575127 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-34523cde-85746cb8b-z54nv" Sep 11 04:12:37.668821 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:37.668793 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvfjw\" (UniqueName: \"kubernetes.io/projected/d2280354-4bbe-429e-baea-2c5c34e4297b-kube-api-access-hvfjw\") pod \"d2280354-4bbe-429e-baea-2c5c34e4297b\" (UID: \"d2280354-4bbe-429e-baea-2c5c34e4297b\") " Sep 11 04:12:37.668992 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:37.668915 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2280354-4bbe-429e-baea-2c5c34e4297b-plugins-config-volume\") pod \"d2280354-4bbe-429e-baea-2c5c34e4297b\" (UID: \"d2280354-4bbe-429e-baea-2c5c34e4297b\") " Sep 11 04:12:37.669346 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:37.669311 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2280354-4bbe-429e-baea-2c5c34e4297b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d2280354-4bbe-429e-baea-2c5c34e4297b" (UID: "d2280354-4bbe-429e-baea-2c5c34e4297b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 04:12:37.670875 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:37.670847 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2280354-4bbe-429e-baea-2c5c34e4297b-kube-api-access-hvfjw" (OuterVolumeSpecName: "kube-api-access-hvfjw") pod "d2280354-4bbe-429e-baea-2c5c34e4297b" (UID: "d2280354-4bbe-429e-baea-2c5c34e4297b"). InnerVolumeSpecName "kube-api-access-hvfjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:12:37.769542 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:37.769459 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2280354-4bbe-429e-baea-2c5c34e4297b-plugins-config-volume\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:12:37.769542 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:37.769484 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvfjw\" (UniqueName: \"kubernetes.io/projected/d2280354-4bbe-429e-baea-2c5c34e4297b-kube-api-access-hvfjw\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:12:37.867623 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:37.867591 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:38.192022 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:38.191983 2568 generic.go:356] "Generic (PLEG): container finished" podID="d2280354-4bbe-429e-baea-2c5c34e4297b" containerID="d24f2d424cb2780e48c253c851ccda23cf901521dceb5da65c140c6315296f6b" exitCode=0 Sep 11 04:12:38.192516 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:38.192034 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-34523cde-85746cb8b-z54nv" Sep 11 04:12:38.192516 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:38.192070 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-34523cde-85746cb8b-z54nv" event={"ID":"d2280354-4bbe-429e-baea-2c5c34e4297b","Type":"ContainerDied","Data":"d24f2d424cb2780e48c253c851ccda23cf901521dceb5da65c140c6315296f6b"} Sep 11 04:12:38.192516 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:38.192126 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-34523cde-85746cb8b-z54nv" event={"ID":"d2280354-4bbe-429e-baea-2c5c34e4297b","Type":"ContainerDied","Data":"609ab2448776aedd50496a26b576aa042ffc668ad154b2833e319249b1d6f778"} Sep 11 04:12:38.192516 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:38.192148 2568 scope.go:117] "RemoveContainer" containerID="d24f2d424cb2780e48c253c851ccda23cf901521dceb5da65c140c6315296f6b" Sep 11 04:12:38.194622 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:38.194589 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" event={"ID":"8795982f-8f14-4d43-948c-f1966fc1be21","Type":"ContainerStarted","Data":"550342e0e85a719f2bd75ccb3fa0bf4c7ebde4f1862d93dee1a843bcb131def9"} Sep 11 04:12:38.200947 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:38.200930 2568 scope.go:117] "RemoveContainer" containerID="d24f2d424cb2780e48c253c851ccda23cf901521dceb5da65c140c6315296f6b" Sep 11 04:12:38.201206 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:12:38.201184 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d24f2d424cb2780e48c253c851ccda23cf901521dceb5da65c140c6315296f6b\": container with ID starting with d24f2d424cb2780e48c253c851ccda23cf901521dceb5da65c140c6315296f6b not found: ID does not exist" containerID="d24f2d424cb2780e48c253c851ccda23cf901521dceb5da65c140c6315296f6b" Sep 11 04:12:38.201273 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:38.201213 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d24f2d424cb2780e48c253c851ccda23cf901521dceb5da65c140c6315296f6b"} err="failed to get container status \"d24f2d424cb2780e48c253c851ccda23cf901521dceb5da65c140c6315296f6b\": rpc error: code = NotFound desc = could not find container \"d24f2d424cb2780e48c253c851ccda23cf901521dceb5da65c140c6315296f6b\": container with ID starting with d24f2d424cb2780e48c253c851ccda23cf901521dceb5da65c140c6315296f6b not found: ID does not exist" Sep 11 04:12:38.209830 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:38.209809 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-34523cde-85746cb8b-z54nv"] Sep 11 04:12:38.215705 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:38.215675 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-34523cde-85746cb8b-z54nv"] Sep 11 04:12:38.996003 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:38.995961 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" Sep 11 04:12:38.996246 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:38.996222 2568 patch_prober.go:28] interesting pod/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 11 04:12:38.996298 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:38.996276 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" podUID="8795982f-8f14-4d43-948c-f1966fc1be21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 11 04:12:39.769117 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:39.769079 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2280354-4bbe-429e-baea-2c5c34e4297b" path="/var/lib/kubelet/pods/d2280354-4bbe-429e-baea-2c5c34e4297b/volumes" Sep 11 04:12:39.995712 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:39.995668 2568 patch_prober.go:28] interesting pod/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 11 04:12:39.995939 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:39.995731 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" podUID="8795982f-8f14-4d43-948c-f1966fc1be21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 11 04:12:40.996593 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:40.996557 2568 patch_prober.go:28] interesting pod/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 11 04:12:40.996986 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:40.996620 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" podUID="8795982f-8f14-4d43-948c-f1966fc1be21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 11 04:12:41.996487 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:41.996448 2568 patch_prober.go:28] interesting pod/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 11 04:12:41.996744 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:41.996506 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" podUID="8795982f-8f14-4d43-948c-f1966fc1be21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 11 04:12:42.996379 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:42.996338 2568 patch_prober.go:28] interesting pod/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Sep 11 04:12:42.996553 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:42.996414 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" podUID="8795982f-8f14-4d43-948c-f1966fc1be21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Sep 11 04:12:43.999388 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:43.999355 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" Sep 11 04:12:43.999826 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:43.999691 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" Sep 11 04:12:44.000458 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:44.000436 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" Sep 11 04:12:44.314764 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:44.314665 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:44.381408 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:44.381367 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:44.591186 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:44.591104 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:44.959096 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:44.959055 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:45.060013 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:45.059979 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:45.237731 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:45.237647 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:45.523578 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:45.523497 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:47.020830 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:47.020797 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:47.451746 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:47.451675 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:47.661819 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:47.661782 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj"] Sep 11 04:12:47.662349 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:47.662329 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2280354-4bbe-429e-baea-2c5c34e4297b" containerName="payload-pre-processing" Sep 11 04:12:47.662451 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:47.662351 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2280354-4bbe-429e-baea-2c5c34e4297b" containerName="payload-pre-processing" Sep 11 04:12:47.662451 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:47.662439 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2280354-4bbe-429e-baea-2c5c34e4297b" containerName="payload-pre-processing" Sep 11 04:12:47.671482 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:47.671448 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" Sep 11 04:12:47.674032 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:47.674010 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-9342ed-openshift-default-dockercfg-lcgc9\"" Sep 11 04:12:47.680064 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:47.680030 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj"] Sep 11 04:12:47.704601 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:47.704530 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:47.761820 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:47.761771 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eee5be4-2a65-4513-8c1e-47db274ca367-istio-podinfo\") pod \"e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj\" (UID: \"6eee5be4-2a65-4513-8c1e-47db274ca367\") " pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" Sep 11 04:12:47.762025 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:47.761831 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eee5be4-2a65-4513-8c1e-47db274ca367-istio-envoy\") pod \"e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj\" (UID: \"6eee5be4-2a65-4513-8c1e-47db274ca367\") " pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" Sep 11 04:12:47.762025 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:47.761879 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kppln\" (UniqueName: \"kubernetes.io/projected/6eee5be4-2a65-4513-8c1e-47db274ca367-kube-api-access-kppln\") pod \"e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj\" (UID: \"6eee5be4-2a65-4513-8c1e-47db274ca367\") " pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" Sep 11 04:12:47.762025 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:47.761924 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eee5be4-2a65-4513-8c1e-47db274ca367-istio-data\") pod \"e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj\" (UID: \"6eee5be4-2a65-4513-8c1e-47db274ca367\") " pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" Sep 11 04:12:47.762025 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:47.761950 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eee5be4-2a65-4513-8c1e-47db274ca367-istiod-ca-cert\") pod \"e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj\" (UID: \"6eee5be4-2a65-4513-8c1e-47db274ca367\") " pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" Sep 11 04:12:47.762025 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:47.762022 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eee5be4-2a65-4513-8c1e-47db274ca367-workload-socket\") pod \"e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj\" (UID: \"6eee5be4-2a65-4513-8c1e-47db274ca367\") " pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" Sep 11 04:12:47.762344 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:47.762076 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eee5be4-2a65-4513-8c1e-47db274ca367-credential-socket\") pod \"e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj\" (UID: \"6eee5be4-2a65-4513-8c1e-47db274ca367\") " pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" Sep 11 04:12:47.762344 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:47.762112 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eee5be4-2a65-4513-8c1e-47db274ca367-workload-certs\") pod \"e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj\" (UID: \"6eee5be4-2a65-4513-8c1e-47db274ca367\") " pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" Sep 11 04:12:47.762344 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:47.762144 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eee5be4-2a65-4513-8c1e-47db274ca367-istio-token\") pod \"e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj\" (UID: \"6eee5be4-2a65-4513-8c1e-47db274ca367\") " pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" Sep 11 04:12:47.862975 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:47.862938 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eee5be4-2a65-4513-8c1e-47db274ca367-istio-podinfo\") pod \"e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj\" (UID: \"6eee5be4-2a65-4513-8c1e-47db274ca367\") " pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" Sep 11 04:12:47.863208 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:47.863002 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eee5be4-2a65-4513-8c1e-47db274ca367-istio-envoy\") pod \"e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj\" (UID: \"6eee5be4-2a65-4513-8c1e-47db274ca367\") " pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" Sep 11 04:12:47.863208 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:47.863110 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kppln\" (UniqueName: \"kubernetes.io/projected/6eee5be4-2a65-4513-8c1e-47db274ca367-kube-api-access-kppln\") pod \"e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj\" (UID: \"6eee5be4-2a65-4513-8c1e-47db274ca367\") " pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" Sep 11 04:12:47.863208 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:47.863160 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eee5be4-2a65-4513-8c1e-47db274ca367-istio-data\") pod \"e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj\" (UID: \"6eee5be4-2a65-4513-8c1e-47db274ca367\") " pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" Sep 11 04:12:47.863208 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:47.863183 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eee5be4-2a65-4513-8c1e-47db274ca367-istiod-ca-cert\") pod \"e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj\" (UID: \"6eee5be4-2a65-4513-8c1e-47db274ca367\") " pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" Sep 11 04:12:47.863453 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:47.863348 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eee5be4-2a65-4513-8c1e-47db274ca367-workload-socket\") pod \"e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj\" (UID: \"6eee5be4-2a65-4513-8c1e-47db274ca367\") " pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" Sep 11 04:12:47.863453 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:47.863386 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eee5be4-2a65-4513-8c1e-47db274ca367-credential-socket\") pod \"e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj\" (UID: \"6eee5be4-2a65-4513-8c1e-47db274ca367\") " pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" Sep 11 04:12:47.863453 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:47.863417 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eee5be4-2a65-4513-8c1e-47db274ca367-workload-certs\") pod \"e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj\" (UID: \"6eee5be4-2a65-4513-8c1e-47db274ca367\") " pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" Sep 11 04:12:47.863453 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:47.863441 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eee5be4-2a65-4513-8c1e-47db274ca367-istio-token\") pod \"e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj\" (UID: \"6eee5be4-2a65-4513-8c1e-47db274ca367\") " pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" Sep 11 04:12:47.863957 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:47.863934 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eee5be4-2a65-4513-8c1e-47db274ca367-istio-data\") pod \"e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj\" (UID: \"6eee5be4-2a65-4513-8c1e-47db274ca367\") " pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" Sep 11 04:12:47.864093 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:47.864037 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eee5be4-2a65-4513-8c1e-47db274ca367-workload-socket\") pod \"e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj\" (UID: \"6eee5be4-2a65-4513-8c1e-47db274ca367\") " pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" Sep 11 04:12:47.864147 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:47.864115 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eee5be4-2a65-4513-8c1e-47db274ca367-credential-socket\") pod \"e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj\" (UID: \"6eee5be4-2a65-4513-8c1e-47db274ca367\") " pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" Sep 11 04:12:47.864352 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:47.864331 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eee5be4-2a65-4513-8c1e-47db274ca367-workload-certs\") pod \"e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj\" (UID: \"6eee5be4-2a65-4513-8c1e-47db274ca367\") " pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" Sep 11 04:12:47.864605 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:47.864585 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eee5be4-2a65-4513-8c1e-47db274ca367-istiod-ca-cert\") pod \"e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj\" (UID: \"6eee5be4-2a65-4513-8c1e-47db274ca367\") " pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" Sep 11 04:12:47.865679 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:47.865653 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eee5be4-2a65-4513-8c1e-47db274ca367-istio-envoy\") pod \"e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj\" (UID: \"6eee5be4-2a65-4513-8c1e-47db274ca367\") " pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" Sep 11 04:12:47.865797 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:47.865782 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eee5be4-2a65-4513-8c1e-47db274ca367-istio-podinfo\") pod \"e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj\" (UID: \"6eee5be4-2a65-4513-8c1e-47db274ca367\") " pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" Sep 11 04:12:47.871806 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:47.871778 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eee5be4-2a65-4513-8c1e-47db274ca367-istio-token\") pod \"e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj\" (UID: \"6eee5be4-2a65-4513-8c1e-47db274ca367\") " pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" Sep 11 04:12:47.871982 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:47.871963 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kppln\" (UniqueName: \"kubernetes.io/projected/6eee5be4-2a65-4513-8c1e-47db274ca367-kube-api-access-kppln\") pod \"e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj\" (UID: \"6eee5be4-2a65-4513-8c1e-47db274ca367\") " pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" Sep 11 04:12:47.897963 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:47.897927 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:47.984001 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:47.983927 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" Sep 11 04:12:48.263706 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:48.263674 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj"] Sep 11 04:12:48.265749 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:12:48.265721 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eee5be4_2a65_4513_8c1e_47db274ca367.slice/crio-00e674f5fc8c119c3f689b96a13eac1435306e909e06f47933aaac12cbd477f4 WatchSource:0}: Error finding container 00e674f5fc8c119c3f689b96a13eac1435306e909e06f47933aaac12cbd477f4: Status 404 returned error can't find the container with id 00e674f5fc8c119c3f689b96a13eac1435306e909e06f47933aaac12cbd477f4 Sep 11 04:12:48.267785 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:48.267753 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 04:12:48.267874 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:48.267816 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 04:12:48.267874 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:48.267843 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 04:12:49.252256 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:49.252215 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" event={"ID":"6eee5be4-2a65-4513-8c1e-47db274ca367","Type":"ContainerStarted","Data":"28b3f2b7dc4594fb91a56c8b3869cf01bc8cef34ce0eb47ab45af272d5dcf75d"} Sep 11 04:12:49.252441 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:49.252319 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" event={"ID":"6eee5be4-2a65-4513-8c1e-47db274ca367","Type":"ContainerStarted","Data":"00e674f5fc8c119c3f689b96a13eac1435306e909e06f47933aaac12cbd477f4"} Sep 11 04:12:49.273707 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:49.273651 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" podStartSLOduration=2.273626863 podStartE2EDuration="2.273626863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:12:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:12:49.27227192 +0000 UTC m=+1184.025903316" watchObservedRunningTime="2026-09-11 04:12:49.273626863 +0000 UTC m=+1184.027258265" Sep 11 04:12:49.984814 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:49.984769 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" Sep 11 04:12:49.989692 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:49.989668 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" Sep 11 04:12:50.261450 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:50.261361 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" Sep 11 04:12:50.262500 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:50.262480 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" Sep 11 04:12:51.016615 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:51.016585 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:51.079301 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:51.079268 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:51.176946 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:51.176908 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:51.282745 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:51.282651 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:51.379481 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:51.379445 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:52.533762 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:52.533725 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:52.646885 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:52.646848 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:52.752350 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:52.752311 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:53.030745 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:53.030707 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:53.136410 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:53.136377 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:53.869819 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:53.869777 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:57.630596 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:57.630562 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:57.706971 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:57.706934 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:58.587460 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:58.587426 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:58.916424 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:58.916380 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:12:59.016493 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:12:59.016460 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:13:01.860870 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:01.860838 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:13:01.945134 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:01.945086 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:13:03.231336 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:03.231302 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:13:03.334035 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:03.333997 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:13:03.764209 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:03.764154 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:13:03.936580 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:03.936545 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:13:04.038020 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:04.037925 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:13:04.226233 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:04.226197 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-772e41-c65794cd9-jjrdd"] Sep 11 04:13:04.229577 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:04.229559 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-772e41-c65794cd9-jjrdd" Sep 11 04:13:04.233315 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:04.233284 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-772e41\"" Sep 11 04:13:04.233693 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:04.233532 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-772e41-dockercfg-c94sd\"" Sep 11 04:13:04.244650 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:04.244617 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-772e41-c65794cd9-jjrdd"] Sep 11 04:13:04.307599 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:04.307500 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f5e76f9-b2ad-4676-b83c-a8c52adc5233-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-772e41-c65794cd9-jjrdd\" (UID: \"5f5e76f9-b2ad-4676-b83c-a8c52adc5233\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-772e41-c65794cd9-jjrdd" Sep 11 04:13:04.307599 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:04.307554 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6tzv\" (UniqueName: \"kubernetes.io/projected/5f5e76f9-b2ad-4676-b83c-a8c52adc5233-kube-api-access-w6tzv\") pod \"payload-processing-e2e-shared-a-w2-772e41-c65794cd9-jjrdd\" (UID: \"5f5e76f9-b2ad-4676-b83c-a8c52adc5233\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-772e41-c65794cd9-jjrdd" Sep 11 04:13:04.408695 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:04.408641 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f5e76f9-b2ad-4676-b83c-a8c52adc5233-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-772e41-c65794cd9-jjrdd\" (UID: \"5f5e76f9-b2ad-4676-b83c-a8c52adc5233\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-772e41-c65794cd9-jjrdd" Sep 11 04:13:04.408940 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:04.408728 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6tzv\" (UniqueName: \"kubernetes.io/projected/5f5e76f9-b2ad-4676-b83c-a8c52adc5233-kube-api-access-w6tzv\") pod \"payload-processing-e2e-shared-a-w2-772e41-c65794cd9-jjrdd\" (UID: \"5f5e76f9-b2ad-4676-b83c-a8c52adc5233\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-772e41-c65794cd9-jjrdd" Sep 11 04:13:04.409458 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:04.409433 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f5e76f9-b2ad-4676-b83c-a8c52adc5233-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-772e41-c65794cd9-jjrdd\" (UID: \"5f5e76f9-b2ad-4676-b83c-a8c52adc5233\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-772e41-c65794cd9-jjrdd" Sep 11 04:13:04.425652 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:04.425621 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6tzv\" (UniqueName: \"kubernetes.io/projected/5f5e76f9-b2ad-4676-b83c-a8c52adc5233-kube-api-access-w6tzv\") pod \"payload-processing-e2e-shared-a-w2-772e41-c65794cd9-jjrdd\" (UID: \"5f5e76f9-b2ad-4676-b83c-a8c52adc5233\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-772e41-c65794cd9-jjrdd" Sep 11 04:13:04.542365 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:04.542326 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-772e41-c65794cd9-jjrdd" Sep 11 04:13:04.841256 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:04.841216 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-772e41-c65794cd9-jjrdd"] Sep 11 04:13:04.844656 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:13:04.844588 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f5e76f9_b2ad_4676_b83c_a8c52adc5233.slice/crio-ae8631c4e1048284d3d112f1105fc22685be488a27b5848e44ceed154d2ee1f6 WatchSource:0}: Error finding container ae8631c4e1048284d3d112f1105fc22685be488a27b5848e44ceed154d2ee1f6: Status 404 returned error can't find the container with id ae8631c4e1048284d3d112f1105fc22685be488a27b5848e44ceed154d2ee1f6 Sep 11 04:13:05.339079 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:05.339027 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-772e41-c65794cd9-jjrdd" event={"ID":"5f5e76f9-b2ad-4676-b83c-a8c52adc5233","Type":"ContainerStarted","Data":"df536be10c52aa5045816109ad785700d673ab14f5518d70fc046c52631713ee"} Sep 11 04:13:05.339583 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:05.339103 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-772e41-c65794cd9-jjrdd" event={"ID":"5f5e76f9-b2ad-4676-b83c-a8c52adc5233","Type":"ContainerStarted","Data":"ae8631c4e1048284d3d112f1105fc22685be488a27b5848e44ceed154d2ee1f6"} Sep 11 04:13:05.339583 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:05.339225 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-772e41-c65794cd9-jjrdd" Sep 11 04:13:05.358747 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:05.358685 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-772e41-c65794cd9-jjrdd" podStartSLOduration=1.358663856 podStartE2EDuration="1.358663856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:13:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:13:05.356313809 +0000 UTC m=+1200.109945211" watchObservedRunningTime="2026-09-11 04:13:05.358663856 +0000 UTC m=+1200.112295241" Sep 11 04:13:05.784072 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:05.784022 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-5tpdc_9112557d-d0ef-44c7-8b2a-f6ecd0a980d6/console-operator/1.log" Sep 11 04:13:05.785841 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:05.785815 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-5tpdc_9112557d-d0ef-44c7-8b2a-f6ecd0a980d6/console-operator/1.log" Sep 11 04:13:06.303274 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:06.303229 2568 patch_prober.go:28] interesting pod/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 11 04:13:06.303488 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:06.303307 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" podUID="6eee5be4-2a65-4513-8c1e-47db274ca367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 04:13:07.351487 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:07.351454 2568 generic.go:356] "Generic (PLEG): container finished" podID="6eee5be4-2a65-4513-8c1e-47db274ca367" containerID="28b3f2b7dc4594fb91a56c8b3869cf01bc8cef34ce0eb47ab45af272d5dcf75d" exitCode=0 Sep 11 04:13:07.351955 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:07.351529 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" event={"ID":"6eee5be4-2a65-4513-8c1e-47db274ca367","Type":"ContainerDied","Data":"28b3f2b7dc4594fb91a56c8b3869cf01bc8cef34ce0eb47ab45af272d5dcf75d"} Sep 11 04:13:07.352030 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:07.351980 2568 scope.go:117] "RemoveContainer" containerID="28b3f2b7dc4594fb91a56c8b3869cf01bc8cef34ce0eb47ab45af272d5dcf75d" Sep 11 04:13:07.353091 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:07.353060 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 04:13:07.353155 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:07.353132 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 04:13:07.353193 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:07.353164 2568 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"} Sep 11 04:13:08.362037 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:08.361994 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" event={"ID":"6eee5be4-2a65-4513-8c1e-47db274ca367","Type":"ContainerStarted","Data":"df8c80348d8d9ee05a0f37a4dfcc7119d2d1e6e12ffd5caec7a11c32ef786302"} Sep 11 04:13:08.984328 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:08.984292 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" Sep 11 04:13:08.984566 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:08.984547 2568 patch_prober.go:28] interesting pod/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 11 04:13:08.984613 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:08.984593 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" podUID="6eee5be4-2a65-4513-8c1e-47db274ca367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 11 04:13:09.985228 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:09.985191 2568 patch_prober.go:28] interesting pod/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 11 04:13:09.985629 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:09.985246 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" podUID="6eee5be4-2a65-4513-8c1e-47db274ca367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 11 04:13:10.984423 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:10.984385 2568 patch_prober.go:28] interesting pod/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 11 04:13:10.984686 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:10.984464 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" podUID="6eee5be4-2a65-4513-8c1e-47db274ca367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 11 04:13:11.984594 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:11.984557 2568 patch_prober.go:28] interesting pod/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 11 04:13:11.984965 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:11.984612 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" podUID="6eee5be4-2a65-4513-8c1e-47db274ca367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 11 04:13:12.984586 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:12.984551 2568 patch_prober.go:28] interesting pod/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Sep 11 04:13:12.984850 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:12.984607 2568 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" podUID="6eee5be4-2a65-4513-8c1e-47db274ca367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Sep 11 04:13:13.576462 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:13.576425 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:13:13.675835 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:13.675796 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:13:13.987541 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:13.987507 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" Sep 11 04:13:13.987918 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:13.987840 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" Sep 11 04:13:13.988404 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:13.988388 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" Sep 11 04:13:16.197433 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:16.197394 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:13:16.345812 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:16.345770 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-772e41-c65794cd9-jjrdd" Sep 11 04:13:19.654714 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:19.654674 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:13:19.720201 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:19.720170 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:13:22.699333 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:22.699285 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:13:25.618248 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:25.618206 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:13:26.901883 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:26.901839 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:13:27.261328 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:27.261217 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:13:27.602323 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:27.602223 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:13:27.932000 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:27.931953 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:13:31.120732 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:31.120704 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:13:31.205965 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:31.205930 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:13:32.177129 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:32.177026 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8229c-5fbd746f56-lhh7v"] Sep 11 04:13:32.182639 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:32.182616 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8229c-5fbd746f56-lhh7v" Sep 11 04:13:32.185201 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:32.185179 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-f8229c-serving-cert\"" Sep 11 04:13:32.190963 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:32.190942 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8229c-5fbd746f56-lhh7v"] Sep 11 04:13:32.204966 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:32.204912 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:13:32.267721 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:32.267679 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfs25\" (UniqueName: \"kubernetes.io/projected/2f4c1059-02c1-42b5-aed3-052f39640577-kube-api-access-gfs25\") pod \"maas-api-e2e-shared-b-w2-f8229c-5fbd746f56-lhh7v\" (UID: \"2f4c1059-02c1-42b5-aed3-052f39640577\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8229c-5fbd746f56-lhh7v" Sep 11 04:13:32.267897 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:32.267727 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f4c1059-02c1-42b5-aed3-052f39640577-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-f8229c-5fbd746f56-lhh7v\" (UID: \"2f4c1059-02c1-42b5-aed3-052f39640577\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8229c-5fbd746f56-lhh7v" Sep 11 04:13:32.267897 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:32.267816 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f4c1059-02c1-42b5-aed3-052f39640577-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f8229c-5fbd746f56-lhh7v\" (UID: \"2f4c1059-02c1-42b5-aed3-052f39640577\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8229c-5fbd746f56-lhh7v" Sep 11 04:13:32.267897 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:32.267845 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f4c1059-02c1-42b5-aed3-052f39640577-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-f8229c-5fbd746f56-lhh7v\" (UID: \"2f4c1059-02c1-42b5-aed3-052f39640577\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8229c-5fbd746f56-lhh7v" Sep 11 04:13:32.269787 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:32.269740 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:13:32.368768 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:32.368739 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f4c1059-02c1-42b5-aed3-052f39640577-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-f8229c-5fbd746f56-lhh7v\" (UID: \"2f4c1059-02c1-42b5-aed3-052f39640577\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8229c-5fbd746f56-lhh7v" Sep 11 04:13:32.368946 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:32.368777 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfs25\" (UniqueName: \"kubernetes.io/projected/2f4c1059-02c1-42b5-aed3-052f39640577-kube-api-access-gfs25\") pod \"maas-api-e2e-shared-b-w2-f8229c-5fbd746f56-lhh7v\" (UID: \"2f4c1059-02c1-42b5-aed3-052f39640577\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8229c-5fbd746f56-lhh7v" Sep 11 04:13:32.368946 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:32.368921 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f4c1059-02c1-42b5-aed3-052f39640577-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-f8229c-5fbd746f56-lhh7v\" (UID: \"2f4c1059-02c1-42b5-aed3-052f39640577\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8229c-5fbd746f56-lhh7v" Sep 11 04:13:32.369131 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:32.369111 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f4c1059-02c1-42b5-aed3-052f39640577-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f8229c-5fbd746f56-lhh7v\" (UID: \"2f4c1059-02c1-42b5-aed3-052f39640577\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8229c-5fbd746f56-lhh7v" Sep 11 04:13:32.375296 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:32.375260 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:13:32.378087 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:32.378065 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfs25\" (UniqueName: \"kubernetes.io/projected/2f4c1059-02c1-42b5-aed3-052f39640577-kube-api-access-gfs25\") pod \"maas-api-e2e-shared-b-w2-f8229c-5fbd746f56-lhh7v\" (UID: \"2f4c1059-02c1-42b5-aed3-052f39640577\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8229c-5fbd746f56-lhh7v" Sep 11 04:13:32.378286 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:32.369653 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f4c1059-02c1-42b5-aed3-052f39640577-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-f8229c-5fbd746f56-lhh7v\" (UID: \"2f4c1059-02c1-42b5-aed3-052f39640577\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8229c-5fbd746f56-lhh7v" Sep 11 04:13:32.380040 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:32.379981 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f4c1059-02c1-42b5-aed3-052f39640577-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-f8229c-5fbd746f56-lhh7v\" (UID: \"2f4c1059-02c1-42b5-aed3-052f39640577\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8229c-5fbd746f56-lhh7v" Sep 11 04:13:32.382796 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:32.382774 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f4c1059-02c1-42b5-aed3-052f39640577-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-f8229c-5fbd746f56-lhh7v\" (UID: \"2f4c1059-02c1-42b5-aed3-052f39640577\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8229c-5fbd746f56-lhh7v" Sep 11 04:13:32.495950 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:32.495879 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8229c-5fbd746f56-lhh7v" Sep 11 04:13:32.733456 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:32.733420 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:13:32.781141 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:32.781032 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8229c-5fbd746f56-lhh7v"] Sep 11 04:13:32.784387 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:13:32.784345 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f4c1059_02c1_42b5_aed3_052f39640577.slice/crio-6b9996add55db1b9f6e77c3719d6fd3ea509afaf9e47f6c173987385b699cc72 WatchSource:0}: Error finding container 6b9996add55db1b9f6e77c3719d6fd3ea509afaf9e47f6c173987385b699cc72: Status 404 returned error can't find the container with id 6b9996add55db1b9f6e77c3719d6fd3ea509afaf9e47f6c173987385b699cc72 Sep 11 04:13:32.846308 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:32.846274 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:13:32.916494 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:32.916463 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8229c-77b67bb94b-krzmq"] Sep 11 04:13:32.921095 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:32.921076 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8229c-77b67bb94b-krzmq" Sep 11 04:13:32.923553 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:32.923533 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-f8229c-dockercfg-v8nbc\"" Sep 11 04:13:32.923663 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:32.923550 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-f8229c\"" Sep 11 04:13:32.931989 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:32.931966 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8229c-77b67bb94b-krzmq"] Sep 11 04:13:33.076684 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:33.076591 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dwbj\" (UniqueName: \"kubernetes.io/projected/37f9cd2a-fedc-494f-a25f-a85b04719a84-kube-api-access-4dwbj\") pod \"payload-pre-processing-e2e-shared-b-w2-f8229c-77b67bb94b-krzmq\" (UID: \"37f9cd2a-fedc-494f-a25f-a85b04719a84\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8229c-77b67bb94b-krzmq" Sep 11 04:13:33.076684 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:33.076633 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37f9cd2a-fedc-494f-a25f-a85b04719a84-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-f8229c-77b67bb94b-krzmq\" (UID: \"37f9cd2a-fedc-494f-a25f-a85b04719a84\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8229c-77b67bb94b-krzmq" Sep 11 04:13:33.177377 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:33.177350 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dwbj\" (UniqueName: \"kubernetes.io/projected/37f9cd2a-fedc-494f-a25f-a85b04719a84-kube-api-access-4dwbj\") pod \"payload-pre-processing-e2e-shared-b-w2-f8229c-77b67bb94b-krzmq\" (UID: \"37f9cd2a-fedc-494f-a25f-a85b04719a84\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8229c-77b67bb94b-krzmq" Sep 11 04:13:33.177716 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:33.177393 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37f9cd2a-fedc-494f-a25f-a85b04719a84-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-f8229c-77b67bb94b-krzmq\" (UID: \"37f9cd2a-fedc-494f-a25f-a85b04719a84\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8229c-77b67bb94b-krzmq" Sep 11 04:13:33.178436 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:33.178404 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37f9cd2a-fedc-494f-a25f-a85b04719a84-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-f8229c-77b67bb94b-krzmq\" (UID: \"37f9cd2a-fedc-494f-a25f-a85b04719a84\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8229c-77b67bb94b-krzmq" Sep 11 04:13:33.186272 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:33.186254 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dwbj\" (UniqueName: \"kubernetes.io/projected/37f9cd2a-fedc-494f-a25f-a85b04719a84-kube-api-access-4dwbj\") pod \"payload-pre-processing-e2e-shared-b-w2-f8229c-77b67bb94b-krzmq\" (UID: \"37f9cd2a-fedc-494f-a25f-a85b04719a84\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8229c-77b67bb94b-krzmq" Sep 11 04:13:33.233374 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:33.233344 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8229c-77b67bb94b-krzmq" Sep 11 04:13:33.489838 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:33.489803 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8229c-5fbd746f56-lhh7v" event={"ID":"2f4c1059-02c1-42b5-aed3-052f39640577","Type":"ContainerStarted","Data":"7a0ee62fce4370d03291cf330b6c4d3a365f95a75352ac09d04ab099ea11a4f8"} Sep 11 04:13:33.490012 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:33.489862 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8229c-5fbd746f56-lhh7v" event={"ID":"2f4c1059-02c1-42b5-aed3-052f39640577","Type":"ContainerStarted","Data":"6b9996add55db1b9f6e77c3719d6fd3ea509afaf9e47f6c173987385b699cc72"} Sep 11 04:13:33.490113 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:33.490013 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8229c-5fbd746f56-lhh7v" Sep 11 04:13:33.511238 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:33.511211 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8229c-77b67bb94b-krzmq"] Sep 11 04:13:33.511764 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:33.511719 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8229c-5fbd746f56-lhh7v" podStartSLOduration=1.161166074 podStartE2EDuration="1.511699956s" totalImagesPullingTime="350.533882ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:13:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:13:33.511101615 +0000 UTC m=+1228.264733018" watchObservedRunningTime="2026-09-11 04:13:33.511699956 +0000 UTC m=+1228.265331358" Sep 11 04:13:33.514003 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:13:33.513980 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37f9cd2a_fedc_494f_a25f_a85b04719a84.slice/crio-10148dd8af96f56c0b98e8048230914bc7ac6c551b126d1b3aebbc3ecfc6219d WatchSource:0}: Error finding container 10148dd8af96f56c0b98e8048230914bc7ac6c551b126d1b3aebbc3ecfc6219d: Status 404 returned error can't find the container with id 10148dd8af96f56c0b98e8048230914bc7ac6c551b126d1b3aebbc3ecfc6219d Sep 11 04:13:34.497017 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:34.496980 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8229c-77b67bb94b-krzmq" event={"ID":"37f9cd2a-fedc-494f-a25f-a85b04719a84","Type":"ContainerStarted","Data":"876620718c77ec7845193217f4b5f481d8307473fd32b21a74170cddca03b7e1"} Sep 11 04:13:34.497456 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:34.497166 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8229c-77b67bb94b-krzmq" Sep 11 04:13:34.497456 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:34.497189 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8229c-77b67bb94b-krzmq" event={"ID":"37f9cd2a-fedc-494f-a25f-a85b04719a84","Type":"ContainerStarted","Data":"10148dd8af96f56c0b98e8048230914bc7ac6c551b126d1b3aebbc3ecfc6219d"} Sep 11 04:13:34.516381 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:34.516281 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8229c-77b67bb94b-krzmq" podStartSLOduration=2.516261144 podStartE2EDuration="2.516261144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:13:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:13:34.515080938 +0000 UTC m=+1229.268712341" watchObservedRunningTime="2026-09-11 04:13:34.516261144 +0000 UTC m=+1229.269892547" Sep 11 04:13:39.502573 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:39.502544 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8229c-5fbd746f56-lhh7v" Sep 11 04:13:42.481041 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:42.480996 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:13:45.505837 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:45.505806 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8229c-77b67bb94b-krzmq" Sep 11 04:13:56.101219 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:56.101180 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:13:56.918875 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:56.918836 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8077-bc1790"] Sep 11 04:13:56.922573 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:56.922555 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8077-bc1790" Sep 11 04:13:56.925712 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:56.925688 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zqmz7\"" Sep 11 04:13:56.934798 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:56.934769 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8077-bc1790"] Sep 11 04:13:56.963399 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:56.963372 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prp7n\" (UniqueName: \"kubernetes.io/projected/2de1b9a8-7f4f-4e34-bb5d-34151603001e-kube-api-access-prp7n\") pod \"test-curl-8077-bc1790\" (UID: \"2de1b9a8-7f4f-4e34-bb5d-34151603001e\") " pod="opendatahub/test-curl-8077-bc1790" Sep 11 04:13:57.064374 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:57.064333 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prp7n\" (UniqueName: \"kubernetes.io/projected/2de1b9a8-7f4f-4e34-bb5d-34151603001e-kube-api-access-prp7n\") pod \"test-curl-8077-bc1790\" (UID: \"2de1b9a8-7f4f-4e34-bb5d-34151603001e\") " pod="opendatahub/test-curl-8077-bc1790" Sep 11 04:13:57.072414 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:57.072375 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prp7n\" (UniqueName: \"kubernetes.io/projected/2de1b9a8-7f4f-4e34-bb5d-34151603001e-kube-api-access-prp7n\") pod \"test-curl-8077-bc1790\" (UID: \"2de1b9a8-7f4f-4e34-bb5d-34151603001e\") " pod="opendatahub/test-curl-8077-bc1790" Sep 11 04:13:57.233494 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:57.233407 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8077-bc1790" Sep 11 04:13:57.504468 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:57.504440 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8077-bc1790"] Sep 11 04:13:57.506327 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:13:57.506293 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2de1b9a8_7f4f_4e34_bb5d_34151603001e.slice/crio-1e34374ca7a79021b298e0a3ba7a261f3b49cab4c2b461ce63195ed211586d97 WatchSource:0}: Error finding container 1e34374ca7a79021b298e0a3ba7a261f3b49cab4c2b461ce63195ed211586d97: Status 404 returned error can't find the container with id 1e34374ca7a79021b298e0a3ba7a261f3b49cab4c2b461ce63195ed211586d97 Sep 11 04:13:57.624100 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:57.624065 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8077-bc1790" event={"ID":"2de1b9a8-7f4f-4e34-bb5d-34151603001e","Type":"ContainerStarted","Data":"322f7cbf6880ca180f56f736f1259f4733594e16860773e0c6dc4b2a3786f560"} Sep 11 04:13:57.624100 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:57.624106 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8077-bc1790" event={"ID":"2de1b9a8-7f4f-4e34-bb5d-34151603001e","Type":"ContainerStarted","Data":"1e34374ca7a79021b298e0a3ba7a261f3b49cab4c2b461ce63195ed211586d97"} Sep 11 04:13:57.639276 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:57.639224 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8077-bc1790" podStartSLOduration=1.639210699 podStartE2EDuration="1.639210699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:13:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:13:57.636963953 +0000 UTC m=+1252.390595356" watchObservedRunningTime="2026-09-11 04:13:57.639210699 +0000 UTC m=+1252.392842134" Sep 11 04:13:58.271249 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:58.271206 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8077-bc1790"] Sep 11 04:13:58.275467 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:58.275430 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8077-bc1790"] Sep 11 04:13:58.397845 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:58.397773 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8077-adf710"] Sep 11 04:13:58.398182 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:58.398168 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2de1b9a8-7f4f-4e34-bb5d-34151603001e" containerName="test-curl-8077-bc1790" Sep 11 04:13:58.398230 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:58.398184 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2de1b9a8-7f4f-4e34-bb5d-34151603001e" containerName="test-curl-8077-bc1790" Sep 11 04:13:58.398272 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:58.398250 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2de1b9a8-7f4f-4e34-bb5d-34151603001e" containerName="test-curl-8077-bc1790" Sep 11 04:13:58.401235 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:58.401218 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8077-adf710" Sep 11 04:13:58.408001 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:58.407964 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8077-adf710"] Sep 11 04:13:58.476404 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:58.476370 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjcqw\" (UniqueName: \"kubernetes.io/projected/5c029c38-c2e9-45f1-81b5-bdfe40418c96-kube-api-access-wjcqw\") pod \"test-curl-8077-adf710\" (UID: \"5c029c38-c2e9-45f1-81b5-bdfe40418c96\") " pod="opendatahub/test-curl-8077-adf710" Sep 11 04:13:58.577744 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:58.577716 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjcqw\" (UniqueName: \"kubernetes.io/projected/5c029c38-c2e9-45f1-81b5-bdfe40418c96-kube-api-access-wjcqw\") pod \"test-curl-8077-adf710\" (UID: \"5c029c38-c2e9-45f1-81b5-bdfe40418c96\") " pod="opendatahub/test-curl-8077-adf710" Sep 11 04:13:58.586486 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:58.586455 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjcqw\" (UniqueName: \"kubernetes.io/projected/5c029c38-c2e9-45f1-81b5-bdfe40418c96-kube-api-access-wjcqw\") pod \"test-curl-8077-adf710\" (UID: \"5c029c38-c2e9-45f1-81b5-bdfe40418c96\") " pod="opendatahub/test-curl-8077-adf710" Sep 11 04:13:58.629195 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:13:58.629164 2568 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8077-bc1790" container="test-curl-8077-bc1790" Sep 11 04:13:58.712185 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:58.712144 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8077-adf710" Sep 11 04:13:58.985646 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:58.985616 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8077-adf710"] Sep 11 04:13:58.987399 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:13:58.987370 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c029c38_c2e9_45f1_81b5_bdfe40418c96.slice/crio-2e2c8a1dc69e4c7c7a65d87d77aa82909c83e54fd3002b483ecf71a0a547d17f WatchSource:0}: Error finding container 2e2c8a1dc69e4c7c7a65d87d77aa82909c83e54fd3002b483ecf71a0a547d17f: Status 404 returned error can't find the container with id 2e2c8a1dc69e4c7c7a65d87d77aa82909c83e54fd3002b483ecf71a0a547d17f Sep 11 04:13:59.634930 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:59.634895 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8077-adf710" event={"ID":"5c029c38-c2e9-45f1-81b5-bdfe40418c96","Type":"ContainerStarted","Data":"2b369c5530a6e4ed7627511243000b54409ab0d47be849553bff26a3c9f641cf"} Sep 11 04:13:59.635440 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:59.634944 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8077-adf710" event={"ID":"5c029c38-c2e9-45f1-81b5-bdfe40418c96","Type":"ContainerStarted","Data":"2e2c8a1dc69e4c7c7a65d87d77aa82909c83e54fd3002b483ecf71a0a547d17f"} Sep 11 04:13:59.635440 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:59.634993 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8077-bc1790" podUID="2de1b9a8-7f4f-4e34-bb5d-34151603001e" containerName="test-curl-8077-bc1790" containerID="cri-o://322f7cbf6880ca180f56f736f1259f4733594e16860773e0c6dc4b2a3786f560" gracePeriod=2 Sep 11 04:13:59.652319 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:13:59.652266 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8077-adf710" podStartSLOduration=1.652252276 podStartE2EDuration="1.652252276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:13:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:13:59.650206877 +0000 UTC m=+1254.403838279" watchObservedRunningTime="2026-09-11 04:13:59.652252276 +0000 UTC m=+1254.405883676" Sep 11 04:13:59.652541 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:13:59.652515 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8077-bc1790\" is forbidden: User \"system:node:ip-10-0-140-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-30.ec2.internal' and this object" podUID="2de1b9a8-7f4f-4e34-bb5d-34151603001e" pod="opendatahub/test-curl-8077-bc1790" Sep 11 04:13:59.654295 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:13:59.654271 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8077-bc1790\" is forbidden: User \"system:node:ip-10-0-140-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-30.ec2.internal' and this object" podUID="2de1b9a8-7f4f-4e34-bb5d-34151603001e" pod="opendatahub/test-curl-8077-bc1790" Sep 11 04:14:00.270757 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:00.270711 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8077-adf710"] Sep 11 04:14:00.275460 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:00.275396 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8077-adf710"] Sep 11 04:14:00.405032 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:00.404996 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8077-5ff588"] Sep 11 04:14:00.405401 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:00.405387 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c029c38-c2e9-45f1-81b5-bdfe40418c96" containerName="test-curl-8077-adf710" Sep 11 04:14:00.405500 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:00.405402 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c029c38-c2e9-45f1-81b5-bdfe40418c96" containerName="test-curl-8077-adf710" Sep 11 04:14:00.405500 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:00.405463 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c029c38-c2e9-45f1-81b5-bdfe40418c96" containerName="test-curl-8077-adf710" Sep 11 04:14:00.408483 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:00.408463 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8077-5ff588" Sep 11 04:14:00.417257 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:00.417231 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8077-5ff588"] Sep 11 04:14:00.496649 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:00.491020 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjhnw\" (UniqueName: \"kubernetes.io/projected/137df77b-77e8-4e1c-8e3f-1b7969d735e1-kube-api-access-zjhnw\") pod \"test-curl-8077-5ff588\" (UID: \"137df77b-77e8-4e1c-8e3f-1b7969d735e1\") " pod="opendatahub/test-curl-8077-5ff588" Sep 11 04:14:00.591636 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:00.591592 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjhnw\" (UniqueName: \"kubernetes.io/projected/137df77b-77e8-4e1c-8e3f-1b7969d735e1-kube-api-access-zjhnw\") pod \"test-curl-8077-5ff588\" (UID: \"137df77b-77e8-4e1c-8e3f-1b7969d735e1\") " pod="opendatahub/test-curl-8077-5ff588" Sep 11 04:14:00.602871 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:00.602839 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjhnw\" (UniqueName: \"kubernetes.io/projected/137df77b-77e8-4e1c-8e3f-1b7969d735e1-kube-api-access-zjhnw\") pod \"test-curl-8077-5ff588\" (UID: \"137df77b-77e8-4e1c-8e3f-1b7969d735e1\") " pod="opendatahub/test-curl-8077-5ff588" Sep 11 04:14:00.640356 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:14:00.640326 2568 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8077-adf710" container="test-curl-8077-adf710" Sep 11 04:14:00.718564 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:00.718477 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8077-5ff588" Sep 11 04:14:00.989176 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:00.989148 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8077-5ff588"] Sep 11 04:14:00.991387 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:14:00.991361 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod137df77b_77e8_4e1c_8e3f_1b7969d735e1.slice/crio-10d3d06eec43e93fb4d252125a6d6e26cb6616536166d03788a984d11427aa78 WatchSource:0}: Error finding container 10d3d06eec43e93fb4d252125a6d6e26cb6616536166d03788a984d11427aa78: Status 404 returned error can't find the container with id 10d3d06eec43e93fb4d252125a6d6e26cb6616536166d03788a984d11427aa78 Sep 11 04:14:01.046829 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:01.046798 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:14:01.647662 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:01.647628 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8077-5ff588" event={"ID":"137df77b-77e8-4e1c-8e3f-1b7969d735e1","Type":"ContainerStarted","Data":"aaf464bbb7722d1853b746186fae43f6cffe5d3c628240def30e24d6f671dc9e"} Sep 11 04:14:01.648085 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:01.647676 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8077-5ff588" event={"ID":"137df77b-77e8-4e1c-8e3f-1b7969d735e1","Type":"ContainerStarted","Data":"10d3d06eec43e93fb4d252125a6d6e26cb6616536166d03788a984d11427aa78"} Sep 11 04:14:01.648085 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:01.647679 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8077-adf710" podUID="5c029c38-c2e9-45f1-81b5-bdfe40418c96" containerName="test-curl-8077-adf710" containerID="cri-o://2b369c5530a6e4ed7627511243000b54409ab0d47be849553bff26a3c9f641cf" gracePeriod=2 Sep 11 04:14:01.661372 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:01.661330 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8077-5ff588" podStartSLOduration=1.661315735 podStartE2EDuration="1.661315735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:14:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:14:01.660466991 +0000 UTC m=+1256.414098388" watchObservedRunningTime="2026-09-11 04:14:01.661315735 +0000 UTC m=+1256.414947138" Sep 11 04:14:01.662370 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:14:01.662346 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8077-adf710\" is forbidden: User \"system:node:ip-10-0-140-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-30.ec2.internal' and this object" podUID="5c029c38-c2e9-45f1-81b5-bdfe40418c96" pod="opendatahub/test-curl-8077-adf710" Sep 11 04:14:01.664363 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:14:01.664339 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8077-adf710\" is forbidden: User \"system:node:ip-10-0-140-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-30.ec2.internal' and this object" podUID="5c029c38-c2e9-45f1-81b5-bdfe40418c96" pod="opendatahub/test-curl-8077-adf710" Sep 11 04:14:01.822270 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:01.822247 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8077-bc1790" Sep 11 04:14:01.901540 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:01.901514 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-prp7n\" (UniqueName: \"kubernetes.io/projected/2de1b9a8-7f4f-4e34-bb5d-34151603001e-kube-api-access-prp7n\") pod \"2de1b9a8-7f4f-4e34-bb5d-34151603001e\" (UID: \"2de1b9a8-7f4f-4e34-bb5d-34151603001e\") " Sep 11 04:14:01.903383 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:01.903350 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2de1b9a8-7f4f-4e34-bb5d-34151603001e-kube-api-access-prp7n" (OuterVolumeSpecName: "kube-api-access-prp7n") pod "2de1b9a8-7f4f-4e34-bb5d-34151603001e" (UID: "2de1b9a8-7f4f-4e34-bb5d-34151603001e"). InnerVolumeSpecName "kube-api-access-prp7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:14:02.002103 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:02.002070 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prp7n\" (UniqueName: \"kubernetes.io/projected/2de1b9a8-7f4f-4e34-bb5d-34151603001e-kube-api-access-prp7n\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:14:02.256963 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:02.256879 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8077-5ff588"] Sep 11 04:14:02.261280 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:02.261253 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8077-5ff588"] Sep 11 04:14:02.384915 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:02.384878 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8077-79f5c0"] Sep 11 04:14:02.385284 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:02.385270 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="137df77b-77e8-4e1c-8e3f-1b7969d735e1" containerName="test-curl-8077-5ff588" Sep 11 04:14:02.385337 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:02.385285 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="137df77b-77e8-4e1c-8e3f-1b7969d735e1" containerName="test-curl-8077-5ff588" Sep 11 04:14:02.385374 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:02.385353 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="137df77b-77e8-4e1c-8e3f-1b7969d735e1" containerName="test-curl-8077-5ff588" Sep 11 04:14:02.388416 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:02.388400 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8077-79f5c0" Sep 11 04:14:02.395258 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:02.395232 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8077-79f5c0"] Sep 11 04:14:02.405775 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:02.405750 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgmpl\" (UniqueName: \"kubernetes.io/projected/4b7b5488-01df-4990-99eb-d1dc7354acf6-kube-api-access-lgmpl\") pod \"test-curl-8077-79f5c0\" (UID: \"4b7b5488-01df-4990-99eb-d1dc7354acf6\") " pod="opendatahub/test-curl-8077-79f5c0" Sep 11 04:14:02.507195 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:02.507098 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgmpl\" (UniqueName: \"kubernetes.io/projected/4b7b5488-01df-4990-99eb-d1dc7354acf6-kube-api-access-lgmpl\") pod \"test-curl-8077-79f5c0\" (UID: \"4b7b5488-01df-4990-99eb-d1dc7354acf6\") " pod="opendatahub/test-curl-8077-79f5c0" Sep 11 04:14:02.520871 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:02.520839 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgmpl\" (UniqueName: \"kubernetes.io/projected/4b7b5488-01df-4990-99eb-d1dc7354acf6-kube-api-access-lgmpl\") pod \"test-curl-8077-79f5c0\" (UID: \"4b7b5488-01df-4990-99eb-d1dc7354acf6\") " pod="opendatahub/test-curl-8077-79f5c0" Sep 11 04:14:02.654390 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:02.654353 2568 generic.go:356] "Generic (PLEG): container finished" podID="2de1b9a8-7f4f-4e34-bb5d-34151603001e" containerID="322f7cbf6880ca180f56f736f1259f4733594e16860773e0c6dc4b2a3786f560" exitCode=137 Sep 11 04:14:02.654390 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:02.654385 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8077-bc1790" Sep 11 04:14:02.654929 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:02.654447 2568 scope.go:117] "RemoveContainer" containerID="322f7cbf6880ca180f56f736f1259f4733594e16860773e0c6dc4b2a3786f560" Sep 11 04:14:02.654929 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:14:02.654716 2568 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8077-5ff588" container="test-curl-8077-5ff588" Sep 11 04:14:02.663665 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:02.663637 2568 scope.go:117] "RemoveContainer" containerID="322f7cbf6880ca180f56f736f1259f4733594e16860773e0c6dc4b2a3786f560" Sep 11 04:14:02.663930 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:14:02.663912 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"322f7cbf6880ca180f56f736f1259f4733594e16860773e0c6dc4b2a3786f560\": container with ID starting with 322f7cbf6880ca180f56f736f1259f4733594e16860773e0c6dc4b2a3786f560 not found: ID does not exist" containerID="322f7cbf6880ca180f56f736f1259f4733594e16860773e0c6dc4b2a3786f560" Sep 11 04:14:02.663980 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:02.663940 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"322f7cbf6880ca180f56f736f1259f4733594e16860773e0c6dc4b2a3786f560"} err="failed to get container status \"322f7cbf6880ca180f56f736f1259f4733594e16860773e0c6dc4b2a3786f560\": rpc error: code = NotFound desc = could not find container \"322f7cbf6880ca180f56f736f1259f4733594e16860773e0c6dc4b2a3786f560\": container with ID starting with 322f7cbf6880ca180f56f736f1259f4733594e16860773e0c6dc4b2a3786f560 not found: ID does not exist" Sep 11 04:14:02.665184 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:14:02.665165 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8077-bc1790\" is forbidden: User \"system:node:ip-10-0-140-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-30.ec2.internal' and this object" podUID="2de1b9a8-7f4f-4e34-bb5d-34151603001e" pod="opendatahub/test-curl-8077-bc1790" Sep 11 04:14:02.698393 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:02.698356 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8077-79f5c0" Sep 11 04:14:02.970086 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:02.970042 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8077-79f5c0"] Sep 11 04:14:02.972040 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:14:02.972008 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b7b5488_01df_4990_99eb_d1dc7354acf6.slice/crio-28943ca2bb20dcd49e42a7a18bedbfb4a23f682d28f872b212e86f7bc31bbb2b WatchSource:0}: Error finding container 28943ca2bb20dcd49e42a7a18bedbfb4a23f682d28f872b212e86f7bc31bbb2b: Status 404 returned error can't find the container with id 28943ca2bb20dcd49e42a7a18bedbfb4a23f682d28f872b212e86f7bc31bbb2b Sep 11 04:14:03.662973 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:03.662942 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8077-79f5c0" event={"ID":"4b7b5488-01df-4990-99eb-d1dc7354acf6","Type":"ContainerStarted","Data":"4ee1350c7a15b4240628a1ab4c8f55f0c18cd271dec5e27333f2aae513093aba"} Sep 11 04:14:03.663362 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:03.662987 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8077-79f5c0" event={"ID":"4b7b5488-01df-4990-99eb-d1dc7354acf6","Type":"ContainerStarted","Data":"28943ca2bb20dcd49e42a7a18bedbfb4a23f682d28f872b212e86f7bc31bbb2b"} Sep 11 04:14:03.663362 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:03.663072 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8077-5ff588" podUID="137df77b-77e8-4e1c-8e3f-1b7969d735e1" containerName="test-curl-8077-5ff588" containerID="cri-o://aaf464bbb7722d1853b746186fae43f6cffe5d3c628240def30e24d6f671dc9e" gracePeriod=2 Sep 11 04:14:03.677776 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:03.677732 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8077-79f5c0" podStartSLOduration=1.677715526 podStartE2EDuration="1.677715526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:14:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:14:03.676062752 +0000 UTC m=+1258.429694154" watchObservedRunningTime="2026-09-11 04:14:03.677715526 +0000 UTC m=+1258.431346928" Sep 11 04:14:03.677933 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:14:03.677910 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8077-5ff588\" is forbidden: User \"system:node:ip-10-0-140-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-30.ec2.internal' and this object" podUID="137df77b-77e8-4e1c-8e3f-1b7969d735e1" pod="opendatahub/test-curl-8077-5ff588" Sep 11 04:14:03.679645 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:14:03.679622 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8077-bc1790\" is forbidden: User \"system:node:ip-10-0-140-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-30.ec2.internal' and this object" podUID="2de1b9a8-7f4f-4e34-bb5d-34151603001e" pod="opendatahub/test-curl-8077-bc1790" Sep 11 04:14:03.681235 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:14:03.681215 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8077-5ff588\" is forbidden: User \"system:node:ip-10-0-140-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-30.ec2.internal' and this object" podUID="137df77b-77e8-4e1c-8e3f-1b7969d735e1" pod="opendatahub/test-curl-8077-5ff588" Sep 11 04:14:03.682742 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:14:03.682723 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8077-bc1790\" is forbidden: User \"system:node:ip-10-0-140-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-30.ec2.internal' and this object" podUID="2de1b9a8-7f4f-4e34-bb5d-34151603001e" pod="opendatahub/test-curl-8077-bc1790" Sep 11 04:14:03.771038 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:03.770971 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2de1b9a8-7f4f-4e34-bb5d-34151603001e" path="/var/lib/kubelet/pods/2de1b9a8-7f4f-4e34-bb5d-34151603001e/volumes" Sep 11 04:14:03.842159 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:03.842128 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8077-adf710" Sep 11 04:14:03.916058 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:03.916025 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wjcqw\" (UniqueName: \"kubernetes.io/projected/5c029c38-c2e9-45f1-81b5-bdfe40418c96-kube-api-access-wjcqw\") pod \"5c029c38-c2e9-45f1-81b5-bdfe40418c96\" (UID: \"5c029c38-c2e9-45f1-81b5-bdfe40418c96\") " Sep 11 04:14:03.917830 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:03.917807 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c029c38-c2e9-45f1-81b5-bdfe40418c96-kube-api-access-wjcqw" (OuterVolumeSpecName: "kube-api-access-wjcqw") pod "5c029c38-c2e9-45f1-81b5-bdfe40418c96" (UID: "5c029c38-c2e9-45f1-81b5-bdfe40418c96"). InnerVolumeSpecName "kube-api-access-wjcqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:14:04.017297 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:04.017261 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjcqw\" (UniqueName: \"kubernetes.io/projected/5c029c38-c2e9-45f1-81b5-bdfe40418c96-kube-api-access-wjcqw\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:14:04.285436 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:04.285348 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8077-79f5c0"] Sep 11 04:14:04.291989 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:04.291955 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8077-79f5c0"] Sep 11 04:14:04.669876 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:04.669841 2568 generic.go:356] "Generic (PLEG): container finished" podID="5c029c38-c2e9-45f1-81b5-bdfe40418c96" containerID="2b369c5530a6e4ed7627511243000b54409ab0d47be849553bff26a3c9f641cf" exitCode=137 Sep 11 04:14:04.670367 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:04.669905 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8077-adf710" Sep 11 04:14:04.670367 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:04.669953 2568 scope.go:117] "RemoveContainer" containerID="2b369c5530a6e4ed7627511243000b54409ab0d47be849553bff26a3c9f641cf" Sep 11 04:14:04.670367 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:14:04.670218 2568 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8077-79f5c0" container="test-curl-8077-79f5c0" Sep 11 04:14:04.678874 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:04.678860 2568 scope.go:117] "RemoveContainer" containerID="2b369c5530a6e4ed7627511243000b54409ab0d47be849553bff26a3c9f641cf" Sep 11 04:14:04.679150 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:14:04.679129 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b369c5530a6e4ed7627511243000b54409ab0d47be849553bff26a3c9f641cf\": container with ID starting with 2b369c5530a6e4ed7627511243000b54409ab0d47be849553bff26a3c9f641cf not found: ID does not exist" containerID="2b369c5530a6e4ed7627511243000b54409ab0d47be849553bff26a3c9f641cf" Sep 11 04:14:04.679216 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:04.679160 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b369c5530a6e4ed7627511243000b54409ab0d47be849553bff26a3c9f641cf"} err="failed to get container status \"2b369c5530a6e4ed7627511243000b54409ab0d47be849553bff26a3c9f641cf\": rpc error: code = NotFound desc = could not find container \"2b369c5530a6e4ed7627511243000b54409ab0d47be849553bff26a3c9f641cf\": container with ID starting with 2b369c5530a6e4ed7627511243000b54409ab0d47be849553bff26a3c9f641cf not found: ID does not exist" Sep 11 04:14:04.680285 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:14:04.680257 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8077-adf710\" is forbidden: User \"system:node:ip-10-0-140-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-30.ec2.internal' and this object" podUID="5c029c38-c2e9-45f1-81b5-bdfe40418c96" pod="opendatahub/test-curl-8077-adf710" Sep 11 04:14:04.918392 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:04.918348 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:14:05.296487 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:05.296450 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:14:05.448876 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:05.448825 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:14:05.680613 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:05.679314 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8077-79f5c0" podUID="4b7b5488-01df-4990-99eb-d1dc7354acf6" containerName="test-curl-8077-79f5c0" containerID="cri-o://4ee1350c7a15b4240628a1ab4c8f55f0c18cd271dec5e27333f2aae513093aba" gracePeriod=2 Sep 11 04:14:05.681497 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:14:05.681464 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8077-79f5c0\" is forbidden: User \"system:node:ip-10-0-140-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-30.ec2.internal' and this object" podUID="4b7b5488-01df-4990-99eb-d1dc7354acf6" pod="opendatahub/test-curl-8077-79f5c0" Sep 11 04:14:05.685651 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:14:05.685614 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8077-adf710\" is forbidden: User \"system:node:ip-10-0-140-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-30.ec2.internal' and this object" podUID="5c029c38-c2e9-45f1-81b5-bdfe40418c96" pod="opendatahub/test-curl-8077-adf710" Sep 11 04:14:05.768065 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:14:05.767992 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8077-79f5c0\" is forbidden: User \"system:node:ip-10-0-140-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-30.ec2.internal' and this object" podUID="4b7b5488-01df-4990-99eb-d1dc7354acf6" pod="opendatahub/test-curl-8077-79f5c0" Sep 11 04:14:05.770317 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:14:05.770275 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8077-adf710\" is forbidden: User \"system:node:ip-10-0-140-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-30.ec2.internal' and this object" podUID="5c029c38-c2e9-45f1-81b5-bdfe40418c96" pod="opendatahub/test-curl-8077-adf710" Sep 11 04:14:05.770750 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:05.770728 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c029c38-c2e9-45f1-81b5-bdfe40418c96" path="/var/lib/kubelet/pods/5c029c38-c2e9-45f1-81b5-bdfe40418c96/volumes" Sep 11 04:14:05.849178 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:05.849153 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8077-5ff588" Sep 11 04:14:05.851448 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:14:05.851417 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8077-5ff588\" is forbidden: User \"system:node:ip-10-0-140-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-30.ec2.internal' and this object" podUID="137df77b-77e8-4e1c-8e3f-1b7969d735e1" pod="opendatahub/test-curl-8077-5ff588" Sep 11 04:14:05.888788 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:05.888760 2568 scope.go:117] "RemoveContainer" containerID="903a28f32ae14d7edae13fe93804da1ffb04dbf3ba3e1d4c026c026ff837a417" Sep 11 04:14:05.932690 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:05.932619 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjhnw\" (UniqueName: \"kubernetes.io/projected/137df77b-77e8-4e1c-8e3f-1b7969d735e1-kube-api-access-zjhnw\") pod \"137df77b-77e8-4e1c-8e3f-1b7969d735e1\" (UID: \"137df77b-77e8-4e1c-8e3f-1b7969d735e1\") " Sep 11 04:14:05.934669 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:05.934644 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/137df77b-77e8-4e1c-8e3f-1b7969d735e1-kube-api-access-zjhnw" (OuterVolumeSpecName: "kube-api-access-zjhnw") pod "137df77b-77e8-4e1c-8e3f-1b7969d735e1" (UID: "137df77b-77e8-4e1c-8e3f-1b7969d735e1"). InnerVolumeSpecName "kube-api-access-zjhnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:14:06.033256 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:06.033216 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjhnw\" (UniqueName: \"kubernetes.io/projected/137df77b-77e8-4e1c-8e3f-1b7969d735e1-kube-api-access-zjhnw\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:14:06.686399 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:06.686364 2568 generic.go:356] "Generic (PLEG): container finished" podID="137df77b-77e8-4e1c-8e3f-1b7969d735e1" containerID="aaf464bbb7722d1853b746186fae43f6cffe5d3c628240def30e24d6f671dc9e" exitCode=137 Sep 11 04:14:06.686940 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:06.686405 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8077-5ff588" Sep 11 04:14:06.686940 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:06.686424 2568 scope.go:117] "RemoveContainer" containerID="aaf464bbb7722d1853b746186fae43f6cffe5d3c628240def30e24d6f671dc9e" Sep 11 04:14:06.688465 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:14:06.688441 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8077-5ff588\" is forbidden: User \"system:node:ip-10-0-140-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-30.ec2.internal' and this object" podUID="137df77b-77e8-4e1c-8e3f-1b7969d735e1" pod="opendatahub/test-curl-8077-5ff588" Sep 11 04:14:06.695015 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:06.694997 2568 scope.go:117] "RemoveContainer" containerID="aaf464bbb7722d1853b746186fae43f6cffe5d3c628240def30e24d6f671dc9e" Sep 11 04:14:06.695300 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:14:06.695282 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaf464bbb7722d1853b746186fae43f6cffe5d3c628240def30e24d6f671dc9e\": container with ID starting with aaf464bbb7722d1853b746186fae43f6cffe5d3c628240def30e24d6f671dc9e not found: ID does not exist" containerID="aaf464bbb7722d1853b746186fae43f6cffe5d3c628240def30e24d6f671dc9e" Sep 11 04:14:06.695362 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:06.695311 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaf464bbb7722d1853b746186fae43f6cffe5d3c628240def30e24d6f671dc9e"} err="failed to get container status \"aaf464bbb7722d1853b746186fae43f6cffe5d3c628240def30e24d6f671dc9e\": rpc error: code = NotFound desc = could not find container \"aaf464bbb7722d1853b746186fae43f6cffe5d3c628240def30e24d6f671dc9e\": container with ID starting with aaf464bbb7722d1853b746186fae43f6cffe5d3c628240def30e24d6f671dc9e not found: ID does not exist" Sep 11 04:14:06.696573 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:14:06.696552 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8077-5ff588\" is forbidden: User \"system:node:ip-10-0-140-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-30.ec2.internal' and this object" podUID="137df77b-77e8-4e1c-8e3f-1b7969d735e1" pod="opendatahub/test-curl-8077-5ff588" Sep 11 04:14:07.769599 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:07.769557 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="137df77b-77e8-4e1c-8e3f-1b7969d735e1" path="/var/lib/kubelet/pods/137df77b-77e8-4e1c-8e3f-1b7969d735e1/volumes" Sep 11 04:14:07.860217 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:07.860197 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8077-79f5c0" Sep 11 04:14:07.947490 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:07.947417 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgmpl\" (UniqueName: \"kubernetes.io/projected/4b7b5488-01df-4990-99eb-d1dc7354acf6-kube-api-access-lgmpl\") pod \"4b7b5488-01df-4990-99eb-d1dc7354acf6\" (UID: \"4b7b5488-01df-4990-99eb-d1dc7354acf6\") " Sep 11 04:14:07.949442 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:07.949411 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b7b5488-01df-4990-99eb-d1dc7354acf6-kube-api-access-lgmpl" (OuterVolumeSpecName: "kube-api-access-lgmpl") pod "4b7b5488-01df-4990-99eb-d1dc7354acf6" (UID: "4b7b5488-01df-4990-99eb-d1dc7354acf6"). InnerVolumeSpecName "kube-api-access-lgmpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:14:08.048959 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:08.048924 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgmpl\" (UniqueName: \"kubernetes.io/projected/4b7b5488-01df-4990-99eb-d1dc7354acf6-kube-api-access-lgmpl\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:14:08.276465 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:08.276343 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:14:08.365683 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:08.365651 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:14:08.700158 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:08.700122 2568 generic.go:356] "Generic (PLEG): container finished" podID="4b7b5488-01df-4990-99eb-d1dc7354acf6" containerID="4ee1350c7a15b4240628a1ab4c8f55f0c18cd271dec5e27333f2aae513093aba" exitCode=137 Sep 11 04:14:08.700337 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:08.700169 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8077-79f5c0" Sep 11 04:14:08.700337 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:08.700228 2568 scope.go:117] "RemoveContainer" containerID="4ee1350c7a15b4240628a1ab4c8f55f0c18cd271dec5e27333f2aae513093aba" Sep 11 04:14:08.709828 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:08.709810 2568 scope.go:117] "RemoveContainer" containerID="4ee1350c7a15b4240628a1ab4c8f55f0c18cd271dec5e27333f2aae513093aba" Sep 11 04:14:08.710156 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:14:08.710138 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ee1350c7a15b4240628a1ab4c8f55f0c18cd271dec5e27333f2aae513093aba\": container with ID starting with 4ee1350c7a15b4240628a1ab4c8f55f0c18cd271dec5e27333f2aae513093aba not found: ID does not exist" containerID="4ee1350c7a15b4240628a1ab4c8f55f0c18cd271dec5e27333f2aae513093aba" Sep 11 04:14:08.710235 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:08.710163 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ee1350c7a15b4240628a1ab4c8f55f0c18cd271dec5e27333f2aae513093aba"} err="failed to get container status \"4ee1350c7a15b4240628a1ab4c8f55f0c18cd271dec5e27333f2aae513093aba\": rpc error: code = NotFound desc = could not find container \"4ee1350c7a15b4240628a1ab4c8f55f0c18cd271dec5e27333f2aae513093aba\": container with ID starting with 4ee1350c7a15b4240628a1ab4c8f55f0c18cd271dec5e27333f2aae513093aba not found: ID does not exist" Sep 11 04:14:08.710799 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:14:08.710781 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8077-79f5c0\" is forbidden: User \"system:node:ip-10-0-140-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-30.ec2.internal' and this object" podUID="4b7b5488-01df-4990-99eb-d1dc7354acf6" pod="opendatahub/test-curl-8077-79f5c0" Sep 11 04:14:09.482472 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:09.482438 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-354ac6fe-7f56b65ddd-wwg4b"] Sep 11 04:14:09.482993 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:09.482973 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b7b5488-01df-4990-99eb-d1dc7354acf6" containerName="test-curl-8077-79f5c0" Sep 11 04:14:09.483103 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:09.482996 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b7b5488-01df-4990-99eb-d1dc7354acf6" containerName="test-curl-8077-79f5c0" Sep 11 04:14:09.483167 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:09.483156 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b7b5488-01df-4990-99eb-d1dc7354acf6" containerName="test-curl-8077-79f5c0" Sep 11 04:14:09.490676 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:09.490646 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-354ac6fe-7f56b65ddd-wwg4b" Sep 11 04:14:09.493737 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:09.493704 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-354ac6fe-serving-cert\"" Sep 11 04:14:09.499379 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:09.499356 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-354ac6fe-7f56b65ddd-wwg4b"] Sep 11 04:14:09.509671 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:14:09.509644 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8077-79f5c0\" is forbidden: User \"system:node:ip-10-0-140-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-30.ec2.internal' and this object" podUID="4b7b5488-01df-4990-99eb-d1dc7354acf6" pod="opendatahub/test-curl-8077-79f5c0" Sep 11 04:14:09.516283 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:09.516257 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:14:09.559737 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:09.559704 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqg7h\" (UniqueName: \"kubernetes.io/projected/8819fb73-856b-442b-a160-f9f51f4385ca-kube-api-access-pqg7h\") pod \"maas-api-e2e-mt-354ac6fe-7f56b65ddd-wwg4b\" (UID: \"8819fb73-856b-442b-a160-f9f51f4385ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-354ac6fe-7f56b65ddd-wwg4b" Sep 11 04:14:09.559889 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:09.559814 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8819fb73-856b-442b-a160-f9f51f4385ca-maas-api-tls\") pod \"maas-api-e2e-mt-354ac6fe-7f56b65ddd-wwg4b\" (UID: \"8819fb73-856b-442b-a160-f9f51f4385ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-354ac6fe-7f56b65ddd-wwg4b" Sep 11 04:14:09.559889 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:09.559848 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8819fb73-856b-442b-a160-f9f51f4385ca-openshift-service-ca\") pod \"maas-api-e2e-mt-354ac6fe-7f56b65ddd-wwg4b\" (UID: \"8819fb73-856b-442b-a160-f9f51f4385ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-354ac6fe-7f56b65ddd-wwg4b" Sep 11 04:14:09.559889 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:09.559888 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8819fb73-856b-442b-a160-f9f51f4385ca-metrics-certs\") pod \"maas-api-e2e-mt-354ac6fe-7f56b65ddd-wwg4b\" (UID: \"8819fb73-856b-442b-a160-f9f51f4385ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-354ac6fe-7f56b65ddd-wwg4b" Sep 11 04:14:09.606928 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:09.606898 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:14:09.661291 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:09.661263 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8819fb73-856b-442b-a160-f9f51f4385ca-metrics-certs\") pod \"maas-api-e2e-mt-354ac6fe-7f56b65ddd-wwg4b\" (UID: \"8819fb73-856b-442b-a160-f9f51f4385ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-354ac6fe-7f56b65ddd-wwg4b" Sep 11 04:14:09.661418 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:09.661314 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqg7h\" (UniqueName: \"kubernetes.io/projected/8819fb73-856b-442b-a160-f9f51f4385ca-kube-api-access-pqg7h\") pod \"maas-api-e2e-mt-354ac6fe-7f56b65ddd-wwg4b\" (UID: \"8819fb73-856b-442b-a160-f9f51f4385ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-354ac6fe-7f56b65ddd-wwg4b" Sep 11 04:14:09.661418 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:09.661369 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8819fb73-856b-442b-a160-f9f51f4385ca-maas-api-tls\") pod \"maas-api-e2e-mt-354ac6fe-7f56b65ddd-wwg4b\" (UID: \"8819fb73-856b-442b-a160-f9f51f4385ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-354ac6fe-7f56b65ddd-wwg4b" Sep 11 04:14:09.661418 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:09.661389 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8819fb73-856b-442b-a160-f9f51f4385ca-openshift-service-ca\") pod \"maas-api-e2e-mt-354ac6fe-7f56b65ddd-wwg4b\" (UID: \"8819fb73-856b-442b-a160-f9f51f4385ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-354ac6fe-7f56b65ddd-wwg4b" Sep 11 04:14:09.661537 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:14:09.661494 2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-354ac6fe-serving-cert: secret "maas-api-e2e-mt-354ac6fe-serving-cert" not found Sep 11 04:14:09.661582 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:14:09.661564 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8819fb73-856b-442b-a160-f9f51f4385ca-maas-api-tls podName:8819fb73-856b-442b-a160-f9f51f4385ca nodeName:}" failed. No retries permitted until 2026-09-11 04:14:10.161544166 +0000 UTC m=+1264.915175564 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8819fb73-856b-442b-a160-f9f51f4385ca-maas-api-tls") pod "maas-api-e2e-mt-354ac6fe-7f56b65ddd-wwg4b" (UID: "8819fb73-856b-442b-a160-f9f51f4385ca") : secret "maas-api-e2e-mt-354ac6fe-serving-cert" not found Sep 11 04:14:09.662094 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:09.662043 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8819fb73-856b-442b-a160-f9f51f4385ca-openshift-service-ca\") pod \"maas-api-e2e-mt-354ac6fe-7f56b65ddd-wwg4b\" (UID: \"8819fb73-856b-442b-a160-f9f51f4385ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-354ac6fe-7f56b65ddd-wwg4b" Sep 11 04:14:09.663837 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:09.663813 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8819fb73-856b-442b-a160-f9f51f4385ca-metrics-certs\") pod \"maas-api-e2e-mt-354ac6fe-7f56b65ddd-wwg4b\" (UID: \"8819fb73-856b-442b-a160-f9f51f4385ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-354ac6fe-7f56b65ddd-wwg4b" Sep 11 04:14:09.672630 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:09.672607 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqg7h\" (UniqueName: \"kubernetes.io/projected/8819fb73-856b-442b-a160-f9f51f4385ca-kube-api-access-pqg7h\") pod \"maas-api-e2e-mt-354ac6fe-7f56b65ddd-wwg4b\" (UID: \"8819fb73-856b-442b-a160-f9f51f4385ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-354ac6fe-7f56b65ddd-wwg4b" Sep 11 04:14:09.768032 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:09.767954 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b7b5488-01df-4990-99eb-d1dc7354acf6" path="/var/lib/kubelet/pods/4b7b5488-01df-4990-99eb-d1dc7354acf6/volumes" Sep 11 04:14:09.880932 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:09.880895 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:14:09.988399 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:09.988361 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:14:10.094719 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:10.094635 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:14:10.127933 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:10.127903 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-354ac6fe-584dcccccc-sz2zv"] Sep 11 04:14:10.131666 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:10.131644 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-354ac6fe-584dcccccc-sz2zv" Sep 11 04:14:10.135031 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:10.135010 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-354ac6fe-dockercfg-5bjdh\"" Sep 11 04:14:10.135159 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:10.135111 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-354ac6fe\"" Sep 11 04:14:10.142642 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:10.142617 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-354ac6fe-584dcccccc-sz2zv"] Sep 11 04:14:10.166760 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:10.166727 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpr8f\" (UniqueName: \"kubernetes.io/projected/cfd1c84a-ab1c-454a-a40a-457c3986ab4d-kube-api-access-zpr8f\") pod \"payload-processing-e2e-mt-354ac6fe-584dcccccc-sz2zv\" (UID: \"cfd1c84a-ab1c-454a-a40a-457c3986ab4d\") " pod="openshift-ingress/payload-processing-e2e-mt-354ac6fe-584dcccccc-sz2zv" Sep 11 04:14:10.166893 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:10.166790 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8819fb73-856b-442b-a160-f9f51f4385ca-maas-api-tls\") pod \"maas-api-e2e-mt-354ac6fe-7f56b65ddd-wwg4b\" (UID: \"8819fb73-856b-442b-a160-f9f51f4385ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-354ac6fe-7f56b65ddd-wwg4b" Sep 11 04:14:10.166893 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:10.166834 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfd1c84a-ab1c-454a-a40a-457c3986ab4d-plugins-config-volume\") pod \"payload-processing-e2e-mt-354ac6fe-584dcccccc-sz2zv\" (UID: \"cfd1c84a-ab1c-454a-a40a-457c3986ab4d\") " pod="openshift-ingress/payload-processing-e2e-mt-354ac6fe-584dcccccc-sz2zv" Sep 11 04:14:10.169175 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:10.169146 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8819fb73-856b-442b-a160-f9f51f4385ca-maas-api-tls\") pod \"maas-api-e2e-mt-354ac6fe-7f56b65ddd-wwg4b\" (UID: \"8819fb73-856b-442b-a160-f9f51f4385ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-354ac6fe-7f56b65ddd-wwg4b" Sep 11 04:14:10.268102 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:10.268057 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfd1c84a-ab1c-454a-a40a-457c3986ab4d-plugins-config-volume\") pod \"payload-processing-e2e-mt-354ac6fe-584dcccccc-sz2zv\" (UID: \"cfd1c84a-ab1c-454a-a40a-457c3986ab4d\") " pod="openshift-ingress/payload-processing-e2e-mt-354ac6fe-584dcccccc-sz2zv" Sep 11 04:14:10.268343 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:10.268150 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpr8f\" (UniqueName: \"kubernetes.io/projected/cfd1c84a-ab1c-454a-a40a-457c3986ab4d-kube-api-access-zpr8f\") pod \"payload-processing-e2e-mt-354ac6fe-584dcccccc-sz2zv\" (UID: \"cfd1c84a-ab1c-454a-a40a-457c3986ab4d\") " pod="openshift-ingress/payload-processing-e2e-mt-354ac6fe-584dcccccc-sz2zv" Sep 11 04:14:10.268751 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:10.268731 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfd1c84a-ab1c-454a-a40a-457c3986ab4d-plugins-config-volume\") pod \"payload-processing-e2e-mt-354ac6fe-584dcccccc-sz2zv\" (UID: \"cfd1c84a-ab1c-454a-a40a-457c3986ab4d\") " pod="openshift-ingress/payload-processing-e2e-mt-354ac6fe-584dcccccc-sz2zv" Sep 11 04:14:10.277229 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:10.277199 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpr8f\" (UniqueName: \"kubernetes.io/projected/cfd1c84a-ab1c-454a-a40a-457c3986ab4d-kube-api-access-zpr8f\") pod \"payload-processing-e2e-mt-354ac6fe-584dcccccc-sz2zv\" (UID: \"cfd1c84a-ab1c-454a-a40a-457c3986ab4d\") " pod="openshift-ingress/payload-processing-e2e-mt-354ac6fe-584dcccccc-sz2zv" Sep 11 04:14:10.408528 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:10.408501 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-354ac6fe-7f56b65ddd-wwg4b" Sep 11 04:14:10.447799 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:10.447337 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-354ac6fe-584dcccccc-sz2zv" Sep 11 04:14:10.684557 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:10.684482 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-354ac6fe-7f56b65ddd-wwg4b"] Sep 11 04:14:10.687956 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:14:10.687920 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8819fb73_856b_442b_a160_f9f51f4385ca.slice/crio-161ba9b0da8d836a3a72b92866e3ce800bf408931508c3e414c475d79afc967d WatchSource:0}: Error finding container 161ba9b0da8d836a3a72b92866e3ce800bf408931508c3e414c475d79afc967d: Status 404 returned error can't find the container with id 161ba9b0da8d836a3a72b92866e3ce800bf408931508c3e414c475d79afc967d Sep 11 04:14:10.712032 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:10.711998 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-354ac6fe-7f56b65ddd-wwg4b" event={"ID":"8819fb73-856b-442b-a160-f9f51f4385ca","Type":"ContainerStarted","Data":"161ba9b0da8d836a3a72b92866e3ce800bf408931508c3e414c475d79afc967d"} Sep 11 04:14:10.737067 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:10.737026 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-354ac6fe-584dcccccc-sz2zv"] Sep 11 04:14:10.739260 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:14:10.739228 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfd1c84a_ab1c_454a_a40a_457c3986ab4d.slice/crio-74c7ba372ebb9112851a68bf978c5409decbdb80488c99a379ef9f6c7eec242b WatchSource:0}: Error finding container 74c7ba372ebb9112851a68bf978c5409decbdb80488c99a379ef9f6c7eec242b: Status 404 returned error can't find the container with id 74c7ba372ebb9112851a68bf978c5409decbdb80488c99a379ef9f6c7eec242b Sep 11 04:14:11.720107 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:11.720064 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-354ac6fe-7f56b65ddd-wwg4b" event={"ID":"8819fb73-856b-442b-a160-f9f51f4385ca","Type":"ContainerStarted","Data":"c717407e7c92414c737d1c8bf0205d5532c7c4beeb8f789cfa66f157b4062985"} Sep 11 04:14:11.720573 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:11.720184 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-354ac6fe-7f56b65ddd-wwg4b" Sep 11 04:14:11.721735 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:11.721711 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-354ac6fe-584dcccccc-sz2zv" event={"ID":"cfd1c84a-ab1c-454a-a40a-457c3986ab4d","Type":"ContainerStarted","Data":"3957f8a387089878cca89421cb95324528fa9e9ea7da0cbb27e3dcc2813d9f8e"} Sep 11 04:14:11.721862 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:11.721759 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-354ac6fe-584dcccccc-sz2zv" event={"ID":"cfd1c84a-ab1c-454a-a40a-457c3986ab4d","Type":"ContainerStarted","Data":"74c7ba372ebb9112851a68bf978c5409decbdb80488c99a379ef9f6c7eec242b"} Sep 11 04:14:11.721862 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:11.721790 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-354ac6fe-584dcccccc-sz2zv" Sep 11 04:14:11.740337 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:11.740281 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-354ac6fe-7f56b65ddd-wwg4b" podStartSLOduration=2.2645182200000002 podStartE2EDuration="2.740261397s" totalImagesPullingTime="475.743177ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:14:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:14:11.737189321 +0000 UTC m=+1266.490820728" watchObservedRunningTime="2026-09-11 04:14:11.740261397 +0000 UTC m=+1266.493892801" Sep 11 04:14:11.754096 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:11.754011 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-354ac6fe-584dcccccc-sz2zv" podStartSLOduration=1.753994485 podStartE2EDuration="1.753994485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:14:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:14:11.753237125 +0000 UTC m=+1266.506868528" watchObservedRunningTime="2026-09-11 04:14:11.753994485 +0000 UTC m=+1266.507625887" Sep 11 04:14:15.182779 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:15.182740 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:14:17.732214 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:17.732183 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-354ac6fe-7f56b65ddd-wwg4b" Sep 11 04:14:22.729208 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:22.729178 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-354ac6fe-584dcccccc-sz2zv" Sep 11 04:14:31.505976 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:31.505936 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:14:33.289204 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:33.289149 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:14:39.118915 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:14:39.118870 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:15:00.170231 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:00.170196 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f8229c-29818335-cpn7p"] Sep 11 04:15:00.173852 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:00.173835 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f8229c-29818335-cpn7p" Sep 11 04:15:00.176451 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:00.176429 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6brsr\"" Sep 11 04:15:00.189416 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:00.189381 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f8229c-29818335-cpn7p"] Sep 11 04:15:00.190469 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:00.190443 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f8229c-29818335-cpn7p" Sep 11 04:15:00.474235 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:00.474210 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f8229c-29818335-cpn7p"] Sep 11 04:15:00.476660 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:15:00.476632 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc0b3d9d_eeba_48a4_a619_b01a96c5ed17.slice/crio-aa5449cdd83ff71afb72893d4cf897fdc330f1f4325ec19fb01cb4291c66ec51 WatchSource:0}: Error finding container aa5449cdd83ff71afb72893d4cf897fdc330f1f4325ec19fb01cb4291c66ec51: Status 404 returned error can't find the container with id aa5449cdd83ff71afb72893d4cf897fdc330f1f4325ec19fb01cb4291c66ec51 Sep 11 04:15:00.980783 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:00.980751 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f8229c-29818335-cpn7p" event={"ID":"cc0b3d9d-eeba-48a4-a619-b01a96c5ed17","Type":"ContainerStarted","Data":"e49ebe998fb40dd210eb17a5877e56c00c97c672b73119ce70553edd70b29bf9"} Sep 11 04:15:00.980958 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:00.980806 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f8229c-29818335-cpn7p" event={"ID":"cc0b3d9d-eeba-48a4-a619-b01a96c5ed17","Type":"ContainerStarted","Data":"aa5449cdd83ff71afb72893d4cf897fdc330f1f4325ec19fb01cb4291c66ec51"} Sep 11 04:15:00.995712 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:00.995627 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f8229c-29818335-cpn7p" podStartSLOduration=0.995611603 podStartE2EDuration="995.611603ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:15:00.994367007 +0000 UTC m=+1315.747998408" watchObservedRunningTime="2026-09-11 04:15:00.995611603 +0000 UTC m=+1315.749243005" Sep 11 04:15:01.987374 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:01.987338 2568 generic.go:356] "Generic (PLEG): container finished" podID="cc0b3d9d-eeba-48a4-a619-b01a96c5ed17" containerID="e49ebe998fb40dd210eb17a5877e56c00c97c672b73119ce70553edd70b29bf9" exitCode=0 Sep 11 04:15:01.987777 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:01.987422 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f8229c-29818335-cpn7p" event={"ID":"cc0b3d9d-eeba-48a4-a619-b01a96c5ed17","Type":"ContainerDied","Data":"e49ebe998fb40dd210eb17a5877e56c00c97c672b73119ce70553edd70b29bf9"} Sep 11 04:15:03.134230 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:03.134197 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f8229c-29818335-cpn7p" Sep 11 04:15:04.001677 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:04.001643 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f8229c-29818335-cpn7p" Sep 11 04:15:04.001846 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:04.001682 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f8229c-29818335-cpn7p" event={"ID":"cc0b3d9d-eeba-48a4-a619-b01a96c5ed17","Type":"ContainerDied","Data":"aa5449cdd83ff71afb72893d4cf897fdc330f1f4325ec19fb01cb4291c66ec51"} Sep 11 04:15:04.001846 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:04.001737 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa5449cdd83ff71afb72893d4cf897fdc330f1f4325ec19fb01cb4291c66ec51" Sep 11 04:15:23.794567 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:23.794526 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-354ac6fe-7f56b65ddd-wwg4b"] Sep 11 04:15:23.795220 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:23.794799 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-354ac6fe-7f56b65ddd-wwg4b" podUID="8819fb73-856b-442b-a160-f9f51f4385ca" containerName="maas-api" containerID="cri-o://c717407e7c92414c737d1c8bf0205d5532c7c4beeb8f789cfa66f157b4062985" gracePeriod=30 Sep 11 04:15:23.827451 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:23.827419 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:15:23.855561 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:23.855529 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-354ac6fe-584dcccccc-sz2zv"] Sep 11 04:15:23.855752 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:23.855730 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-354ac6fe-584dcccccc-sz2zv" podUID="cfd1c84a-ab1c-454a-a40a-457c3986ab4d" containerName="payload-processing" containerID="cri-o://3957f8a387089878cca89421cb95324528fa9e9ea7da0cbb27e3dcc2813d9f8e" gracePeriod=30 Sep 11 04:15:23.926107 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:23.926036 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:15:24.023905 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:24.023868 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:15:24.088950 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:24.088908 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:15:24.098957 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:24.098927 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-354ac6fe-7f56b65ddd-wwg4b" Sep 11 04:15:24.100783 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:24.100760 2568 generic.go:356] "Generic (PLEG): container finished" podID="8819fb73-856b-442b-a160-f9f51f4385ca" containerID="c717407e7c92414c737d1c8bf0205d5532c7c4beeb8f789cfa66f157b4062985" exitCode=0 Sep 11 04:15:24.100899 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:24.100791 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-354ac6fe-7f56b65ddd-wwg4b" event={"ID":"8819fb73-856b-442b-a160-f9f51f4385ca","Type":"ContainerDied","Data":"c717407e7c92414c737d1c8bf0205d5532c7c4beeb8f789cfa66f157b4062985"} Sep 11 04:15:24.100899 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:24.100805 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-354ac6fe-7f56b65ddd-wwg4b" Sep 11 04:15:24.100899 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:24.100845 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-354ac6fe-7f56b65ddd-wwg4b" event={"ID":"8819fb73-856b-442b-a160-f9f51f4385ca","Type":"ContainerDied","Data":"161ba9b0da8d836a3a72b92866e3ce800bf408931508c3e414c475d79afc967d"} Sep 11 04:15:24.100899 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:24.100872 2568 scope.go:117] "RemoveContainer" containerID="c717407e7c92414c737d1c8bf0205d5532c7c4beeb8f789cfa66f157b4062985" Sep 11 04:15:24.102587 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:24.102568 2568 generic.go:356] "Generic (PLEG): container finished" podID="cfd1c84a-ab1c-454a-a40a-457c3986ab4d" containerID="3957f8a387089878cca89421cb95324528fa9e9ea7da0cbb27e3dcc2813d9f8e" exitCode=0 Sep 11 04:15:24.102711 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:24.102590 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-354ac6fe-584dcccccc-sz2zv" event={"ID":"cfd1c84a-ab1c-454a-a40a-457c3986ab4d","Type":"ContainerDied","Data":"3957f8a387089878cca89421cb95324528fa9e9ea7da0cbb27e3dcc2813d9f8e"} Sep 11 04:15:24.109707 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:24.109690 2568 scope.go:117] "RemoveContainer" containerID="c717407e7c92414c737d1c8bf0205d5532c7c4beeb8f789cfa66f157b4062985" Sep 11 04:15:24.110006 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:15:24.109986 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c717407e7c92414c737d1c8bf0205d5532c7c4beeb8f789cfa66f157b4062985\": container with ID starting with c717407e7c92414c737d1c8bf0205d5532c7c4beeb8f789cfa66f157b4062985 not found: ID does not exist" containerID="c717407e7c92414c737d1c8bf0205d5532c7c4beeb8f789cfa66f157b4062985" Sep 11 04:15:24.110080 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:24.110013 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c717407e7c92414c737d1c8bf0205d5532c7c4beeb8f789cfa66f157b4062985"} err="failed to get container status \"c717407e7c92414c737d1c8bf0205d5532c7c4beeb8f789cfa66f157b4062985\": rpc error: code = NotFound desc = could not find container \"c717407e7c92414c737d1c8bf0205d5532c7c4beeb8f789cfa66f157b4062985\": container with ID starting with c717407e7c92414c737d1c8bf0205d5532c7c4beeb8f789cfa66f157b4062985 not found: ID does not exist" Sep 11 04:15:24.146416 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:24.146384 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-354ac6fe-584dcccccc-sz2zv" Sep 11 04:15:24.227713 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:24.227671 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8819fb73-856b-442b-a160-f9f51f4385ca-metrics-certs\") pod \"8819fb73-856b-442b-a160-f9f51f4385ca\" (UID: \"8819fb73-856b-442b-a160-f9f51f4385ca\") " Sep 11 04:15:24.228004 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:24.227749 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zpr8f\" (UniqueName: \"kubernetes.io/projected/cfd1c84a-ab1c-454a-a40a-457c3986ab4d-kube-api-access-zpr8f\") pod \"cfd1c84a-ab1c-454a-a40a-457c3986ab4d\" (UID: \"cfd1c84a-ab1c-454a-a40a-457c3986ab4d\") " Sep 11 04:15:24.228004 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:24.227815 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqg7h\" (UniqueName: \"kubernetes.io/projected/8819fb73-856b-442b-a160-f9f51f4385ca-kube-api-access-pqg7h\") pod \"8819fb73-856b-442b-a160-f9f51f4385ca\" (UID: \"8819fb73-856b-442b-a160-f9f51f4385ca\") " Sep 11 04:15:24.228004 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:24.227859 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8819fb73-856b-442b-a160-f9f51f4385ca-openshift-service-ca\") pod \"8819fb73-856b-442b-a160-f9f51f4385ca\" (UID: \"8819fb73-856b-442b-a160-f9f51f4385ca\") " Sep 11 04:15:24.228004 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:24.227876 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfd1c84a-ab1c-454a-a40a-457c3986ab4d-plugins-config-volume\") pod \"cfd1c84a-ab1c-454a-a40a-457c3986ab4d\" (UID: \"cfd1c84a-ab1c-454a-a40a-457c3986ab4d\") " Sep 11 04:15:24.228004 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:24.227913 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8819fb73-856b-442b-a160-f9f51f4385ca-maas-api-tls\") pod \"8819fb73-856b-442b-a160-f9f51f4385ca\" (UID: \"8819fb73-856b-442b-a160-f9f51f4385ca\") " Sep 11 04:15:24.228386 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:24.228359 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfd1c84a-ab1c-454a-a40a-457c3986ab4d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cfd1c84a-ab1c-454a-a40a-457c3986ab4d" (UID: "cfd1c84a-ab1c-454a-a40a-457c3986ab4d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 04:15:24.228440 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:24.228355 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8819fb73-856b-442b-a160-f9f51f4385ca-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8819fb73-856b-442b-a160-f9f51f4385ca" (UID: "8819fb73-856b-442b-a160-f9f51f4385ca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 04:15:24.230125 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:24.230095 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8819fb73-856b-442b-a160-f9f51f4385ca-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8819fb73-856b-442b-a160-f9f51f4385ca" (UID: "8819fb73-856b-442b-a160-f9f51f4385ca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 04:15:24.230214 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:24.230123 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8819fb73-856b-442b-a160-f9f51f4385ca-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8819fb73-856b-442b-a160-f9f51f4385ca" (UID: "8819fb73-856b-442b-a160-f9f51f4385ca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 04:15:24.230388 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:24.230370 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfd1c84a-ab1c-454a-a40a-457c3986ab4d-kube-api-access-zpr8f" (OuterVolumeSpecName: "kube-api-access-zpr8f") pod "cfd1c84a-ab1c-454a-a40a-457c3986ab4d" (UID: "cfd1c84a-ab1c-454a-a40a-457c3986ab4d"). InnerVolumeSpecName "kube-api-access-zpr8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:15:24.230451 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:24.230396 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8819fb73-856b-442b-a160-f9f51f4385ca-kube-api-access-pqg7h" (OuterVolumeSpecName: "kube-api-access-pqg7h") pod "8819fb73-856b-442b-a160-f9f51f4385ca" (UID: "8819fb73-856b-442b-a160-f9f51f4385ca"). InnerVolumeSpecName "kube-api-access-pqg7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:15:24.329785 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:24.329700 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpr8f\" (UniqueName: \"kubernetes.io/projected/cfd1c84a-ab1c-454a-a40a-457c3986ab4d-kube-api-access-zpr8f\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:15:24.329785 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:24.329728 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqg7h\" (UniqueName: \"kubernetes.io/projected/8819fb73-856b-442b-a160-f9f51f4385ca-kube-api-access-pqg7h\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:15:24.329785 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:24.329740 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8819fb73-856b-442b-a160-f9f51f4385ca-openshift-service-ca\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:15:24.329785 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:24.329749 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cfd1c84a-ab1c-454a-a40a-457c3986ab4d-plugins-config-volume\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:15:24.329785 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:24.329758 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8819fb73-856b-442b-a160-f9f51f4385ca-maas-api-tls\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:15:24.329785 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:24.329765 2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8819fb73-856b-442b-a160-f9f51f4385ca-metrics-certs\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:15:24.423340 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:24.423311 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-354ac6fe-7f56b65ddd-wwg4b"] Sep 11 04:15:24.427369 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:24.427333 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-354ac6fe-7f56b65ddd-wwg4b"] Sep 11 04:15:25.109945 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:25.109862 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-354ac6fe-584dcccccc-sz2zv" Sep 11 04:15:25.110401 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:25.109869 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-354ac6fe-584dcccccc-sz2zv" event={"ID":"cfd1c84a-ab1c-454a-a40a-457c3986ab4d","Type":"ContainerDied","Data":"74c7ba372ebb9112851a68bf978c5409decbdb80488c99a379ef9f6c7eec242b"} Sep 11 04:15:25.110401 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:25.109998 2568 scope.go:117] "RemoveContainer" containerID="3957f8a387089878cca89421cb95324528fa9e9ea7da0cbb27e3dcc2813d9f8e" Sep 11 04:15:25.134505 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:25.134478 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-354ac6fe-584dcccccc-sz2zv"] Sep 11 04:15:25.139805 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:25.139781 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-354ac6fe-584dcccccc-sz2zv"] Sep 11 04:15:25.768298 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:25.768267 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8819fb73-856b-442b-a160-f9f51f4385ca" path="/var/lib/kubelet/pods/8819fb73-856b-442b-a160-f9f51f4385ca/volumes" Sep 11 04:15:25.768617 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:25.768604 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfd1c84a-ab1c-454a-a40a-457c3986ab4d" path="/var/lib/kubelet/pods/cfd1c84a-ab1c-454a-a40a-457c3986ab4d/volumes" Sep 11 04:15:27.402634 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:27.402583 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:15:32.427161 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:32.427070 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:15:35.958463 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:35.958426 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:15:36.060669 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:15:36.060636 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:16:05.932942 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:05.932909 2568 scope.go:117] "RemoveContainer" containerID="c5c2beba3104a9de5ce9f872409ed027689757c44a213405feb300e2b524ee1c" Sep 11 04:16:08.801912 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:08.801878 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:16:09.698243 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:09.698209 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:16:11.327800 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:11.327761 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8080-1b7a74"] Sep 11 04:16:11.328209 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:11.328168 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8819fb73-856b-442b-a160-f9f51f4385ca" containerName="maas-api" Sep 11 04:16:11.328209 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:11.328180 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="8819fb73-856b-442b-a160-f9f51f4385ca" containerName="maas-api" Sep 11 04:16:11.328209 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:11.328190 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfd1c84a-ab1c-454a-a40a-457c3986ab4d" containerName="payload-processing" Sep 11 04:16:11.328209 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:11.328196 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfd1c84a-ab1c-454a-a40a-457c3986ab4d" containerName="payload-processing" Sep 11 04:16:11.328209 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:11.328203 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc0b3d9d-eeba-48a4-a619-b01a96c5ed17" containerName="cleanup" Sep 11 04:16:11.328209 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:11.328208 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc0b3d9d-eeba-48a4-a619-b01a96c5ed17" containerName="cleanup" Sep 11 04:16:11.328408 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:11.328279 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="8819fb73-856b-442b-a160-f9f51f4385ca" containerName="maas-api" Sep 11 04:16:11.328408 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:11.328288 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc0b3d9d-eeba-48a4-a619-b01a96c5ed17" containerName="cleanup" Sep 11 04:16:11.328408 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:11.328296 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfd1c84a-ab1c-454a-a40a-457c3986ab4d" containerName="payload-processing" Sep 11 04:16:11.332810 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:11.332793 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8080-1b7a74" Sep 11 04:16:11.335454 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:11.335348 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zqmz7\"" Sep 11 04:16:11.337764 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:11.337738 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8080-1b7a74"] Sep 11 04:16:11.434756 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:11.434695 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhl4c\" (UniqueName: \"kubernetes.io/projected/05098df5-8a5a-453c-a56d-3ba996dd44d4-kube-api-access-zhl4c\") pod \"test-curl-8080-1b7a74\" (UID: \"05098df5-8a5a-453c-a56d-3ba996dd44d4\") " pod="opendatahub/test-curl-8080-1b7a74" Sep 11 04:16:11.535724 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:11.535677 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhl4c\" (UniqueName: \"kubernetes.io/projected/05098df5-8a5a-453c-a56d-3ba996dd44d4-kube-api-access-zhl4c\") pod \"test-curl-8080-1b7a74\" (UID: \"05098df5-8a5a-453c-a56d-3ba996dd44d4\") " pod="opendatahub/test-curl-8080-1b7a74" Sep 11 04:16:11.543954 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:11.543921 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhl4c\" (UniqueName: \"kubernetes.io/projected/05098df5-8a5a-453c-a56d-3ba996dd44d4-kube-api-access-zhl4c\") pod \"test-curl-8080-1b7a74\" (UID: \"05098df5-8a5a-453c-a56d-3ba996dd44d4\") " pod="opendatahub/test-curl-8080-1b7a74" Sep 11 04:16:11.643769 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:11.643675 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8080-1b7a74" Sep 11 04:16:11.918007 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:11.917983 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8080-1b7a74"] Sep 11 04:16:11.920193 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:16:11.920158 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05098df5_8a5a_453c_a56d_3ba996dd44d4.slice/crio-f5c22646e42e4a40b537337248d48b632880e49942f59bdc244c805029e137b3 WatchSource:0}: Error finding container f5c22646e42e4a40b537337248d48b632880e49942f59bdc244c805029e137b3: Status 404 returned error can't find the container with id f5c22646e42e4a40b537337248d48b632880e49942f59bdc244c805029e137b3 Sep 11 04:16:12.369647 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:12.369544 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8080-1b7a74" event={"ID":"05098df5-8a5a-453c-a56d-3ba996dd44d4","Type":"ContainerStarted","Data":"e47ddd193385fa39c9a4eb81b0e4bd78209264029ebb023a2dbc7b78c834ddfa"} Sep 11 04:16:12.369647 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:12.369598 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8080-1b7a74" event={"ID":"05098df5-8a5a-453c-a56d-3ba996dd44d4","Type":"ContainerStarted","Data":"f5c22646e42e4a40b537337248d48b632880e49942f59bdc244c805029e137b3"} Sep 11 04:16:12.384162 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:12.384110 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8080-1b7a74" podStartSLOduration=1.384095483 podStartE2EDuration="1.384095483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:16:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:16:12.383148124 +0000 UTC m=+1387.136779528" watchObservedRunningTime="2026-09-11 04:16:12.384095483 +0000 UTC m=+1387.137726890" Sep 11 04:16:13.008968 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:13.008930 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8080-1b7a74"] Sep 11 04:16:13.013338 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:13.013309 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8080-1b7a74"] Sep 11 04:16:13.374408 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:16:13.374379 2568 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8080-1b7a74" container="test-curl-8080-1b7a74" Sep 11 04:16:14.379386 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:14.379343 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8080-1b7a74" podUID="05098df5-8a5a-453c-a56d-3ba996dd44d4" containerName="test-curl-8080-1b7a74" containerID="cri-o://e47ddd193385fa39c9a4eb81b0e4bd78209264029ebb023a2dbc7b78c834ddfa" gracePeriod=2 Sep 11 04:16:15.318791 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:15.318752 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:16:16.565341 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:16.565311 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8080-1b7a74" Sep 11 04:16:16.680995 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:16.680958 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zhl4c\" (UniqueName: \"kubernetes.io/projected/05098df5-8a5a-453c-a56d-3ba996dd44d4-kube-api-access-zhl4c\") pod \"05098df5-8a5a-453c-a56d-3ba996dd44d4\" (UID: \"05098df5-8a5a-453c-a56d-3ba996dd44d4\") " Sep 11 04:16:16.683191 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:16.683157 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05098df5-8a5a-453c-a56d-3ba996dd44d4-kube-api-access-zhl4c" (OuterVolumeSpecName: "kube-api-access-zhl4c") pod "05098df5-8a5a-453c-a56d-3ba996dd44d4" (UID: "05098df5-8a5a-453c-a56d-3ba996dd44d4"). InnerVolumeSpecName "kube-api-access-zhl4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:16:16.781729 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:16.781694 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhl4c\" (UniqueName: \"kubernetes.io/projected/05098df5-8a5a-453c-a56d-3ba996dd44d4-kube-api-access-zhl4c\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:16:17.395254 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:17.395220 2568 generic.go:356] "Generic (PLEG): container finished" podID="05098df5-8a5a-453c-a56d-3ba996dd44d4" containerID="e47ddd193385fa39c9a4eb81b0e4bd78209264029ebb023a2dbc7b78c834ddfa" exitCode=137 Sep 11 04:16:17.395411 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:17.395259 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8080-1b7a74" Sep 11 04:16:17.395411 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:17.395319 2568 scope.go:117] "RemoveContainer" containerID="e47ddd193385fa39c9a4eb81b0e4bd78209264029ebb023a2dbc7b78c834ddfa" Sep 11 04:16:17.403948 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:17.403926 2568 scope.go:117] "RemoveContainer" containerID="e47ddd193385fa39c9a4eb81b0e4bd78209264029ebb023a2dbc7b78c834ddfa" Sep 11 04:16:17.404240 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:16:17.404220 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e47ddd193385fa39c9a4eb81b0e4bd78209264029ebb023a2dbc7b78c834ddfa\": container with ID starting with e47ddd193385fa39c9a4eb81b0e4bd78209264029ebb023a2dbc7b78c834ddfa not found: ID does not exist" containerID="e47ddd193385fa39c9a4eb81b0e4bd78209264029ebb023a2dbc7b78c834ddfa" Sep 11 04:16:17.404309 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:17.404252 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e47ddd193385fa39c9a4eb81b0e4bd78209264029ebb023a2dbc7b78c834ddfa"} err="failed to get container status \"e47ddd193385fa39c9a4eb81b0e4bd78209264029ebb023a2dbc7b78c834ddfa\": rpc error: code = NotFound desc = could not find container \"e47ddd193385fa39c9a4eb81b0e4bd78209264029ebb023a2dbc7b78c834ddfa\": container with ID starting with e47ddd193385fa39c9a4eb81b0e4bd78209264029ebb023a2dbc7b78c834ddfa not found: ID does not exist" Sep 11 04:16:17.405519 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:16:17.405495 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8080-1b7a74\" is forbidden: User \"system:node:ip-10-0-140-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-30.ec2.internal' and this object" podUID="05098df5-8a5a-453c-a56d-3ba996dd44d4" pod="opendatahub/test-curl-8080-1b7a74" Sep 11 04:16:17.768321 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:17.768292 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05098df5-8a5a-453c-a56d-3ba996dd44d4" path="/var/lib/kubelet/pods/05098df5-8a5a-453c-a56d-3ba996dd44d4/volumes" Sep 11 04:16:23.803182 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:23.803145 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:16:25.104341 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:25.104303 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:16:26.507867 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:26.507832 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:16:29.894802 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:29.894765 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:16:29.984934 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:29.984895 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:16:30.162551 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:30.162510 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:16:31.511683 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:31.511637 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:16:31.611826 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:31.611781 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:16:31.711727 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:31.711688 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:16:32.360298 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:32.360251 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:16:32.461150 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:32.461101 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:16:32.569510 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:32.569471 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:16:33.937572 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:33.937527 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:16:39.699772 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:39.699736 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:16:39.774384 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:39.774339 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:16:40.411156 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:40.411117 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:16:46.448135 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:46.448093 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:16:46.671441 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:46.671398 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:16:46.798749 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:46.798656 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:16:49.109770 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:49.109736 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:16:57.256995 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:57.256955 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:16:57.504129 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:16:57.504092 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:17:18.605938 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:18.605899 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:17:22.718343 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:22.718308 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-9342ed/sub-test-model-9342ed-kserve-76d7db7b5-shdsk"] Sep 11 04:17:22.718755 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:22.718720 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05098df5-8a5a-453c-a56d-3ba996dd44d4" containerName="test-curl-8080-1b7a74" Sep 11 04:17:22.718755 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:22.718735 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="05098df5-8a5a-453c-a56d-3ba996dd44d4" containerName="test-curl-8080-1b7a74" Sep 11 04:17:22.718912 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:22.718896 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="05098df5-8a5a-453c-a56d-3ba996dd44d4" containerName="test-curl-8080-1b7a74" Sep 11 04:17:22.723179 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:22.723161 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9342ed/sub-test-model-9342ed-kserve-76d7db7b5-shdsk" Sep 11 04:17:22.725812 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:22.725791 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-9342ed\"/\"openshift-service-ca.crt\"" Sep 11 04:17:22.726688 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:22.726663 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-9342ed\"/\"kube-root-ca.crt\"" Sep 11 04:17:22.726788 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:22.726725 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-9342ed\"/\"default-dockercfg-552lm\"" Sep 11 04:17:22.726788 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:22.726743 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-9342ed\"/\"sub-test-model-9342ed-kserve-self-signed-certs\"" Sep 11 04:17:22.735066 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:22.735022 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9342ed/sub-test-model-9342ed-kserve-76d7db7b5-shdsk"] Sep 11 04:17:22.799749 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:22.799717 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:17:22.853777 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:22.853734 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/082edca2-cfea-4818-9e08-277a51456382-tmp-dir\") pod \"sub-test-model-9342ed-kserve-76d7db7b5-shdsk\" (UID: \"082edca2-cfea-4818-9e08-277a51456382\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/sub-test-model-9342ed-kserve-76d7db7b5-shdsk" Sep 11 04:17:22.853983 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:22.853864 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/082edca2-cfea-4818-9e08-277a51456382-dshm\") pod \"sub-test-model-9342ed-kserve-76d7db7b5-shdsk\" (UID: \"082edca2-cfea-4818-9e08-277a51456382\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/sub-test-model-9342ed-kserve-76d7db7b5-shdsk" Sep 11 04:17:22.853983 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:22.853915 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/082edca2-cfea-4818-9e08-277a51456382-tls-certs\") pod \"sub-test-model-9342ed-kserve-76d7db7b5-shdsk\" (UID: \"082edca2-cfea-4818-9e08-277a51456382\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/sub-test-model-9342ed-kserve-76d7db7b5-shdsk" Sep 11 04:17:22.854166 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:22.853989 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw756\" (UniqueName: \"kubernetes.io/projected/082edca2-cfea-4818-9e08-277a51456382-kube-api-access-mw756\") pod \"sub-test-model-9342ed-kserve-76d7db7b5-shdsk\" (UID: \"082edca2-cfea-4818-9e08-277a51456382\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/sub-test-model-9342ed-kserve-76d7db7b5-shdsk" Sep 11 04:17:22.854166 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:22.854035 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/082edca2-cfea-4818-9e08-277a51456382-home\") pod \"sub-test-model-9342ed-kserve-76d7db7b5-shdsk\" (UID: \"082edca2-cfea-4818-9e08-277a51456382\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/sub-test-model-9342ed-kserve-76d7db7b5-shdsk" Sep 11 04:17:22.854277 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:22.854174 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/082edca2-cfea-4818-9e08-277a51456382-model-cache\") pod \"sub-test-model-9342ed-kserve-76d7db7b5-shdsk\" (UID: \"082edca2-cfea-4818-9e08-277a51456382\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/sub-test-model-9342ed-kserve-76d7db7b5-shdsk" Sep 11 04:17:22.955542 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:22.955512 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/082edca2-cfea-4818-9e08-277a51456382-dshm\") pod \"sub-test-model-9342ed-kserve-76d7db7b5-shdsk\" (UID: \"082edca2-cfea-4818-9e08-277a51456382\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/sub-test-model-9342ed-kserve-76d7db7b5-shdsk" Sep 11 04:17:22.955699 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:22.955559 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/082edca2-cfea-4818-9e08-277a51456382-tls-certs\") pod \"sub-test-model-9342ed-kserve-76d7db7b5-shdsk\" (UID: \"082edca2-cfea-4818-9e08-277a51456382\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/sub-test-model-9342ed-kserve-76d7db7b5-shdsk" Sep 11 04:17:22.955699 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:22.955597 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw756\" (UniqueName: \"kubernetes.io/projected/082edca2-cfea-4818-9e08-277a51456382-kube-api-access-mw756\") pod \"sub-test-model-9342ed-kserve-76d7db7b5-shdsk\" (UID: \"082edca2-cfea-4818-9e08-277a51456382\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/sub-test-model-9342ed-kserve-76d7db7b5-shdsk" Sep 11 04:17:22.955699 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:22.955617 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/082edca2-cfea-4818-9e08-277a51456382-home\") pod \"sub-test-model-9342ed-kserve-76d7db7b5-shdsk\" (UID: \"082edca2-cfea-4818-9e08-277a51456382\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/sub-test-model-9342ed-kserve-76d7db7b5-shdsk" Sep 11 04:17:22.955699 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:22.955663 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/082edca2-cfea-4818-9e08-277a51456382-model-cache\") pod \"sub-test-model-9342ed-kserve-76d7db7b5-shdsk\" (UID: \"082edca2-cfea-4818-9e08-277a51456382\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/sub-test-model-9342ed-kserve-76d7db7b5-shdsk" Sep 11 04:17:22.955903 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:22.955727 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/082edca2-cfea-4818-9e08-277a51456382-tmp-dir\") pod \"sub-test-model-9342ed-kserve-76d7db7b5-shdsk\" (UID: \"082edca2-cfea-4818-9e08-277a51456382\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/sub-test-model-9342ed-kserve-76d7db7b5-shdsk" Sep 11 04:17:22.956127 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:22.956099 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/082edca2-cfea-4818-9e08-277a51456382-home\") pod \"sub-test-model-9342ed-kserve-76d7db7b5-shdsk\" (UID: \"082edca2-cfea-4818-9e08-277a51456382\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/sub-test-model-9342ed-kserve-76d7db7b5-shdsk" Sep 11 04:17:22.956241 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:22.956129 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/082edca2-cfea-4818-9e08-277a51456382-tmp-dir\") pod \"sub-test-model-9342ed-kserve-76d7db7b5-shdsk\" (UID: \"082edca2-cfea-4818-9e08-277a51456382\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/sub-test-model-9342ed-kserve-76d7db7b5-shdsk" Sep 11 04:17:22.956241 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:22.956163 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/082edca2-cfea-4818-9e08-277a51456382-model-cache\") pod \"sub-test-model-9342ed-kserve-76d7db7b5-shdsk\" (UID: \"082edca2-cfea-4818-9e08-277a51456382\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/sub-test-model-9342ed-kserve-76d7db7b5-shdsk" Sep 11 04:17:22.957880 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:22.957859 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/082edca2-cfea-4818-9e08-277a51456382-dshm\") pod \"sub-test-model-9342ed-kserve-76d7db7b5-shdsk\" (UID: \"082edca2-cfea-4818-9e08-277a51456382\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/sub-test-model-9342ed-kserve-76d7db7b5-shdsk" Sep 11 04:17:22.957981 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:22.957964 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/082edca2-cfea-4818-9e08-277a51456382-tls-certs\") pod \"sub-test-model-9342ed-kserve-76d7db7b5-shdsk\" (UID: \"082edca2-cfea-4818-9e08-277a51456382\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/sub-test-model-9342ed-kserve-76d7db7b5-shdsk" Sep 11 04:17:22.965571 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:22.965551 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw756\" (UniqueName: \"kubernetes.io/projected/082edca2-cfea-4818-9e08-277a51456382-kube-api-access-mw756\") pod \"sub-test-model-9342ed-kserve-76d7db7b5-shdsk\" (UID: \"082edca2-cfea-4818-9e08-277a51456382\") " pod="ai-tenant-e2e-shared-b-w3-9342ed/sub-test-model-9342ed-kserve-76d7db7b5-shdsk" Sep 11 04:17:23.031926 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:23.031852 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9342ed/sub-test-model-9342ed-kserve-76d7db7b5-shdsk" Sep 11 04:17:23.303299 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:23.303271 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9342ed/sub-test-model-9342ed-kserve-76d7db7b5-shdsk"] Sep 11 04:17:23.305508 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:17:23.305476 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod082edca2_cfea_4818_9e08_277a51456382.slice/crio-5aa61542bb14e96aeff3923a2e644edc9c3c8ea834af67240aa17becf70055af WatchSource:0}: Error finding container 5aa61542bb14e96aeff3923a2e644edc9c3c8ea834af67240aa17becf70055af: Status 404 returned error can't find the container with id 5aa61542bb14e96aeff3923a2e644edc9c3c8ea834af67240aa17becf70055af Sep 11 04:17:23.727130 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:23.727091 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9342ed/sub-test-model-9342ed-kserve-76d7db7b5-shdsk" event={"ID":"082edca2-cfea-4818-9e08-277a51456382","Type":"ContainerStarted","Data":"5aa61542bb14e96aeff3923a2e644edc9c3c8ea834af67240aa17becf70055af"} Sep 11 04:17:24.733843 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:24.733804 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9342ed/sub-test-model-9342ed-kserve-76d7db7b5-shdsk" event={"ID":"082edca2-cfea-4818-9e08-277a51456382","Type":"ContainerStarted","Data":"a289b9c03191385bfa6e64ce1c777865ba95fe6d6df0d11441a75047c302a6cd"} Sep 11 04:17:24.763652 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:24.763603 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-9342ed/sub-test-model-9342ed-kserve-76d7db7b5-shdsk" podStartSLOduration=1.820697832 podStartE2EDuration="2.763587174s" totalImagesPullingTime="942.889342ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:17:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:17:24.760347633 +0000 UTC m=+1459.513979034" watchObservedRunningTime="2026-09-11 04:17:24.763587174 +0000 UTC m=+1459.517218574" Sep 11 04:17:25.335799 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:25.335760 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:17:25.432951 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:25.432914 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:17:25.717456 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:25.717420 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:17:26.133628 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:26.133541 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:17:26.229817 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:26.229781 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:17:29.487329 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:29.487287 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:17:29.580954 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:29.580918 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:17:30.625090 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:30.625041 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bf30f8-587697d65f-nd4kb"] Sep 11 04:17:30.629342 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:30.629315 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf30f8-587697d65f-nd4kb" Sep 11 04:17:30.631858 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:30.631835 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-bf30f8-serving-cert\"" Sep 11 04:17:30.642857 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:30.642830 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bf30f8-587697d65f-nd4kb"] Sep 11 04:17:30.660217 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:30.660186 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:17:30.725279 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:30.725244 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0613c781-47be-4eb7-866d-e00a9a4dd027-maas-api-tls\") pod \"maas-api-e2e-api-a-bf30f8-587697d65f-nd4kb\" (UID: \"0613c781-47be-4eb7-866d-e00a9a4dd027\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf30f8-587697d65f-nd4kb" Sep 11 04:17:30.725436 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:30.725286 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cflrg\" (UniqueName: \"kubernetes.io/projected/0613c781-47be-4eb7-866d-e00a9a4dd027-kube-api-access-cflrg\") pod \"maas-api-e2e-api-a-bf30f8-587697d65f-nd4kb\" (UID: \"0613c781-47be-4eb7-866d-e00a9a4dd027\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf30f8-587697d65f-nd4kb" Sep 11 04:17:30.725436 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:30.725397 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0613c781-47be-4eb7-866d-e00a9a4dd027-metrics-certs\") pod \"maas-api-e2e-api-a-bf30f8-587697d65f-nd4kb\" (UID: \"0613c781-47be-4eb7-866d-e00a9a4dd027\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf30f8-587697d65f-nd4kb" Sep 11 04:17:30.725618 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:30.725440 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0613c781-47be-4eb7-866d-e00a9a4dd027-openshift-service-ca\") pod \"maas-api-e2e-api-a-bf30f8-587697d65f-nd4kb\" (UID: \"0613c781-47be-4eb7-866d-e00a9a4dd027\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf30f8-587697d65f-nd4kb" Sep 11 04:17:30.756326 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:30.756288 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:17:30.826282 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:30.826252 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0613c781-47be-4eb7-866d-e00a9a4dd027-maas-api-tls\") pod \"maas-api-e2e-api-a-bf30f8-587697d65f-nd4kb\" (UID: \"0613c781-47be-4eb7-866d-e00a9a4dd027\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf30f8-587697d65f-nd4kb" Sep 11 04:17:30.826470 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:30.826288 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cflrg\" (UniqueName: \"kubernetes.io/projected/0613c781-47be-4eb7-866d-e00a9a4dd027-kube-api-access-cflrg\") pod \"maas-api-e2e-api-a-bf30f8-587697d65f-nd4kb\" (UID: \"0613c781-47be-4eb7-866d-e00a9a4dd027\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf30f8-587697d65f-nd4kb" Sep 11 04:17:30.826470 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:30.826330 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0613c781-47be-4eb7-866d-e00a9a4dd027-metrics-certs\") pod \"maas-api-e2e-api-a-bf30f8-587697d65f-nd4kb\" (UID: \"0613c781-47be-4eb7-866d-e00a9a4dd027\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf30f8-587697d65f-nd4kb" Sep 11 04:17:30.826470 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:30.826354 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0613c781-47be-4eb7-866d-e00a9a4dd027-openshift-service-ca\") pod \"maas-api-e2e-api-a-bf30f8-587697d65f-nd4kb\" (UID: \"0613c781-47be-4eb7-866d-e00a9a4dd027\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf30f8-587697d65f-nd4kb" Sep 11 04:17:30.826470 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:17:30.826406 2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-bf30f8-serving-cert: secret "maas-api-e2e-api-a-bf30f8-serving-cert" not found Sep 11 04:17:30.826709 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:17:30.826481 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0613c781-47be-4eb7-866d-e00a9a4dd027-maas-api-tls podName:0613c781-47be-4eb7-866d-e00a9a4dd027 nodeName:}" failed. No retries permitted until 2026-09-11 04:17:31.326460007 +0000 UTC m=+1466.080091403 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0613c781-47be-4eb7-866d-e00a9a4dd027-maas-api-tls") pod "maas-api-e2e-api-a-bf30f8-587697d65f-nd4kb" (UID: "0613c781-47be-4eb7-866d-e00a9a4dd027") : secret "maas-api-e2e-api-a-bf30f8-serving-cert" not found Sep 11 04:17:30.826962 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:30.826945 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0613c781-47be-4eb7-866d-e00a9a4dd027-openshift-service-ca\") pod \"maas-api-e2e-api-a-bf30f8-587697d65f-nd4kb\" (UID: \"0613c781-47be-4eb7-866d-e00a9a4dd027\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf30f8-587697d65f-nd4kb" Sep 11 04:17:30.828909 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:30.828889 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0613c781-47be-4eb7-866d-e00a9a4dd027-metrics-certs\") pod \"maas-api-e2e-api-a-bf30f8-587697d65f-nd4kb\" (UID: \"0613c781-47be-4eb7-866d-e00a9a4dd027\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf30f8-587697d65f-nd4kb" Sep 11 04:17:30.835566 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:30.835529 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cflrg\" (UniqueName: \"kubernetes.io/projected/0613c781-47be-4eb7-866d-e00a9a4dd027-kube-api-access-cflrg\") pod \"maas-api-e2e-api-a-bf30f8-587697d65f-nd4kb\" (UID: \"0613c781-47be-4eb7-866d-e00a9a4dd027\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf30f8-587697d65f-nd4kb" Sep 11 04:17:31.153792 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:31.153753 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:17:31.299195 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:31.299156 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:17:31.331304 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:31.331268 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0613c781-47be-4eb7-866d-e00a9a4dd027-maas-api-tls\") pod \"maas-api-e2e-api-a-bf30f8-587697d65f-nd4kb\" (UID: \"0613c781-47be-4eb7-866d-e00a9a4dd027\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf30f8-587697d65f-nd4kb" Sep 11 04:17:31.333672 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:31.333647 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0613c781-47be-4eb7-866d-e00a9a4dd027-maas-api-tls\") pod \"maas-api-e2e-api-a-bf30f8-587697d65f-nd4kb\" (UID: \"0613c781-47be-4eb7-866d-e00a9a4dd027\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf30f8-587697d65f-nd4kb" Sep 11 04:17:31.391018 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:31.390978 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:17:31.544760 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:31.544671 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf30f8-587697d65f-nd4kb" Sep 11 04:17:31.821141 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:31.821112 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bf30f8-587697d65f-nd4kb"] Sep 11 04:17:31.823303 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:17:31.823271 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0613c781_47be_4eb7_866d_e00a9a4dd027.slice/crio-26d1424886875d74c9d341da040601dddc008b9cbe87d7c3b655e7a4a974d45a WatchSource:0}: Error finding container 26d1424886875d74c9d341da040601dddc008b9cbe87d7c3b655e7a4a974d45a: Status 404 returned error can't find the container with id 26d1424886875d74c9d341da040601dddc008b9cbe87d7c3b655e7a4a974d45a Sep 11 04:17:31.824568 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:31.824550 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 04:17:32.777896 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:32.777845 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf30f8-587697d65f-nd4kb" event={"ID":"0613c781-47be-4eb7-866d-e00a9a4dd027","Type":"ContainerStarted","Data":"20cd17957ac63af246aa70b51ba57038877eb8704a69297b951eefdef317191a"} Sep 11 04:17:32.778120 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:32.777936 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf30f8-587697d65f-nd4kb" event={"ID":"0613c781-47be-4eb7-866d-e00a9a4dd027","Type":"ContainerStarted","Data":"26d1424886875d74c9d341da040601dddc008b9cbe87d7c3b655e7a4a974d45a"} Sep 11 04:17:32.778120 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:32.777963 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf30f8-587697d65f-nd4kb" Sep 11 04:17:32.796065 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:32.795998 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf30f8-587697d65f-nd4kb" podStartSLOduration=2.325719266 podStartE2EDuration="2.795983862s" totalImagesPullingTime="470.264596ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:17:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:17:32.794341782 +0000 UTC m=+1467.547973185" watchObservedRunningTime="2026-09-11 04:17:32.795983862 +0000 UTC m=+1467.549615263" Sep 11 04:17:33.032354 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:33.032273 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-9342ed/sub-test-model-9342ed-kserve-76d7db7b5-shdsk" Sep 11 04:17:33.032711 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:33.032360 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-9342ed/sub-test-model-9342ed-kserve-76d7db7b5-shdsk" Sep 11 04:17:33.045262 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:33.045234 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-9342ed/sub-test-model-9342ed-kserve-76d7db7b5-shdsk" Sep 11 04:17:33.794759 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:33.794726 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-9342ed/sub-test-model-9342ed-kserve-76d7db7b5-shdsk" Sep 11 04:17:38.789169 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:38.789137 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf30f8-587697d65f-nd4kb" Sep 11 04:17:46.312124 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:46.312087 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:17:52.649757 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:52.649723 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:17:54.512730 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:54.512699 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:17:54.984608 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:54.984574 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:17:55.310834 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:55.310746 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:17:55.420591 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:55.420544 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:17:58.671804 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:58.671771 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:17:58.773740 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:58.773701 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:17:59.807605 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:59.807570 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-599baf-59dd5c9655-wktlh"] Sep 11 04:17:59.812653 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:59.812626 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-599baf-59dd5c9655-wktlh" Sep 11 04:17:59.816257 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:59.816235 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-599baf-serving-cert\"" Sep 11 04:17:59.828022 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:59.827995 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-599baf-59dd5c9655-wktlh"] Sep 11 04:17:59.852093 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:59.852039 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:17:59.900035 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:59.899998 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b643f98-3070-4c1a-834f-0f8dc59db105-openshift-service-ca\") pod \"maas-api-e2e-api-b-599baf-59dd5c9655-wktlh\" (UID: \"6b643f98-3070-4c1a-834f-0f8dc59db105\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-599baf-59dd5c9655-wktlh" Sep 11 04:17:59.900265 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:59.900074 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b643f98-3070-4c1a-834f-0f8dc59db105-metrics-certs\") pod \"maas-api-e2e-api-b-599baf-59dd5c9655-wktlh\" (UID: \"6b643f98-3070-4c1a-834f-0f8dc59db105\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-599baf-59dd5c9655-wktlh" Sep 11 04:17:59.900265 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:59.900171 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b643f98-3070-4c1a-834f-0f8dc59db105-maas-api-tls\") pod \"maas-api-e2e-api-b-599baf-59dd5c9655-wktlh\" (UID: \"6b643f98-3070-4c1a-834f-0f8dc59db105\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-599baf-59dd5c9655-wktlh" Sep 11 04:17:59.900265 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:59.900212 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slxvm\" (UniqueName: \"kubernetes.io/projected/6b643f98-3070-4c1a-834f-0f8dc59db105-kube-api-access-slxvm\") pod \"maas-api-e2e-api-b-599baf-59dd5c9655-wktlh\" (UID: \"6b643f98-3070-4c1a-834f-0f8dc59db105\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-599baf-59dd5c9655-wktlh" Sep 11 04:17:59.953363 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:17:59.953323 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:18:00.000686 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:00.000646 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b643f98-3070-4c1a-834f-0f8dc59db105-openshift-service-ca\") pod \"maas-api-e2e-api-b-599baf-59dd5c9655-wktlh\" (UID: \"6b643f98-3070-4c1a-834f-0f8dc59db105\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-599baf-59dd5c9655-wktlh" Sep 11 04:18:00.000888 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:00.000700 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b643f98-3070-4c1a-834f-0f8dc59db105-metrics-certs\") pod \"maas-api-e2e-api-b-599baf-59dd5c9655-wktlh\" (UID: \"6b643f98-3070-4c1a-834f-0f8dc59db105\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-599baf-59dd5c9655-wktlh" Sep 11 04:18:00.000888 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:00.000732 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b643f98-3070-4c1a-834f-0f8dc59db105-maas-api-tls\") pod \"maas-api-e2e-api-b-599baf-59dd5c9655-wktlh\" (UID: \"6b643f98-3070-4c1a-834f-0f8dc59db105\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-599baf-59dd5c9655-wktlh" Sep 11 04:18:00.000888 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:00.000750 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slxvm\" (UniqueName: \"kubernetes.io/projected/6b643f98-3070-4c1a-834f-0f8dc59db105-kube-api-access-slxvm\") pod \"maas-api-e2e-api-b-599baf-59dd5c9655-wktlh\" (UID: \"6b643f98-3070-4c1a-834f-0f8dc59db105\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-599baf-59dd5c9655-wktlh" Sep 11 04:18:00.001107 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:18:00.000898 2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-599baf-serving-cert: secret "maas-api-e2e-api-b-599baf-serving-cert" not found Sep 11 04:18:00.001107 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:18:00.000987 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b643f98-3070-4c1a-834f-0f8dc59db105-maas-api-tls podName:6b643f98-3070-4c1a-834f-0f8dc59db105 nodeName:}" failed. No retries permitted until 2026-09-11 04:18:00.500965366 +0000 UTC m=+1495.254596775 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6b643f98-3070-4c1a-834f-0f8dc59db105-maas-api-tls") pod "maas-api-e2e-api-b-599baf-59dd5c9655-wktlh" (UID: "6b643f98-3070-4c1a-834f-0f8dc59db105") : secret "maas-api-e2e-api-b-599baf-serving-cert" not found Sep 11 04:18:00.001444 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:00.001423 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b643f98-3070-4c1a-834f-0f8dc59db105-openshift-service-ca\") pod \"maas-api-e2e-api-b-599baf-59dd5c9655-wktlh\" (UID: \"6b643f98-3070-4c1a-834f-0f8dc59db105\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-599baf-59dd5c9655-wktlh" Sep 11 04:18:00.003325 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:00.003306 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b643f98-3070-4c1a-834f-0f8dc59db105-metrics-certs\") pod \"maas-api-e2e-api-b-599baf-59dd5c9655-wktlh\" (UID: \"6b643f98-3070-4c1a-834f-0f8dc59db105\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-599baf-59dd5c9655-wktlh" Sep 11 04:18:00.008970 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:00.008948 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slxvm\" (UniqueName: \"kubernetes.io/projected/6b643f98-3070-4c1a-834f-0f8dc59db105-kube-api-access-slxvm\") pod \"maas-api-e2e-api-b-599baf-59dd5c9655-wktlh\" (UID: \"6b643f98-3070-4c1a-834f-0f8dc59db105\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-599baf-59dd5c9655-wktlh" Sep 11 04:18:00.352857 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:00.352799 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:18:00.451699 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:00.451659 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:18:00.510561 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:00.510526 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b643f98-3070-4c1a-834f-0f8dc59db105-maas-api-tls\") pod \"maas-api-e2e-api-b-599baf-59dd5c9655-wktlh\" (UID: \"6b643f98-3070-4c1a-834f-0f8dc59db105\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-599baf-59dd5c9655-wktlh" Sep 11 04:18:00.513273 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:00.513247 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b643f98-3070-4c1a-834f-0f8dc59db105-maas-api-tls\") pod \"maas-api-e2e-api-b-599baf-59dd5c9655-wktlh\" (UID: \"6b643f98-3070-4c1a-834f-0f8dc59db105\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-599baf-59dd5c9655-wktlh" Sep 11 04:18:00.724912 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:00.724875 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-599baf-59dd5c9655-wktlh" Sep 11 04:18:00.998168 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:00.998142 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-599baf-59dd5c9655-wktlh"] Sep 11 04:18:01.000114 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:18:01.000086 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b643f98_3070_4c1a_834f_0f8dc59db105.slice/crio-9a5e4ec792063db05d7621f4c7692847c562f3dfe549c22ba23efe52d82e40a3 WatchSource:0}: Error finding container 9a5e4ec792063db05d7621f4c7692847c562f3dfe549c22ba23efe52d82e40a3: Status 404 returned error can't find the container with id 9a5e4ec792063db05d7621f4c7692847c562f3dfe549c22ba23efe52d82e40a3 Sep 11 04:18:01.933376 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:01.933341 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-599baf-59dd5c9655-wktlh" event={"ID":"6b643f98-3070-4c1a-834f-0f8dc59db105","Type":"ContainerStarted","Data":"770cf1d1a0621b69fd2e8b30c542f8948c7ffb4d83c49b9cc4663c03f2c8d5fc"} Sep 11 04:18:01.933540 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:01.933396 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-599baf-59dd5c9655-wktlh" event={"ID":"6b643f98-3070-4c1a-834f-0f8dc59db105","Type":"ContainerStarted","Data":"9a5e4ec792063db05d7621f4c7692847c562f3dfe549c22ba23efe52d82e40a3"} Sep 11 04:18:01.933540 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:01.933459 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-599baf-59dd5c9655-wktlh" Sep 11 04:18:01.953126 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:01.953065 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-599baf-59dd5c9655-wktlh" podStartSLOduration=2.553416656 podStartE2EDuration="2.953031585s" totalImagesPullingTime="399.614929ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:17:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:18:01.951346372 +0000 UTC m=+1496.704977774" watchObservedRunningTime="2026-09-11 04:18:01.953031585 +0000 UTC m=+1496.706662988" Sep 11 04:18:05.816653 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:05.816624 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-5tpdc_9112557d-d0ef-44c7-8b2a-f6ecd0a980d6/console-operator/1.log" Sep 11 04:18:05.819553 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:05.819530 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-5tpdc_9112557d-d0ef-44c7-8b2a-f6ecd0a980d6/console-operator/1.log" Sep 11 04:18:05.975038 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:05.975002 2568 scope.go:117] "RemoveContainer" containerID="abae39621b38677eefd5228be3e1b7932853a2ffa01b8d7d39b1534ebb3779dc" Sep 11 04:18:05.986986 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:05.986961 2568 scope.go:117] "RemoveContainer" containerID="8ac3f9d22b0b8a9902cf38c31ef3db5f16010a47716ddea00547bc84a1e04076" Sep 11 04:18:07.946119 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:07.946089 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-599baf-59dd5c9655-wktlh" Sep 11 04:18:08.674567 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:08.674532 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8080-917d64"] Sep 11 04:18:08.677935 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:08.677919 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8080-917d64" Sep 11 04:18:08.680058 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:08.680021 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-zqmz7\"" Sep 11 04:18:08.684651 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:08.684624 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8080-917d64"] Sep 11 04:18:08.787810 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:08.787767 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nkbq\" (UniqueName: \"kubernetes.io/projected/828723db-e981-4f68-a139-f756869f31e7-kube-api-access-6nkbq\") pod \"test-curl-8080-917d64\" (UID: \"828723db-e981-4f68-a139-f756869f31e7\") " pod="opendatahub/test-curl-8080-917d64" Sep 11 04:18:08.888782 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:08.888747 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nkbq\" (UniqueName: \"kubernetes.io/projected/828723db-e981-4f68-a139-f756869f31e7-kube-api-access-6nkbq\") pod \"test-curl-8080-917d64\" (UID: \"828723db-e981-4f68-a139-f756869f31e7\") " pod="opendatahub/test-curl-8080-917d64" Sep 11 04:18:08.897317 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:08.897291 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nkbq\" (UniqueName: \"kubernetes.io/projected/828723db-e981-4f68-a139-f756869f31e7-kube-api-access-6nkbq\") pod \"test-curl-8080-917d64\" (UID: \"828723db-e981-4f68-a139-f756869f31e7\") " pod="opendatahub/test-curl-8080-917d64" Sep 11 04:18:08.987890 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:08.987807 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8080-917d64" Sep 11 04:18:09.257810 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:09.257783 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8080-917d64"] Sep 11 04:18:09.260173 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:18:09.260143 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod828723db_e981_4f68_a139_f756869f31e7.slice/crio-59aadb15ba70da644a4644cc16a57707a9e315f2a1d3de3981c0f24822a3a30a WatchSource:0}: Error finding container 59aadb15ba70da644a4644cc16a57707a9e315f2a1d3de3981c0f24822a3a30a: Status 404 returned error can't find the container with id 59aadb15ba70da644a4644cc16a57707a9e315f2a1d3de3981c0f24822a3a30a Sep 11 04:18:09.977457 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:09.977406 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8080-917d64" event={"ID":"828723db-e981-4f68-a139-f756869f31e7","Type":"ContainerStarted","Data":"a62d16662707f321e5c3fa2727c19858bff7ee83975b0b8f70faa12736a53515"} Sep 11 04:18:09.977639 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:09.977466 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8080-917d64" event={"ID":"828723db-e981-4f68-a139-f756869f31e7","Type":"ContainerStarted","Data":"59aadb15ba70da644a4644cc16a57707a9e315f2a1d3de3981c0f24822a3a30a"} Sep 11 04:18:09.991932 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:09.991883 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8080-917d64" podStartSLOduration=1.9918685919999999 podStartE2EDuration="1.991868592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:18:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:18:09.990756942 +0000 UTC m=+1504.744388344" watchObservedRunningTime="2026-09-11 04:18:09.991868592 +0000 UTC m=+1504.745499988" Sep 11 04:18:10.618388 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:10.618353 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8080-917d64"] Sep 11 04:18:10.623229 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:10.623199 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8080-917d64"] Sep 11 04:18:10.745645 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:10.745612 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8080-f6e6ea"] Sep 11 04:18:10.746064 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:10.746032 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="828723db-e981-4f68-a139-f756869f31e7" containerName="test-curl-8080-917d64" Sep 11 04:18:10.746116 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:10.746071 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="828723db-e981-4f68-a139-f756869f31e7" containerName="test-curl-8080-917d64" Sep 11 04:18:10.746152 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:10.746129 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="828723db-e981-4f68-a139-f756869f31e7" containerName="test-curl-8080-917d64" Sep 11 04:18:10.749290 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:10.749274 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8080-f6e6ea" Sep 11 04:18:10.753798 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:10.753772 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8080-f6e6ea"] Sep 11 04:18:10.910341 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:10.910307 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljr4f\" (UniqueName: \"kubernetes.io/projected/4310a2e0-5424-4f9c-85a5-7e14e259c6a2-kube-api-access-ljr4f\") pod \"test-curl-8080-f6e6ea\" (UID: \"4310a2e0-5424-4f9c-85a5-7e14e259c6a2\") " pod="opendatahub/test-curl-8080-f6e6ea" Sep 11 04:18:10.982436 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:18:10.982400 2568 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8080-917d64" container="test-curl-8080-917d64" Sep 11 04:18:11.011000 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:11.010921 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljr4f\" (UniqueName: \"kubernetes.io/projected/4310a2e0-5424-4f9c-85a5-7e14e259c6a2-kube-api-access-ljr4f\") pod \"test-curl-8080-f6e6ea\" (UID: \"4310a2e0-5424-4f9c-85a5-7e14e259c6a2\") " pod="opendatahub/test-curl-8080-f6e6ea" Sep 11 04:18:11.019182 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:11.019154 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljr4f\" (UniqueName: \"kubernetes.io/projected/4310a2e0-5424-4f9c-85a5-7e14e259c6a2-kube-api-access-ljr4f\") pod \"test-curl-8080-f6e6ea\" (UID: \"4310a2e0-5424-4f9c-85a5-7e14e259c6a2\") " pod="opendatahub/test-curl-8080-f6e6ea" Sep 11 04:18:11.060142 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:11.060120 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8080-f6e6ea" Sep 11 04:18:11.332623 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:11.332403 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8080-f6e6ea"] Sep 11 04:18:11.335198 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:18:11.335164 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4310a2e0_5424_4f9c_85a5_7e14e259c6a2.slice/crio-f2403593c3b80c4e994a96fd13b22f81aeb940384a0c974dc397a8ff7e90ffe7 WatchSource:0}: Error finding container f2403593c3b80c4e994a96fd13b22f81aeb940384a0c974dc397a8ff7e90ffe7: Status 404 returned error can't find the container with id f2403593c3b80c4e994a96fd13b22f81aeb940384a0c974dc397a8ff7e90ffe7 Sep 11 04:18:11.989144 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:11.989107 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8080-f6e6ea" event={"ID":"4310a2e0-5424-4f9c-85a5-7e14e259c6a2","Type":"ContainerStarted","Data":"cc85f8321286fdb7a9ea8e9db1e16dd73f6f5d5efc25b16dccda51103ce28a24"} Sep 11 04:18:11.989144 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:11.989151 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8080-f6e6ea" event={"ID":"4310a2e0-5424-4f9c-85a5-7e14e259c6a2","Type":"ContainerStarted","Data":"f2403593c3b80c4e994a96fd13b22f81aeb940384a0c974dc397a8ff7e90ffe7"} Sep 11 04:18:11.989409 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:11.989183 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8080-917d64" podUID="828723db-e981-4f68-a139-f756869f31e7" containerName="test-curl-8080-917d64" containerID="cri-o://a62d16662707f321e5c3fa2727c19858bff7ee83975b0b8f70faa12736a53515" gracePeriod=2 Sep 11 04:18:12.006274 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:12.006229 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-8080-f6e6ea" podStartSLOduration=2.006216774 podStartE2EDuration="2.006216774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:18:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:18:12.003727383 +0000 UTC m=+1506.757358784" watchObservedRunningTime="2026-09-11 04:18:12.006216774 +0000 UTC m=+1506.759848207" Sep 11 04:18:12.617265 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:12.617230 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8080-f6e6ea"] Sep 11 04:18:12.621264 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:12.621237 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8080-f6e6ea"] Sep 11 04:18:12.994078 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:18:12.994035 2568 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-8080-f6e6ea" container="test-curl-8080-f6e6ea" Sep 11 04:18:13.999834 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:13.999792 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-8080-f6e6ea" podUID="4310a2e0-5424-4f9c-85a5-7e14e259c6a2" containerName="test-curl-8080-f6e6ea" containerID="cri-o://cc85f8321286fdb7a9ea8e9db1e16dd73f6f5d5efc25b16dccda51103ce28a24" gracePeriod=2 Sep 11 04:18:14.159440 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:14.159416 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8080-917d64" Sep 11 04:18:14.301230 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:14.301145 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:18:14.344028 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:14.343993 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nkbq\" (UniqueName: \"kubernetes.io/projected/828723db-e981-4f68-a139-f756869f31e7-kube-api-access-6nkbq\") pod \"828723db-e981-4f68-a139-f756869f31e7\" (UID: \"828723db-e981-4f68-a139-f756869f31e7\") " Sep 11 04:18:14.346168 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:14.346136 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/828723db-e981-4f68-a139-f756869f31e7-kube-api-access-6nkbq" (OuterVolumeSpecName: "kube-api-access-6nkbq") pod "828723db-e981-4f68-a139-f756869f31e7" (UID: "828723db-e981-4f68-a139-f756869f31e7"). InnerVolumeSpecName "kube-api-access-6nkbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:18:14.445409 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:14.445366 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nkbq\" (UniqueName: \"kubernetes.io/projected/828723db-e981-4f68-a139-f756869f31e7-kube-api-access-6nkbq\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:18:14.796767 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:14.796726 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:18:15.006371 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:15.006337 2568 generic.go:356] "Generic (PLEG): container finished" podID="828723db-e981-4f68-a139-f756869f31e7" containerID="a62d16662707f321e5c3fa2727c19858bff7ee83975b0b8f70faa12736a53515" exitCode=137 Sep 11 04:18:15.006756 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:15.006381 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8080-917d64" Sep 11 04:18:15.006756 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:15.006455 2568 scope.go:117] "RemoveContainer" containerID="a62d16662707f321e5c3fa2727c19858bff7ee83975b0b8f70faa12736a53515" Sep 11 04:18:15.015223 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:15.015207 2568 scope.go:117] "RemoveContainer" containerID="a62d16662707f321e5c3fa2727c19858bff7ee83975b0b8f70faa12736a53515" Sep 11 04:18:15.015517 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:18:15.015500 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a62d16662707f321e5c3fa2727c19858bff7ee83975b0b8f70faa12736a53515\": container with ID starting with a62d16662707f321e5c3fa2727c19858bff7ee83975b0b8f70faa12736a53515 not found: ID does not exist" containerID="a62d16662707f321e5c3fa2727c19858bff7ee83975b0b8f70faa12736a53515" Sep 11 04:18:15.015570 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:15.015527 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a62d16662707f321e5c3fa2727c19858bff7ee83975b0b8f70faa12736a53515"} err="failed to get container status \"a62d16662707f321e5c3fa2727c19858bff7ee83975b0b8f70faa12736a53515\": rpc error: code = NotFound desc = could not find container \"a62d16662707f321e5c3fa2727c19858bff7ee83975b0b8f70faa12736a53515\": container with ID starting with a62d16662707f321e5c3fa2727c19858bff7ee83975b0b8f70faa12736a53515 not found: ID does not exist" Sep 11 04:18:15.016936 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:18:15.016914 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8080-917d64\" is forbidden: User \"system:node:ip-10-0-140-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-30.ec2.internal' and this object" podUID="828723db-e981-4f68-a139-f756869f31e7" pod="opendatahub/test-curl-8080-917d64" Sep 11 04:18:15.713241 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:15.713209 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9342ed/sub-test-model-9342ed-kserve-76d7db7b5-shdsk"] Sep 11 04:18:15.715600 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:18:15.715557 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8080-917d64\" is forbidden: User \"system:node:ip-10-0-140-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-30.ec2.internal' and this object" podUID="828723db-e981-4f68-a139-f756869f31e7" pod="opendatahub/test-curl-8080-917d64" Sep 11 04:18:15.734355 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:15.734327 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:18:15.766308 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:18:15.766274 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8080-917d64\" is forbidden: User \"system:node:ip-10-0-140-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-30.ec2.internal' and this object" podUID="828723db-e981-4f68-a139-f756869f31e7" pod="opendatahub/test-curl-8080-917d64" Sep 11 04:18:15.768839 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:15.768808 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="828723db-e981-4f68-a139-f756869f31e7" path="/var/lib/kubelet/pods/828723db-e981-4f68-a139-f756869f31e7/volumes" Sep 11 04:18:16.175946 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:16.175920 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8080-f6e6ea" Sep 11 04:18:16.363762 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:16.363666 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ljr4f\" (UniqueName: \"kubernetes.io/projected/4310a2e0-5424-4f9c-85a5-7e14e259c6a2-kube-api-access-ljr4f\") pod \"4310a2e0-5424-4f9c-85a5-7e14e259c6a2\" (UID: \"4310a2e0-5424-4f9c-85a5-7e14e259c6a2\") " Sep 11 04:18:16.365766 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:16.365735 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4310a2e0-5424-4f9c-85a5-7e14e259c6a2-kube-api-access-ljr4f" (OuterVolumeSpecName: "kube-api-access-ljr4f") pod "4310a2e0-5424-4f9c-85a5-7e14e259c6a2" (UID: "4310a2e0-5424-4f9c-85a5-7e14e259c6a2"). InnerVolumeSpecName "kube-api-access-ljr4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:18:16.464477 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:16.464439 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljr4f\" (UniqueName: \"kubernetes.io/projected/4310a2e0-5424-4f9c-85a5-7e14e259c6a2-kube-api-access-ljr4f\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:18:17.019461 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:17.019420 2568 generic.go:356] "Generic (PLEG): container finished" podID="4310a2e0-5424-4f9c-85a5-7e14e259c6a2" containerID="cc85f8321286fdb7a9ea8e9db1e16dd73f6f5d5efc25b16dccda51103ce28a24" exitCode=137 Sep 11 04:18:17.019461 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:17.019464 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8080-f6e6ea" Sep 11 04:18:17.019793 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:17.019519 2568 scope.go:117] "RemoveContainer" containerID="cc85f8321286fdb7a9ea8e9db1e16dd73f6f5d5efc25b16dccda51103ce28a24" Sep 11 04:18:17.028217 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:17.028204 2568 scope.go:117] "RemoveContainer" containerID="cc85f8321286fdb7a9ea8e9db1e16dd73f6f5d5efc25b16dccda51103ce28a24" Sep 11 04:18:17.028518 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:18:17.028499 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc85f8321286fdb7a9ea8e9db1e16dd73f6f5d5efc25b16dccda51103ce28a24\": container with ID starting with cc85f8321286fdb7a9ea8e9db1e16dd73f6f5d5efc25b16dccda51103ce28a24 not found: ID does not exist" containerID="cc85f8321286fdb7a9ea8e9db1e16dd73f6f5d5efc25b16dccda51103ce28a24" Sep 11 04:18:17.028607 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:17.028524 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc85f8321286fdb7a9ea8e9db1e16dd73f6f5d5efc25b16dccda51103ce28a24"} err="failed to get container status \"cc85f8321286fdb7a9ea8e9db1e16dd73f6f5d5efc25b16dccda51103ce28a24\": rpc error: code = NotFound desc = could not find container \"cc85f8321286fdb7a9ea8e9db1e16dd73f6f5d5efc25b16dccda51103ce28a24\": container with ID starting with cc85f8321286fdb7a9ea8e9db1e16dd73f6f5d5efc25b16dccda51103ce28a24 not found: ID does not exist" Sep 11 04:18:17.029662 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:18:17.029640 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-8080-f6e6ea\" is forbidden: User \"system:node:ip-10-0-140-30.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-140-30.ec2.internal' and this object" podUID="4310a2e0-5424-4f9c-85a5-7e14e259c6a2" pod="opendatahub/test-curl-8080-f6e6ea" Sep 11 04:18:17.769332 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:17.769299 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4310a2e0-5424-4f9c-85a5-7e14e259c6a2" path="/var/lib/kubelet/pods/4310a2e0-5424-4f9c-85a5-7e14e259c6a2/volumes" Sep 11 04:18:20.789718 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:20.789678 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:18:24.325252 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:24.325192 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:18:26.358461 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:26.358419 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:18:27.277294 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:27.277254 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:18:27.309518 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:27.309480 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-cc6708/rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp"] Sep 11 04:18:27.309992 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:27.309976 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4310a2e0-5424-4f9c-85a5-7e14e259c6a2" containerName="test-curl-8080-f6e6ea" Sep 11 04:18:27.310066 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:27.309995 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4310a2e0-5424-4f9c-85a5-7e14e259c6a2" containerName="test-curl-8080-f6e6ea" Sep 11 04:18:27.310120 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:27.310083 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="4310a2e0-5424-4f9c-85a5-7e14e259c6a2" containerName="test-curl-8080-f6e6ea" Sep 11 04:18:27.314993 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:27.314975 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-cc6708/rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp" Sep 11 04:18:27.317534 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:27.317514 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-cc6708\"/\"default-dockercfg-56nqf\"" Sep 11 04:18:27.317654 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:27.317554 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-cc6708\"/\"openshift-service-ca.crt\"" Sep 11 04:18:27.318501 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:27.318475 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-cc6708\"/\"rate-test-model-cc6708-kserve-self-signed-certs\"" Sep 11 04:18:27.318616 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:27.318491 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-cc6708\"/\"kube-root-ca.crt\"" Sep 11 04:18:27.324491 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:27.324467 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cc6708/rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp"] Sep 11 04:18:27.348894 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:27.348857 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:18:27.368966 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:27.368933 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa86d199-8a1e-4915-bca5-1524cbbf3aba-dshm\") pod \"rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp\" (UID: \"fa86d199-8a1e-4915-bca5-1524cbbf3aba\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp" Sep 11 04:18:27.369370 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:27.369024 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa86d199-8a1e-4915-bca5-1524cbbf3aba-tls-certs\") pod \"rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp\" (UID: \"fa86d199-8a1e-4915-bca5-1524cbbf3aba\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp" Sep 11 04:18:27.369370 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:27.369094 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq6db\" (UniqueName: \"kubernetes.io/projected/fa86d199-8a1e-4915-bca5-1524cbbf3aba-kube-api-access-vq6db\") pod \"rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp\" (UID: \"fa86d199-8a1e-4915-bca5-1524cbbf3aba\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp" Sep 11 04:18:27.369370 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:27.369128 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa86d199-8a1e-4915-bca5-1524cbbf3aba-model-cache\") pod \"rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp\" (UID: \"fa86d199-8a1e-4915-bca5-1524cbbf3aba\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp" Sep 11 04:18:27.369370 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:27.369146 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa86d199-8a1e-4915-bca5-1524cbbf3aba-tmp-dir\") pod \"rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp\" (UID: \"fa86d199-8a1e-4915-bca5-1524cbbf3aba\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp" Sep 11 04:18:27.369370 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:27.369229 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa86d199-8a1e-4915-bca5-1524cbbf3aba-home\") pod \"rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp\" (UID: \"fa86d199-8a1e-4915-bca5-1524cbbf3aba\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp" Sep 11 04:18:27.470681 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:27.470634 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa86d199-8a1e-4915-bca5-1524cbbf3aba-tmp-dir\") pod \"rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp\" (UID: \"fa86d199-8a1e-4915-bca5-1524cbbf3aba\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp" Sep 11 04:18:27.470681 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:27.470687 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa86d199-8a1e-4915-bca5-1524cbbf3aba-home\") pod \"rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp\" (UID: \"fa86d199-8a1e-4915-bca5-1524cbbf3aba\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp" Sep 11 04:18:27.470921 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:27.470723 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa86d199-8a1e-4915-bca5-1524cbbf3aba-dshm\") pod \"rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp\" (UID: \"fa86d199-8a1e-4915-bca5-1524cbbf3aba\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp" Sep 11 04:18:27.470921 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:27.470781 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa86d199-8a1e-4915-bca5-1524cbbf3aba-tls-certs\") pod \"rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp\" (UID: \"fa86d199-8a1e-4915-bca5-1524cbbf3aba\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp" Sep 11 04:18:27.470921 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:27.470810 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq6db\" (UniqueName: \"kubernetes.io/projected/fa86d199-8a1e-4915-bca5-1524cbbf3aba-kube-api-access-vq6db\") pod \"rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp\" (UID: \"fa86d199-8a1e-4915-bca5-1524cbbf3aba\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp" Sep 11 04:18:27.470921 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:27.470864 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa86d199-8a1e-4915-bca5-1524cbbf3aba-model-cache\") pod \"rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp\" (UID: \"fa86d199-8a1e-4915-bca5-1524cbbf3aba\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp" Sep 11 04:18:27.471137 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:27.471116 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa86d199-8a1e-4915-bca5-1524cbbf3aba-tmp-dir\") pod \"rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp\" (UID: \"fa86d199-8a1e-4915-bca5-1524cbbf3aba\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp" Sep 11 04:18:27.471253 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:27.471230 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa86d199-8a1e-4915-bca5-1524cbbf3aba-home\") pod \"rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp\" (UID: \"fa86d199-8a1e-4915-bca5-1524cbbf3aba\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp" Sep 11 04:18:27.471313 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:27.471252 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa86d199-8a1e-4915-bca5-1524cbbf3aba-model-cache\") pod \"rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp\" (UID: \"fa86d199-8a1e-4915-bca5-1524cbbf3aba\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp" Sep 11 04:18:27.473188 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:27.473169 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa86d199-8a1e-4915-bca5-1524cbbf3aba-dshm\") pod \"rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp\" (UID: \"fa86d199-8a1e-4915-bca5-1524cbbf3aba\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp" Sep 11 04:18:27.473451 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:27.473432 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa86d199-8a1e-4915-bca5-1524cbbf3aba-tls-certs\") pod \"rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp\" (UID: \"fa86d199-8a1e-4915-bca5-1524cbbf3aba\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp" Sep 11 04:18:27.479133 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:27.479110 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq6db\" (UniqueName: \"kubernetes.io/projected/fa86d199-8a1e-4915-bca5-1524cbbf3aba-kube-api-access-vq6db\") pod \"rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp\" (UID: \"fa86d199-8a1e-4915-bca5-1524cbbf3aba\") " pod="ai-tenant-e2e-shared-a-w3-cc6708/rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp" Sep 11 04:18:27.627007 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:27.626919 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-cc6708/rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp" Sep 11 04:18:27.899042 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:27.899016 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cc6708/rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp"] Sep 11 04:18:27.901665 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:18:27.901635 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa86d199_8a1e_4915_bca5_1524cbbf3aba.slice/crio-5e338b6e8a969f791bcc673d5fc18992107a4fb158bb6cb78010a9cf4910f1aa WatchSource:0}: Error finding container 5e338b6e8a969f791bcc673d5fc18992107a4fb158bb6cb78010a9cf4910f1aa: Status 404 returned error can't find the container with id 5e338b6e8a969f791bcc673d5fc18992107a4fb158bb6cb78010a9cf4910f1aa Sep 11 04:18:28.078821 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:28.078740 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-cc6708/rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp" event={"ID":"fa86d199-8a1e-4915-bca5-1524cbbf3aba","Type":"ContainerStarted","Data":"c6329db6393e60ce89efe06badd132028c7ef046bea4b898107e263ddf6ab345"} Sep 11 04:18:28.078821 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:28.078783 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-cc6708/rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp" event={"ID":"fa86d199-8a1e-4915-bca5-1524cbbf3aba","Type":"ContainerStarted","Data":"5e338b6e8a969f791bcc673d5fc18992107a4fb158bb6cb78010a9cf4910f1aa"} Sep 11 04:18:28.097802 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:28.097754 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-cc6708/rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp" podStartSLOduration=1.097738237 podStartE2EDuration="1.097738237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:18:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:18:28.096552844 +0000 UTC m=+1522.850184245" watchObservedRunningTime="2026-09-11 04:18:28.097738237 +0000 UTC m=+1522.851369640" Sep 11 04:18:30.732472 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:30.732410 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-9342ed/sub-test-model-9342ed-kserve-76d7db7b5-shdsk" podUID="082edca2-cfea-4818-9e08-277a51456382" containerName="main" containerID="cri-o://a289b9c03191385bfa6e64ce1c777865ba95fe6d6df0d11441a75047c302a6cd" gracePeriod=45 Sep 11 04:18:30.983254 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:30.983199 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9342ed/sub-test-model-9342ed-kserve-76d7db7b5-shdsk" Sep 11 04:18:31.096463 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:31.096422 2568 generic.go:356] "Generic (PLEG): container finished" podID="082edca2-cfea-4818-9e08-277a51456382" containerID="a289b9c03191385bfa6e64ce1c777865ba95fe6d6df0d11441a75047c302a6cd" exitCode=0 Sep 11 04:18:31.096636 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:31.096470 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-9342ed/sub-test-model-9342ed-kserve-76d7db7b5-shdsk" Sep 11 04:18:31.096636 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:31.096508 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9342ed/sub-test-model-9342ed-kserve-76d7db7b5-shdsk" event={"ID":"082edca2-cfea-4818-9e08-277a51456382","Type":"ContainerDied","Data":"a289b9c03191385bfa6e64ce1c777865ba95fe6d6df0d11441a75047c302a6cd"} Sep 11 04:18:31.096636 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:31.096554 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-9342ed/sub-test-model-9342ed-kserve-76d7db7b5-shdsk" event={"ID":"082edca2-cfea-4818-9e08-277a51456382","Type":"ContainerDied","Data":"5aa61542bb14e96aeff3923a2e644edc9c3c8ea834af67240aa17becf70055af"} Sep 11 04:18:31.096636 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:31.096578 2568 scope.go:117] "RemoveContainer" containerID="a289b9c03191385bfa6e64ce1c777865ba95fe6d6df0d11441a75047c302a6cd" Sep 11 04:18:31.103834 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:31.103809 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/082edca2-cfea-4818-9e08-277a51456382-model-cache\") pod \"082edca2-cfea-4818-9e08-277a51456382\" (UID: \"082edca2-cfea-4818-9e08-277a51456382\") " Sep 11 04:18:31.103988 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:31.103872 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mw756\" (UniqueName: \"kubernetes.io/projected/082edca2-cfea-4818-9e08-277a51456382-kube-api-access-mw756\") pod \"082edca2-cfea-4818-9e08-277a51456382\" (UID: \"082edca2-cfea-4818-9e08-277a51456382\") " Sep 11 04:18:31.103988 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:31.103926 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/082edca2-cfea-4818-9e08-277a51456382-tmp-dir\") pod \"082edca2-cfea-4818-9e08-277a51456382\" (UID: \"082edca2-cfea-4818-9e08-277a51456382\") " Sep 11 04:18:31.103988 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:31.103957 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/082edca2-cfea-4818-9e08-277a51456382-dshm\") pod \"082edca2-cfea-4818-9e08-277a51456382\" (UID: \"082edca2-cfea-4818-9e08-277a51456382\") " Sep 11 04:18:31.104189 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:31.104006 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/082edca2-cfea-4818-9e08-277a51456382-home\") pod \"082edca2-cfea-4818-9e08-277a51456382\" (UID: \"082edca2-cfea-4818-9e08-277a51456382\") " Sep 11 04:18:31.104189 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:31.104037 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/082edca2-cfea-4818-9e08-277a51456382-tls-certs\") pod \"082edca2-cfea-4818-9e08-277a51456382\" (UID: \"082edca2-cfea-4818-9e08-277a51456382\") " Sep 11 04:18:31.104189 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:31.104142 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/082edca2-cfea-4818-9e08-277a51456382-model-cache" (OuterVolumeSpecName: "model-cache") pod "082edca2-cfea-4818-9e08-277a51456382" (UID: "082edca2-cfea-4818-9e08-277a51456382"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:18:31.104389 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:31.104340 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/082edca2-cfea-4818-9e08-277a51456382-home" (OuterVolumeSpecName: "home") pod "082edca2-cfea-4818-9e08-277a51456382" (UID: "082edca2-cfea-4818-9e08-277a51456382"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:18:31.104389 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:31.104347 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/082edca2-cfea-4818-9e08-277a51456382-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "082edca2-cfea-4818-9e08-277a51456382" (UID: "082edca2-cfea-4818-9e08-277a51456382"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:18:31.104571 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:31.104467 2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/082edca2-cfea-4818-9e08-277a51456382-home\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:18:31.104571 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:31.104496 2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/082edca2-cfea-4818-9e08-277a51456382-model-cache\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:18:31.104571 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:31.104517 2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/082edca2-cfea-4818-9e08-277a51456382-tmp-dir\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:18:31.105746 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:31.105726 2568 scope.go:117] "RemoveContainer" containerID="a289b9c03191385bfa6e64ce1c777865ba95fe6d6df0d11441a75047c302a6cd" Sep 11 04:18:31.106127 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:18:31.106088 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a289b9c03191385bfa6e64ce1c777865ba95fe6d6df0d11441a75047c302a6cd\": container with ID starting with a289b9c03191385bfa6e64ce1c777865ba95fe6d6df0d11441a75047c302a6cd not found: ID does not exist" containerID="a289b9c03191385bfa6e64ce1c777865ba95fe6d6df0d11441a75047c302a6cd" Sep 11 04:18:31.106231 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:31.106134 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a289b9c03191385bfa6e64ce1c777865ba95fe6d6df0d11441a75047c302a6cd"} err="failed to get container status \"a289b9c03191385bfa6e64ce1c777865ba95fe6d6df0d11441a75047c302a6cd\": rpc error: code = NotFound desc = could not find container \"a289b9c03191385bfa6e64ce1c777865ba95fe6d6df0d11441a75047c302a6cd\": container with ID starting with a289b9c03191385bfa6e64ce1c777865ba95fe6d6df0d11441a75047c302a6cd not found: ID does not exist" Sep 11 04:18:31.106357 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:31.106319 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/082edca2-cfea-4818-9e08-277a51456382-dshm" (OuterVolumeSpecName: "dshm") pod "082edca2-cfea-4818-9e08-277a51456382" (UID: "082edca2-cfea-4818-9e08-277a51456382"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:18:31.106443 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:31.106419 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/082edca2-cfea-4818-9e08-277a51456382-kube-api-access-mw756" (OuterVolumeSpecName: "kube-api-access-mw756") pod "082edca2-cfea-4818-9e08-277a51456382" (UID: "082edca2-cfea-4818-9e08-277a51456382"). InnerVolumeSpecName "kube-api-access-mw756". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:18:31.106443 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:31.106416 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/082edca2-cfea-4818-9e08-277a51456382-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "082edca2-cfea-4818-9e08-277a51456382" (UID: "082edca2-cfea-4818-9e08-277a51456382"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 04:18:31.205262 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:31.205219 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw756\" (UniqueName: \"kubernetes.io/projected/082edca2-cfea-4818-9e08-277a51456382-kube-api-access-mw756\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:18:31.205262 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:31.205253 2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/082edca2-cfea-4818-9e08-277a51456382-dshm\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:18:31.205262 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:31.205264 2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/082edca2-cfea-4818-9e08-277a51456382-tls-certs\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:18:31.425239 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:31.425210 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9342ed/sub-test-model-9342ed-kserve-76d7db7b5-shdsk"] Sep 11 04:18:31.429435 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:31.429410 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-9342ed/sub-test-model-9342ed-kserve-76d7db7b5-shdsk"] Sep 11 04:18:31.768946 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:31.768850 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="082edca2-cfea-4818-9e08-277a51456382" path="/var/lib/kubelet/pods/082edca2-cfea-4818-9e08-277a51456382/volumes" Sep 11 04:18:37.627402 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:37.627362 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-cc6708/rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp" Sep 11 04:18:37.627814 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:37.627416 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-cc6708/rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp" Sep 11 04:18:37.640691 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:37.640662 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-cc6708/rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp" Sep 11 04:18:38.148266 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:38.148235 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-cc6708/rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp" Sep 11 04:18:40.542982 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:40.542946 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:18:42.232648 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:42.232618 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bf30f8-587697d65f-nd4kb"] Sep 11 04:18:42.233154 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:42.232835 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf30f8-587697d65f-nd4kb" podUID="0613c781-47be-4eb7-866d-e00a9a4dd027" containerName="maas-api" containerID="cri-o://20cd17957ac63af246aa70b51ba57038877eb8704a69297b951eefdef317191a" gracePeriod=30 Sep 11 04:18:42.266163 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:42.266127 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:18:42.341417 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:42.341367 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:18:42.441927 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:42.441888 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:18:42.482708 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:42.482650 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf30f8-587697d65f-nd4kb" Sep 11 04:18:42.514604 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:42.514570 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:18:42.610848 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:42.610819 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0613c781-47be-4eb7-866d-e00a9a4dd027-openshift-service-ca\") pod \"0613c781-47be-4eb7-866d-e00a9a4dd027\" (UID: \"0613c781-47be-4eb7-866d-e00a9a4dd027\") " Sep 11 04:18:42.611029 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:42.610866 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0613c781-47be-4eb7-866d-e00a9a4dd027-maas-api-tls\") pod \"0613c781-47be-4eb7-866d-e00a9a4dd027\" (UID: \"0613c781-47be-4eb7-866d-e00a9a4dd027\") " Sep 11 04:18:42.611029 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:42.610899 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cflrg\" (UniqueName: \"kubernetes.io/projected/0613c781-47be-4eb7-866d-e00a9a4dd027-kube-api-access-cflrg\") pod \"0613c781-47be-4eb7-866d-e00a9a4dd027\" (UID: \"0613c781-47be-4eb7-866d-e00a9a4dd027\") " Sep 11 04:18:42.611029 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:42.610923 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0613c781-47be-4eb7-866d-e00a9a4dd027-metrics-certs\") pod \"0613c781-47be-4eb7-866d-e00a9a4dd027\" (UID: \"0613c781-47be-4eb7-866d-e00a9a4dd027\") " Sep 11 04:18:42.611306 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:42.611253 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0613c781-47be-4eb7-866d-e00a9a4dd027-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0613c781-47be-4eb7-866d-e00a9a4dd027" (UID: "0613c781-47be-4eb7-866d-e00a9a4dd027"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 04:18:42.613143 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:42.613114 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0613c781-47be-4eb7-866d-e00a9a4dd027-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0613c781-47be-4eb7-866d-e00a9a4dd027" (UID: "0613c781-47be-4eb7-866d-e00a9a4dd027"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 04:18:42.613251 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:42.613151 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0613c781-47be-4eb7-866d-e00a9a4dd027-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0613c781-47be-4eb7-866d-e00a9a4dd027" (UID: "0613c781-47be-4eb7-866d-e00a9a4dd027"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 04:18:42.613251 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:42.613224 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0613c781-47be-4eb7-866d-e00a9a4dd027-kube-api-access-cflrg" (OuterVolumeSpecName: "kube-api-access-cflrg") pod "0613c781-47be-4eb7-866d-e00a9a4dd027" (UID: "0613c781-47be-4eb7-866d-e00a9a4dd027"). InnerVolumeSpecName "kube-api-access-cflrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:18:42.712289 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:42.712258 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0613c781-47be-4eb7-866d-e00a9a4dd027-openshift-service-ca\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:18:42.712289 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:42.712283 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0613c781-47be-4eb7-866d-e00a9a4dd027-maas-api-tls\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:18:42.712289 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:42.712294 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cflrg\" (UniqueName: \"kubernetes.io/projected/0613c781-47be-4eb7-866d-e00a9a4dd027-kube-api-access-cflrg\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:18:42.712525 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:42.712302 2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0613c781-47be-4eb7-866d-e00a9a4dd027-metrics-certs\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:18:42.872394 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:42.872292 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:18:43.163020 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:43.162986 2568 generic.go:356] "Generic (PLEG): container finished" podID="0613c781-47be-4eb7-866d-e00a9a4dd027" containerID="20cd17957ac63af246aa70b51ba57038877eb8704a69297b951eefdef317191a" exitCode=0 Sep 11 04:18:43.163231 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:43.163037 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf30f8-587697d65f-nd4kb" Sep 11 04:18:43.163231 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:43.163067 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf30f8-587697d65f-nd4kb" event={"ID":"0613c781-47be-4eb7-866d-e00a9a4dd027","Type":"ContainerDied","Data":"20cd17957ac63af246aa70b51ba57038877eb8704a69297b951eefdef317191a"} Sep 11 04:18:43.163231 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:43.163109 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-bf30f8-587697d65f-nd4kb" event={"ID":"0613c781-47be-4eb7-866d-e00a9a4dd027","Type":"ContainerDied","Data":"26d1424886875d74c9d341da040601dddc008b9cbe87d7c3b655e7a4a974d45a"} Sep 11 04:18:43.163231 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:43.163131 2568 scope.go:117] "RemoveContainer" containerID="20cd17957ac63af246aa70b51ba57038877eb8704a69297b951eefdef317191a" Sep 11 04:18:43.172265 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:43.172245 2568 scope.go:117] "RemoveContainer" containerID="20cd17957ac63af246aa70b51ba57038877eb8704a69297b951eefdef317191a" Sep 11 04:18:43.172545 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:18:43.172525 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20cd17957ac63af246aa70b51ba57038877eb8704a69297b951eefdef317191a\": container with ID starting with 20cd17957ac63af246aa70b51ba57038877eb8704a69297b951eefdef317191a not found: ID does not exist" containerID="20cd17957ac63af246aa70b51ba57038877eb8704a69297b951eefdef317191a" Sep 11 04:18:43.172615 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:43.172553 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20cd17957ac63af246aa70b51ba57038877eb8704a69297b951eefdef317191a"} err="failed to get container status \"20cd17957ac63af246aa70b51ba57038877eb8704a69297b951eefdef317191a\": rpc error: code = NotFound desc = could not find container \"20cd17957ac63af246aa70b51ba57038877eb8704a69297b951eefdef317191a\": container with ID starting with 20cd17957ac63af246aa70b51ba57038877eb8704a69297b951eefdef317191a not found: ID does not exist" Sep 11 04:18:43.186791 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:43.186756 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bf30f8-587697d65f-nd4kb"] Sep 11 04:18:43.190595 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:43.190566 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-bf30f8-587697d65f-nd4kb"] Sep 11 04:18:43.768585 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:43.768553 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0613c781-47be-4eb7-866d-e00a9a4dd027" path="/var/lib/kubelet/pods/0613c781-47be-4eb7-866d-e00a9a4dd027/volumes" Sep 11 04:18:48.925513 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:48.925467 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:18:48.988387 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:48.988349 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:18:49.203325 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:49.203238 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:18:49.771718 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:49.771682 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:18:51.507585 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:51.507541 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:18:53.693193 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:53.693156 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:18:56.725212 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:56.725180 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:18:59.810151 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:18:59.810110 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:19:05.347821 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:05.347396 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-599baf-59dd5c9655-wktlh"] Sep 11 04:19:05.347821 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:05.347700 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-599baf-59dd5c9655-wktlh" podUID="6b643f98-3070-4c1a-834f-0f8dc59db105" containerName="maas-api" containerID="cri-o://770cf1d1a0621b69fd2e8b30c542f8948c7ffb4d83c49b9cc4663c03f2c8d5fc" gracePeriod=30 Sep 11 04:19:05.380438 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:05.380407 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:19:05.619367 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:05.619340 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-599baf-59dd5c9655-wktlh" Sep 11 04:19:05.716407 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:05.716379 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slxvm\" (UniqueName: \"kubernetes.io/projected/6b643f98-3070-4c1a-834f-0f8dc59db105-kube-api-access-slxvm\") pod \"6b643f98-3070-4c1a-834f-0f8dc59db105\" (UID: \"6b643f98-3070-4c1a-834f-0f8dc59db105\") " Sep 11 04:19:05.716585 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:05.716462 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b643f98-3070-4c1a-834f-0f8dc59db105-maas-api-tls\") pod \"6b643f98-3070-4c1a-834f-0f8dc59db105\" (UID: \"6b643f98-3070-4c1a-834f-0f8dc59db105\") " Sep 11 04:19:05.716585 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:05.716486 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b643f98-3070-4c1a-834f-0f8dc59db105-openshift-service-ca\") pod \"6b643f98-3070-4c1a-834f-0f8dc59db105\" (UID: \"6b643f98-3070-4c1a-834f-0f8dc59db105\") " Sep 11 04:19:05.716585 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:05.716515 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b643f98-3070-4c1a-834f-0f8dc59db105-metrics-certs\") pod \"6b643f98-3070-4c1a-834f-0f8dc59db105\" (UID: \"6b643f98-3070-4c1a-834f-0f8dc59db105\") " Sep 11 04:19:05.716873 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:05.716849 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b643f98-3070-4c1a-834f-0f8dc59db105-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6b643f98-3070-4c1a-834f-0f8dc59db105" (UID: "6b643f98-3070-4c1a-834f-0f8dc59db105"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 04:19:05.718446 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:05.718424 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b643f98-3070-4c1a-834f-0f8dc59db105-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6b643f98-3070-4c1a-834f-0f8dc59db105" (UID: "6b643f98-3070-4c1a-834f-0f8dc59db105"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 04:19:05.718615 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:05.718596 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b643f98-3070-4c1a-834f-0f8dc59db105-kube-api-access-slxvm" (OuterVolumeSpecName: "kube-api-access-slxvm") pod "6b643f98-3070-4c1a-834f-0f8dc59db105" (UID: "6b643f98-3070-4c1a-834f-0f8dc59db105"). InnerVolumeSpecName "kube-api-access-slxvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:19:05.718706 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:05.718685 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b643f98-3070-4c1a-834f-0f8dc59db105-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6b643f98-3070-4c1a-834f-0f8dc59db105" (UID: "6b643f98-3070-4c1a-834f-0f8dc59db105"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 04:19:05.817218 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:05.817190 2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b643f98-3070-4c1a-834f-0f8dc59db105-metrics-certs\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:19:05.817218 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:05.817216 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slxvm\" (UniqueName: \"kubernetes.io/projected/6b643f98-3070-4c1a-834f-0f8dc59db105-kube-api-access-slxvm\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:19:05.817436 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:05.817226 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6b643f98-3070-4c1a-834f-0f8dc59db105-maas-api-tls\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:19:05.817436 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:05.817236 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b643f98-3070-4c1a-834f-0f8dc59db105-openshift-service-ca\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:19:06.288673 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:06.288579 2568 generic.go:356] "Generic (PLEG): container finished" podID="6b643f98-3070-4c1a-834f-0f8dc59db105" containerID="770cf1d1a0621b69fd2e8b30c542f8948c7ffb4d83c49b9cc4663c03f2c8d5fc" exitCode=0 Sep 11 04:19:06.288673 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:06.288626 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-599baf-59dd5c9655-wktlh" Sep 11 04:19:06.288915 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:06.288660 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-599baf-59dd5c9655-wktlh" event={"ID":"6b643f98-3070-4c1a-834f-0f8dc59db105","Type":"ContainerDied","Data":"770cf1d1a0621b69fd2e8b30c542f8948c7ffb4d83c49b9cc4663c03f2c8d5fc"} Sep 11 04:19:06.288915 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:06.288725 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-599baf-59dd5c9655-wktlh" event={"ID":"6b643f98-3070-4c1a-834f-0f8dc59db105","Type":"ContainerDied","Data":"9a5e4ec792063db05d7621f4c7692847c562f3dfe549c22ba23efe52d82e40a3"} Sep 11 04:19:06.288915 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:06.288760 2568 scope.go:117] "RemoveContainer" containerID="770cf1d1a0621b69fd2e8b30c542f8948c7ffb4d83c49b9cc4663c03f2c8d5fc" Sep 11 04:19:06.297644 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:06.297628 2568 scope.go:117] "RemoveContainer" containerID="770cf1d1a0621b69fd2e8b30c542f8948c7ffb4d83c49b9cc4663c03f2c8d5fc" Sep 11 04:19:06.297941 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:19:06.297922 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"770cf1d1a0621b69fd2e8b30c542f8948c7ffb4d83c49b9cc4663c03f2c8d5fc\": container with ID starting with 770cf1d1a0621b69fd2e8b30c542f8948c7ffb4d83c49b9cc4663c03f2c8d5fc not found: ID does not exist" containerID="770cf1d1a0621b69fd2e8b30c542f8948c7ffb4d83c49b9cc4663c03f2c8d5fc" Sep 11 04:19:06.298001 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:06.297953 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"770cf1d1a0621b69fd2e8b30c542f8948c7ffb4d83c49b9cc4663c03f2c8d5fc"} err="failed to get container status \"770cf1d1a0621b69fd2e8b30c542f8948c7ffb4d83c49b9cc4663c03f2c8d5fc\": rpc error: code = NotFound desc = could not find container \"770cf1d1a0621b69fd2e8b30c542f8948c7ffb4d83c49b9cc4663c03f2c8d5fc\": container with ID starting with 770cf1d1a0621b69fd2e8b30c542f8948c7ffb4d83c49b9cc4663c03f2c8d5fc not found: ID does not exist" Sep 11 04:19:06.305723 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:06.305698 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-599baf-59dd5c9655-wktlh"] Sep 11 04:19:06.309025 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:06.309005 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-599baf-59dd5c9655-wktlh"] Sep 11 04:19:07.213096 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:07.213060 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:19:07.769086 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:07.769038 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b643f98-3070-4c1a-834f-0f8dc59db105" path="/var/lib/kubelet/pods/6b643f98-3070-4c1a-834f-0f8dc59db105/volumes" Sep 11 04:19:12.073742 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:12.073701 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:19:12.139914 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:12.139880 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:19:12.353360 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:12.353261 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:19:12.825570 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:12.825515 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:19:15.896904 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:15.896873 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:19:23.001024 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:23.000987 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:19:23.779182 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:23.779152 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:19:23.814467 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:23.814435 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-772e41-c65794cd9-jjrdd"] Sep 11 04:19:23.814717 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:23.814691 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-772e41-c65794cd9-jjrdd" podUID="5f5e76f9-b2ad-4676-b83c-a8c52adc5233" containerName="payload-processing" containerID="cri-o://df536be10c52aa5045816109ad785700d673ab14f5518d70fc046c52631713ee" gracePeriod=30 Sep 11 04:19:23.853819 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:23.853782 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:19:23.918796 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:23.918763 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:19:23.996889 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:23.996852 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:19:24.058017 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:24.057918 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:19:24.079013 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:24.078991 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-772e41-c65794cd9-jjrdd" Sep 11 04:19:24.179782 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:24.179752 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6tzv\" (UniqueName: \"kubernetes.io/projected/5f5e76f9-b2ad-4676-b83c-a8c52adc5233-kube-api-access-w6tzv\") pod \"5f5e76f9-b2ad-4676-b83c-a8c52adc5233\" (UID: \"5f5e76f9-b2ad-4676-b83c-a8c52adc5233\") " Sep 11 04:19:24.179961 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:24.179790 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f5e76f9-b2ad-4676-b83c-a8c52adc5233-plugins-config-volume\") pod \"5f5e76f9-b2ad-4676-b83c-a8c52adc5233\" (UID: \"5f5e76f9-b2ad-4676-b83c-a8c52adc5233\") " Sep 11 04:19:24.180219 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:24.180195 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f5e76f9-b2ad-4676-b83c-a8c52adc5233-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5f5e76f9-b2ad-4676-b83c-a8c52adc5233" (UID: "5f5e76f9-b2ad-4676-b83c-a8c52adc5233"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 04:19:24.181895 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:24.181876 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f5e76f9-b2ad-4676-b83c-a8c52adc5233-kube-api-access-w6tzv" (OuterVolumeSpecName: "kube-api-access-w6tzv") pod "5f5e76f9-b2ad-4676-b83c-a8c52adc5233" (UID: "5f5e76f9-b2ad-4676-b83c-a8c52adc5233"). InnerVolumeSpecName "kube-api-access-w6tzv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:19:24.281234 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:24.281195 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6tzv\" (UniqueName: \"kubernetes.io/projected/5f5e76f9-b2ad-4676-b83c-a8c52adc5233-kube-api-access-w6tzv\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:19:24.281234 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:24.281227 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f5e76f9-b2ad-4676-b83c-a8c52adc5233-plugins-config-volume\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:19:24.386994 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:24.386876 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:19:24.391514 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:24.391484 2568 generic.go:356] "Generic (PLEG): container finished" podID="5f5e76f9-b2ad-4676-b83c-a8c52adc5233" containerID="df536be10c52aa5045816109ad785700d673ab14f5518d70fc046c52631713ee" exitCode=0 Sep 11 04:19:24.391656 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:24.391522 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-772e41-c65794cd9-jjrdd" event={"ID":"5f5e76f9-b2ad-4676-b83c-a8c52adc5233","Type":"ContainerDied","Data":"df536be10c52aa5045816109ad785700d673ab14f5518d70fc046c52631713ee"} Sep 11 04:19:24.391656 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:24.391533 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-772e41-c65794cd9-jjrdd" Sep 11 04:19:24.391656 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:24.391577 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-772e41-c65794cd9-jjrdd" event={"ID":"5f5e76f9-b2ad-4676-b83c-a8c52adc5233","Type":"ContainerDied","Data":"ae8631c4e1048284d3d112f1105fc22685be488a27b5848e44ceed154d2ee1f6"} Sep 11 04:19:24.391656 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:24.391604 2568 scope.go:117] "RemoveContainer" containerID="df536be10c52aa5045816109ad785700d673ab14f5518d70fc046c52631713ee" Sep 11 04:19:24.400551 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:24.400533 2568 scope.go:117] "RemoveContainer" containerID="df536be10c52aa5045816109ad785700d673ab14f5518d70fc046c52631713ee" Sep 11 04:19:24.400832 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:19:24.400810 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df536be10c52aa5045816109ad785700d673ab14f5518d70fc046c52631713ee\": container with ID starting with df536be10c52aa5045816109ad785700d673ab14f5518d70fc046c52631713ee not found: ID does not exist" containerID="df536be10c52aa5045816109ad785700d673ab14f5518d70fc046c52631713ee" Sep 11 04:19:24.400901 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:24.400845 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df536be10c52aa5045816109ad785700d673ab14f5518d70fc046c52631713ee"} err="failed to get container status \"df536be10c52aa5045816109ad785700d673ab14f5518d70fc046c52631713ee\": rpc error: code = NotFound desc = could not find container \"df536be10c52aa5045816109ad785700d673ab14f5518d70fc046c52631713ee\": container with ID starting with df536be10c52aa5045816109ad785700d673ab14f5518d70fc046c52631713ee not found: ID does not exist" Sep 11 04:19:24.415067 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:24.415024 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-772e41-c65794cd9-jjrdd"] Sep 11 04:19:24.419408 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:24.419381 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-772e41-c65794cd9-jjrdd"] Sep 11 04:19:25.768523 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:25.768492 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f5e76f9-b2ad-4676-b83c-a8c52adc5233" path="/var/lib/kubelet/pods/5f5e76f9-b2ad-4676-b83c-a8c52adc5233/volumes" Sep 11 04:19:30.075113 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:30.075081 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:19:30.138114 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:30.138073 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:19:30.353212 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:30.353109 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:19:30.808018 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:30.807973 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:19:37.183655 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:37.183617 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cc6708/rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp"] Sep 11 04:19:37.203550 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:37.203512 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:19:38.098398 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:38.098361 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:19:38.542520 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:38.542482 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:19:38.905067 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:38.905012 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:19:39.122172 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:39.122139 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:19:41.125292 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:41.125250 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8229c-5fbd746f56-lhh7v"] Sep 11 04:19:41.125790 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:41.125463 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8229c-5fbd746f56-lhh7v" podUID="2f4c1059-02c1-42b5-aed3-052f39640577" containerName="maas-api" containerID="cri-o://7a0ee62fce4370d03291cf330b6c4d3a365f95a75352ac09d04ab099ea11a4f8" gracePeriod=30 Sep 11 04:19:41.154253 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:41.154205 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:19:41.157965 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:41.157937 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f8229c-29818335-cpn7p"] Sep 11 04:19:41.160531 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:41.160500 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-f8229c-29818335-cpn7p"] Sep 11 04:19:41.195254 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:41.195216 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8229c-77b67bb94b-krzmq"] Sep 11 04:19:41.195510 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:41.195463 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8229c-77b67bb94b-krzmq" podUID="37f9cd2a-fedc-494f-a25f-a85b04719a84" containerName="payload-pre-processing" containerID="cri-o://876620718c77ec7845193217f4b5f481d8307473fd32b21a74170cddca03b7e1" gracePeriod=30 Sep 11 04:19:41.255437 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:41.255402 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:19:41.328739 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:41.328705 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:19:41.390974 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:41.390946 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:19:41.452248 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:41.452222 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8229c-5fbd746f56-lhh7v" Sep 11 04:19:41.478717 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:41.478606 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:19:41.486380 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:41.486022 2568 generic.go:356] "Generic (PLEG): container finished" podID="2f4c1059-02c1-42b5-aed3-052f39640577" containerID="7a0ee62fce4370d03291cf330b6c4d3a365f95a75352ac09d04ab099ea11a4f8" exitCode=0 Sep 11 04:19:41.486380 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:41.486096 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8229c-5fbd746f56-lhh7v" Sep 11 04:19:41.486380 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:41.486100 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8229c-5fbd746f56-lhh7v" event={"ID":"2f4c1059-02c1-42b5-aed3-052f39640577","Type":"ContainerDied","Data":"7a0ee62fce4370d03291cf330b6c4d3a365f95a75352ac09d04ab099ea11a4f8"} Sep 11 04:19:41.486380 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:41.486208 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8229c-5fbd746f56-lhh7v" event={"ID":"2f4c1059-02c1-42b5-aed3-052f39640577","Type":"ContainerDied","Data":"6b9996add55db1b9f6e77c3719d6fd3ea509afaf9e47f6c173987385b699cc72"} Sep 11 04:19:41.486380 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:41.486279 2568 scope.go:117] "RemoveContainer" containerID="7a0ee62fce4370d03291cf330b6c4d3a365f95a75352ac09d04ab099ea11a4f8" Sep 11 04:19:41.488494 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:41.488410 2568 generic.go:356] "Generic (PLEG): container finished" podID="37f9cd2a-fedc-494f-a25f-a85b04719a84" containerID="876620718c77ec7845193217f4b5f481d8307473fd32b21a74170cddca03b7e1" exitCode=0 Sep 11 04:19:41.488494 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:41.488482 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8229c-77b67bb94b-krzmq" event={"ID":"37f9cd2a-fedc-494f-a25f-a85b04719a84","Type":"ContainerDied","Data":"876620718c77ec7845193217f4b5f481d8307473fd32b21a74170cddca03b7e1"} Sep 11 04:19:41.488638 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:41.488525 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8229c-77b67bb94b-krzmq" event={"ID":"37f9cd2a-fedc-494f-a25f-a85b04719a84","Type":"ContainerDied","Data":"10148dd8af96f56c0b98e8048230914bc7ac6c551b126d1b3aebbc3ecfc6219d"} Sep 11 04:19:41.488638 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:41.488547 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10148dd8af96f56c0b98e8048230914bc7ac6c551b126d1b3aebbc3ecfc6219d" Sep 11 04:19:41.491281 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:41.491263 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8229c-77b67bb94b-krzmq" Sep 11 04:19:41.497633 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:41.497613 2568 scope.go:117] "RemoveContainer" containerID="7a0ee62fce4370d03291cf330b6c4d3a365f95a75352ac09d04ab099ea11a4f8" Sep 11 04:19:41.497939 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:19:41.497921 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a0ee62fce4370d03291cf330b6c4d3a365f95a75352ac09d04ab099ea11a4f8\": container with ID starting with 7a0ee62fce4370d03291cf330b6c4d3a365f95a75352ac09d04ab099ea11a4f8 not found: ID does not exist" containerID="7a0ee62fce4370d03291cf330b6c4d3a365f95a75352ac09d04ab099ea11a4f8" Sep 11 04:19:41.498008 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:41.497946 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a0ee62fce4370d03291cf330b6c4d3a365f95a75352ac09d04ab099ea11a4f8"} err="failed to get container status \"7a0ee62fce4370d03291cf330b6c4d3a365f95a75352ac09d04ab099ea11a4f8\": rpc error: code = NotFound desc = could not find container \"7a0ee62fce4370d03291cf330b6c4d3a365f95a75352ac09d04ab099ea11a4f8\": container with ID starting with 7a0ee62fce4370d03291cf330b6c4d3a365f95a75352ac09d04ab099ea11a4f8 not found: ID does not exist" Sep 11 04:19:41.548626 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:41.548595 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gfs25\" (UniqueName: \"kubernetes.io/projected/2f4c1059-02c1-42b5-aed3-052f39640577-kube-api-access-gfs25\") pod \"2f4c1059-02c1-42b5-aed3-052f39640577\" (UID: \"2f4c1059-02c1-42b5-aed3-052f39640577\") " Sep 11 04:19:41.548626 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:41.548633 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f4c1059-02c1-42b5-aed3-052f39640577-maas-api-tls\") pod \"2f4c1059-02c1-42b5-aed3-052f39640577\" (UID: \"2f4c1059-02c1-42b5-aed3-052f39640577\") " Sep 11 04:19:41.548882 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:41.548660 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f4c1059-02c1-42b5-aed3-052f39640577-openshift-service-ca\") pod \"2f4c1059-02c1-42b5-aed3-052f39640577\" (UID: \"2f4c1059-02c1-42b5-aed3-052f39640577\") " Sep 11 04:19:41.548882 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:41.548702 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f4c1059-02c1-42b5-aed3-052f39640577-metrics-certs\") pod \"2f4c1059-02c1-42b5-aed3-052f39640577\" (UID: \"2f4c1059-02c1-42b5-aed3-052f39640577\") " Sep 11 04:19:41.549093 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:41.549059 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f4c1059-02c1-42b5-aed3-052f39640577-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2f4c1059-02c1-42b5-aed3-052f39640577" (UID: "2f4c1059-02c1-42b5-aed3-052f39640577"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 04:19:41.550874 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:41.550841 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f4c1059-02c1-42b5-aed3-052f39640577-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2f4c1059-02c1-42b5-aed3-052f39640577" (UID: "2f4c1059-02c1-42b5-aed3-052f39640577"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 04:19:41.550874 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:41.550856 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f4c1059-02c1-42b5-aed3-052f39640577-kube-api-access-gfs25" (OuterVolumeSpecName: "kube-api-access-gfs25") pod "2f4c1059-02c1-42b5-aed3-052f39640577" (UID: "2f4c1059-02c1-42b5-aed3-052f39640577"). InnerVolumeSpecName "kube-api-access-gfs25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:19:41.551015 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:41.550893 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f4c1059-02c1-42b5-aed3-052f39640577-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2f4c1059-02c1-42b5-aed3-052f39640577" (UID: "2f4c1059-02c1-42b5-aed3-052f39640577"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 04:19:41.649977 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:41.649877 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37f9cd2a-fedc-494f-a25f-a85b04719a84-plugins-config-volume\") pod \"37f9cd2a-fedc-494f-a25f-a85b04719a84\" (UID: \"37f9cd2a-fedc-494f-a25f-a85b04719a84\") " Sep 11 04:19:41.649977 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:41.649946 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4dwbj\" (UniqueName: \"kubernetes.io/projected/37f9cd2a-fedc-494f-a25f-a85b04719a84-kube-api-access-4dwbj\") pod \"37f9cd2a-fedc-494f-a25f-a85b04719a84\" (UID: \"37f9cd2a-fedc-494f-a25f-a85b04719a84\") " Sep 11 04:19:41.650211 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:41.650184 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfs25\" (UniqueName: \"kubernetes.io/projected/2f4c1059-02c1-42b5-aed3-052f39640577-kube-api-access-gfs25\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:19:41.650211 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:41.650205 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f4c1059-02c1-42b5-aed3-052f39640577-maas-api-tls\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:19:41.650290 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:41.650220 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f4c1059-02c1-42b5-aed3-052f39640577-openshift-service-ca\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:19:41.650290 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:41.650234 2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f4c1059-02c1-42b5-aed3-052f39640577-metrics-certs\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:19:41.650290 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:41.650261 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37f9cd2a-fedc-494f-a25f-a85b04719a84-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "37f9cd2a-fedc-494f-a25f-a85b04719a84" (UID: "37f9cd2a-fedc-494f-a25f-a85b04719a84"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 04:19:41.651982 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:41.651963 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37f9cd2a-fedc-494f-a25f-a85b04719a84-kube-api-access-4dwbj" (OuterVolumeSpecName: "kube-api-access-4dwbj") pod "37f9cd2a-fedc-494f-a25f-a85b04719a84" (UID: "37f9cd2a-fedc-494f-a25f-a85b04719a84"). InnerVolumeSpecName "kube-api-access-4dwbj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:19:41.751343 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:41.751303 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dwbj\" (UniqueName: \"kubernetes.io/projected/37f9cd2a-fedc-494f-a25f-a85b04719a84-kube-api-access-4dwbj\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:19:41.751343 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:41.751334 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/37f9cd2a-fedc-494f-a25f-a85b04719a84-plugins-config-volume\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:19:41.768941 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:41.768907 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc0b3d9d-eeba-48a4-a619-b01a96c5ed17" path="/var/lib/kubelet/pods/cc0b3d9d-eeba-48a4-a619-b01a96c5ed17/volumes" Sep 11 04:19:41.785522 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:41.785491 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:19:41.801848 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:41.801823 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8229c-5fbd746f56-lhh7v"] Sep 11 04:19:41.806084 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:41.806061 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-f8229c-5fbd746f56-lhh7v"] Sep 11 04:19:42.379741 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:42.379628 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:19:42.470508 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:42.470474 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:19:42.494889 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:42.494864 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8229c-77b67bb94b-krzmq" Sep 11 04:19:42.517622 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:42.517590 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8229c-77b67bb94b-krzmq"] Sep 11 04:19:42.524064 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:42.524024 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-f8229c-77b67bb94b-krzmq"] Sep 11 04:19:43.600429 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:43.600395 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fdc690-65b64cbcc7-sd4zv"] Sep 11 04:19:43.600934 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:43.600916 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f5e76f9-b2ad-4676-b83c-a8c52adc5233" containerName="payload-processing" Sep 11 04:19:43.600986 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:43.600936 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f5e76f9-b2ad-4676-b83c-a8c52adc5233" containerName="payload-processing" Sep 11 04:19:43.600986 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:43.600950 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f4c1059-02c1-42b5-aed3-052f39640577" containerName="maas-api" Sep 11 04:19:43.600986 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:43.600955 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f4c1059-02c1-42b5-aed3-052f39640577" containerName="maas-api" Sep 11 04:19:43.600986 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:43.600965 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37f9cd2a-fedc-494f-a25f-a85b04719a84" containerName="payload-pre-processing" Sep 11 04:19:43.600986 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:43.600971 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="37f9cd2a-fedc-494f-a25f-a85b04719a84" containerName="payload-pre-processing" Sep 11 04:19:43.600986 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:43.600980 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b643f98-3070-4c1a-834f-0f8dc59db105" containerName="maas-api" Sep 11 04:19:43.600986 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:43.600986 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b643f98-3070-4c1a-834f-0f8dc59db105" containerName="maas-api" Sep 11 04:19:43.601218 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:43.600997 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="082edca2-cfea-4818-9e08-277a51456382" containerName="main" Sep 11 04:19:43.601218 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:43.601002 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="082edca2-cfea-4818-9e08-277a51456382" containerName="main" Sep 11 04:19:43.601218 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:43.601012 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0613c781-47be-4eb7-866d-e00a9a4dd027" containerName="maas-api" Sep 11 04:19:43.601218 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:43.601018 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="0613c781-47be-4eb7-866d-e00a9a4dd027" containerName="maas-api" Sep 11 04:19:43.601218 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:43.601087 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f5e76f9-b2ad-4676-b83c-a8c52adc5233" containerName="payload-processing" Sep 11 04:19:43.601218 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:43.601096 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="082edca2-cfea-4818-9e08-277a51456382" containerName="main" Sep 11 04:19:43.601218 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:43.601102 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="0613c781-47be-4eb7-866d-e00a9a4dd027" containerName="maas-api" Sep 11 04:19:43.601218 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:43.601109 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b643f98-3070-4c1a-834f-0f8dc59db105" containerName="maas-api" Sep 11 04:19:43.601218 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:43.601114 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f4c1059-02c1-42b5-aed3-052f39640577" containerName="maas-api" Sep 11 04:19:43.601218 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:43.601119 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="37f9cd2a-fedc-494f-a25f-a85b04719a84" containerName="payload-pre-processing" Sep 11 04:19:43.605410 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:43.605392 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fdc690-65b64cbcc7-sd4zv" Sep 11 04:19:43.607906 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:43.607881 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-fdc690-serving-cert\"" Sep 11 04:19:43.612891 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:43.612814 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fdc690-65b64cbcc7-sd4zv"] Sep 11 04:19:43.635016 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:43.634983 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:19:43.691878 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:43.691843 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:19:43.767801 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:43.767767 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c70a619-7132-4acf-a6b2-d8a9bb7a21b7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-fdc690-65b64cbcc7-sd4zv\" (UID: \"9c70a619-7132-4acf-a6b2-d8a9bb7a21b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fdc690-65b64cbcc7-sd4zv" Sep 11 04:19:43.767979 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:43.767820 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlh5d\" (UniqueName: \"kubernetes.io/projected/9c70a619-7132-4acf-a6b2-d8a9bb7a21b7-kube-api-access-dlh5d\") pod \"maas-api-e2e-ipp-a-fdc690-65b64cbcc7-sd4zv\" (UID: \"9c70a619-7132-4acf-a6b2-d8a9bb7a21b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fdc690-65b64cbcc7-sd4zv" Sep 11 04:19:43.767979 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:43.767911 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c70a619-7132-4acf-a6b2-d8a9bb7a21b7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-fdc690-65b64cbcc7-sd4zv\" (UID: \"9c70a619-7132-4acf-a6b2-d8a9bb7a21b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fdc690-65b64cbcc7-sd4zv" Sep 11 04:19:43.768107 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:43.767991 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c70a619-7132-4acf-a6b2-d8a9bb7a21b7-metrics-certs\") pod \"maas-api-e2e-ipp-a-fdc690-65b64cbcc7-sd4zv\" (UID: \"9c70a619-7132-4acf-a6b2-d8a9bb7a21b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fdc690-65b64cbcc7-sd4zv" Sep 11 04:19:43.768590 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:43.768570 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f4c1059-02c1-42b5-aed3-052f39640577" path="/var/lib/kubelet/pods/2f4c1059-02c1-42b5-aed3-052f39640577/volumes" Sep 11 04:19:43.768917 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:43.768904 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37f9cd2a-fedc-494f-a25f-a85b04719a84" path="/var/lib/kubelet/pods/37f9cd2a-fedc-494f-a25f-a85b04719a84/volumes" Sep 11 04:19:43.835062 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:43.835018 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:19:43.869285 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:43.869201 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c70a619-7132-4acf-a6b2-d8a9bb7a21b7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-fdc690-65b64cbcc7-sd4zv\" (UID: \"9c70a619-7132-4acf-a6b2-d8a9bb7a21b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fdc690-65b64cbcc7-sd4zv" Sep 11 04:19:43.869285 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:43.869251 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c70a619-7132-4acf-a6b2-d8a9bb7a21b7-metrics-certs\") pod \"maas-api-e2e-ipp-a-fdc690-65b64cbcc7-sd4zv\" (UID: \"9c70a619-7132-4acf-a6b2-d8a9bb7a21b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fdc690-65b64cbcc7-sd4zv" Sep 11 04:19:43.869512 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:43.869302 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c70a619-7132-4acf-a6b2-d8a9bb7a21b7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-fdc690-65b64cbcc7-sd4zv\" (UID: \"9c70a619-7132-4acf-a6b2-d8a9bb7a21b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fdc690-65b64cbcc7-sd4zv" Sep 11 04:19:43.869512 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:43.869336 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlh5d\" (UniqueName: \"kubernetes.io/projected/9c70a619-7132-4acf-a6b2-d8a9bb7a21b7-kube-api-access-dlh5d\") pod \"maas-api-e2e-ipp-a-fdc690-65b64cbcc7-sd4zv\" (UID: \"9c70a619-7132-4acf-a6b2-d8a9bb7a21b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fdc690-65b64cbcc7-sd4zv" Sep 11 04:19:43.870007 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:43.869981 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c70a619-7132-4acf-a6b2-d8a9bb7a21b7-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-fdc690-65b64cbcc7-sd4zv\" (UID: \"9c70a619-7132-4acf-a6b2-d8a9bb7a21b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fdc690-65b64cbcc7-sd4zv" Sep 11 04:19:43.871903 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:43.871884 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c70a619-7132-4acf-a6b2-d8a9bb7a21b7-maas-api-tls\") pod \"maas-api-e2e-ipp-a-fdc690-65b64cbcc7-sd4zv\" (UID: \"9c70a619-7132-4acf-a6b2-d8a9bb7a21b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fdc690-65b64cbcc7-sd4zv" Sep 11 04:19:43.871967 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:43.871936 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c70a619-7132-4acf-a6b2-d8a9bb7a21b7-metrics-certs\") pod \"maas-api-e2e-ipp-a-fdc690-65b64cbcc7-sd4zv\" (UID: \"9c70a619-7132-4acf-a6b2-d8a9bb7a21b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fdc690-65b64cbcc7-sd4zv" Sep 11 04:19:43.881185 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:43.881161 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlh5d\" (UniqueName: \"kubernetes.io/projected/9c70a619-7132-4acf-a6b2-d8a9bb7a21b7-kube-api-access-dlh5d\") pod \"maas-api-e2e-ipp-a-fdc690-65b64cbcc7-sd4zv\" (UID: \"9c70a619-7132-4acf-a6b2-d8a9bb7a21b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fdc690-65b64cbcc7-sd4zv" Sep 11 04:19:43.919076 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:43.919000 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fdc690-65b64cbcc7-sd4zv" Sep 11 04:19:44.104700 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:44.104660 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:19:44.197403 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:44.197378 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fdc690-65b64cbcc7-sd4zv"] Sep 11 04:19:44.199529 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:19:44.199505 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c70a619_7132_4acf_a6b2_d8a9bb7a21b7.slice/crio-d02d5fb5f7d401e98872b384c2852fa921269187595a7507c92836140abf1fb8 WatchSource:0}: Error finding container d02d5fb5f7d401e98872b384c2852fa921269187595a7507c92836140abf1fb8: Status 404 returned error can't find the container with id d02d5fb5f7d401e98872b384c2852fa921269187595a7507c92836140abf1fb8 Sep 11 04:19:44.201943 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:44.201916 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:19:44.505943 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:44.505851 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fdc690-65b64cbcc7-sd4zv" event={"ID":"9c70a619-7132-4acf-a6b2-d8a9bb7a21b7","Type":"ContainerStarted","Data":"d02d5fb5f7d401e98872b384c2852fa921269187595a7507c92836140abf1fb8"} Sep 11 04:19:45.512715 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:45.512679 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fdc690-65b64cbcc7-sd4zv" event={"ID":"9c70a619-7132-4acf-a6b2-d8a9bb7a21b7","Type":"ContainerStarted","Data":"335ee90e9fec8c9ff0c915c3edf777e1d0d5b11358a8a75ba8aa27a1df47795e"} Sep 11 04:19:45.513144 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:45.512743 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fdc690-65b64cbcc7-sd4zv" Sep 11 04:19:45.531753 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:45.531701 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fdc690-65b64cbcc7-sd4zv" podStartSLOduration=2.129505725 podStartE2EDuration="2.531683527s" totalImagesPullingTime="402.177802ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:19:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:19:45.529839352 +0000 UTC m=+1600.283470758" watchObservedRunningTime="2026-09-11 04:19:45.531683527 +0000 UTC m=+1600.285314950" Sep 11 04:19:47.581187 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:47.581137 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:19:47.646769 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:47.646732 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:19:47.845807 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:47.845722 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:19:51.523870 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:51.523834 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fdc690-65b64cbcc7-sd4zv" Sep 11 04:19:52.202557 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:52.202515 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-cc6708/rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp" podUID="fa86d199-8a1e-4915-bca5-1524cbbf3aba" containerName="main" containerID="cri-o://c6329db6393e60ce89efe06badd132028c7ef046bea4b898107e263ddf6ab345" gracePeriod=45 Sep 11 04:19:52.452255 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:52.452227 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-cc6708/rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp" Sep 11 04:19:52.543547 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:52.543445 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vq6db\" (UniqueName: \"kubernetes.io/projected/fa86d199-8a1e-4915-bca5-1524cbbf3aba-kube-api-access-vq6db\") pod \"fa86d199-8a1e-4915-bca5-1524cbbf3aba\" (UID: \"fa86d199-8a1e-4915-bca5-1524cbbf3aba\") " Sep 11 04:19:52.543547 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:52.543493 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa86d199-8a1e-4915-bca5-1524cbbf3aba-model-cache\") pod \"fa86d199-8a1e-4915-bca5-1524cbbf3aba\" (UID: \"fa86d199-8a1e-4915-bca5-1524cbbf3aba\") " Sep 11 04:19:52.543547 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:52.543545 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa86d199-8a1e-4915-bca5-1524cbbf3aba-dshm\") pod \"fa86d199-8a1e-4915-bca5-1524cbbf3aba\" (UID: \"fa86d199-8a1e-4915-bca5-1524cbbf3aba\") " Sep 11 04:19:52.544150 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:52.543566 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa86d199-8a1e-4915-bca5-1524cbbf3aba-home\") pod \"fa86d199-8a1e-4915-bca5-1524cbbf3aba\" (UID: \"fa86d199-8a1e-4915-bca5-1524cbbf3aba\") " Sep 11 04:19:52.544150 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:52.543603 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa86d199-8a1e-4915-bca5-1524cbbf3aba-tls-certs\") pod \"fa86d199-8a1e-4915-bca5-1524cbbf3aba\" (UID: \"fa86d199-8a1e-4915-bca5-1524cbbf3aba\") " Sep 11 04:19:52.544150 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:52.543635 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa86d199-8a1e-4915-bca5-1524cbbf3aba-tmp-dir\") pod \"fa86d199-8a1e-4915-bca5-1524cbbf3aba\" (UID: \"fa86d199-8a1e-4915-bca5-1524cbbf3aba\") " Sep 11 04:19:52.544150 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:52.543834 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa86d199-8a1e-4915-bca5-1524cbbf3aba-model-cache" (OuterVolumeSpecName: "model-cache") pod "fa86d199-8a1e-4915-bca5-1524cbbf3aba" (UID: "fa86d199-8a1e-4915-bca5-1524cbbf3aba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:19:52.544150 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:52.543955 2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa86d199-8a1e-4915-bca5-1524cbbf3aba-model-cache\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:19:52.544150 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:52.544039 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa86d199-8a1e-4915-bca5-1524cbbf3aba-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fa86d199-8a1e-4915-bca5-1524cbbf3aba" (UID: "fa86d199-8a1e-4915-bca5-1524cbbf3aba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:19:52.544150 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:52.544085 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa86d199-8a1e-4915-bca5-1524cbbf3aba-home" (OuterVolumeSpecName: "home") pod "fa86d199-8a1e-4915-bca5-1524cbbf3aba" (UID: "fa86d199-8a1e-4915-bca5-1524cbbf3aba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:19:52.546176 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:52.546142 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa86d199-8a1e-4915-bca5-1524cbbf3aba-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fa86d199-8a1e-4915-bca5-1524cbbf3aba" (UID: "fa86d199-8a1e-4915-bca5-1524cbbf3aba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 04:19:52.546309 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:52.546193 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa86d199-8a1e-4915-bca5-1524cbbf3aba-dshm" (OuterVolumeSpecName: "dshm") pod "fa86d199-8a1e-4915-bca5-1524cbbf3aba" (UID: "fa86d199-8a1e-4915-bca5-1524cbbf3aba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:19:52.546367 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:52.546298 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa86d199-8a1e-4915-bca5-1524cbbf3aba-kube-api-access-vq6db" (OuterVolumeSpecName: "kube-api-access-vq6db") pod "fa86d199-8a1e-4915-bca5-1524cbbf3aba" (UID: "fa86d199-8a1e-4915-bca5-1524cbbf3aba"). InnerVolumeSpecName "kube-api-access-vq6db". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:19:52.554031 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:52.554002 2568 generic.go:356] "Generic (PLEG): container finished" podID="fa86d199-8a1e-4915-bca5-1524cbbf3aba" containerID="c6329db6393e60ce89efe06badd132028c7ef046bea4b898107e263ddf6ab345" exitCode=0 Sep 11 04:19:52.554195 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:52.554076 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-cc6708/rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp" Sep 11 04:19:52.554195 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:52.554119 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-cc6708/rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp" event={"ID":"fa86d199-8a1e-4915-bca5-1524cbbf3aba","Type":"ContainerDied","Data":"c6329db6393e60ce89efe06badd132028c7ef046bea4b898107e263ddf6ab345"} Sep 11 04:19:52.554195 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:52.554182 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-cc6708/rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp" event={"ID":"fa86d199-8a1e-4915-bca5-1524cbbf3aba","Type":"ContainerDied","Data":"5e338b6e8a969f791bcc673d5fc18992107a4fb158bb6cb78010a9cf4910f1aa"} Sep 11 04:19:52.554371 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:52.554205 2568 scope.go:117] "RemoveContainer" containerID="c6329db6393e60ce89efe06badd132028c7ef046bea4b898107e263ddf6ab345" Sep 11 04:19:52.566449 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:52.566429 2568 scope.go:117] "RemoveContainer" containerID="c6329db6393e60ce89efe06badd132028c7ef046bea4b898107e263ddf6ab345" Sep 11 04:19:52.566715 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:19:52.566696 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6329db6393e60ce89efe06badd132028c7ef046bea4b898107e263ddf6ab345\": container with ID starting with c6329db6393e60ce89efe06badd132028c7ef046bea4b898107e263ddf6ab345 not found: ID does not exist" containerID="c6329db6393e60ce89efe06badd132028c7ef046bea4b898107e263ddf6ab345" Sep 11 04:19:52.566782 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:52.566723 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6329db6393e60ce89efe06badd132028c7ef046bea4b898107e263ddf6ab345"} err="failed to get container status \"c6329db6393e60ce89efe06badd132028c7ef046bea4b898107e263ddf6ab345\": rpc error: code = NotFound desc = could not find container \"c6329db6393e60ce89efe06badd132028c7ef046bea4b898107e263ddf6ab345\": container with ID starting with c6329db6393e60ce89efe06badd132028c7ef046bea4b898107e263ddf6ab345 not found: ID does not exist" Sep 11 04:19:52.577286 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:52.577266 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cc6708/rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp"] Sep 11 04:19:52.580294 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:52.580272 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-cc6708/rate-test-model-cc6708-kserve-77d4c7c7f9-gqqxp"] Sep 11 04:19:52.644610 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:52.644571 2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa86d199-8a1e-4915-bca5-1524cbbf3aba-dshm\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:19:52.644769 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:52.644628 2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa86d199-8a1e-4915-bca5-1524cbbf3aba-home\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:19:52.644769 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:52.644667 2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa86d199-8a1e-4915-bca5-1524cbbf3aba-tls-certs\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:19:52.644769 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:52.644682 2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa86d199-8a1e-4915-bca5-1524cbbf3aba-tmp-dir\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:19:52.644769 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:52.644694 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq6db\" (UniqueName: \"kubernetes.io/projected/fa86d199-8a1e-4915-bca5-1524cbbf3aba-kube-api-access-vq6db\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:19:53.768775 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:19:53.768734 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa86d199-8a1e-4915-bca5-1524cbbf3aba" path="/var/lib/kubelet/pods/fa86d199-8a1e-4915-bca5-1524cbbf3aba/volumes" Sep 11 04:20:06.033149 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:06.033114 2568 scope.go:117] "RemoveContainer" containerID="876620718c77ec7845193217f4b5f481d8307473fd32b21a74170cddca03b7e1" Sep 11 04:20:07.368992 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:07.368952 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:20:08.099704 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:08.099673 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:20:08.433183 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:08.433140 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:20:08.723226 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:08.723141 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:20:11.449268 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:11.449228 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:20:11.529531 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:11.529490 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:20:12.568531 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:12.568498 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a5a85-7b88d78d8f-tvh8q"] Sep 11 04:20:12.569074 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:12.569035 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa86d199-8a1e-4915-bca5-1524cbbf3aba" containerName="main" Sep 11 04:20:12.569074 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:12.569072 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa86d199-8a1e-4915-bca5-1524cbbf3aba" containerName="main" Sep 11 04:20:12.569221 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:12.569162 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa86d199-8a1e-4915-bca5-1524cbbf3aba" containerName="main" Sep 11 04:20:12.572238 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:12.572192 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a5a85-7b88d78d8f-tvh8q" Sep 11 04:20:12.575318 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:12.575091 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-7a5a85-serving-cert\"" Sep 11 04:20:12.586323 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:12.586300 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a5a85-7b88d78d8f-tvh8q"] Sep 11 04:20:12.604603 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:12.604576 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:20:12.616754 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:12.616730 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4dfd252-ab7f-47da-8ad4-d4c89165ba6c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7a5a85-7b88d78d8f-tvh8q\" (UID: \"a4dfd252-ab7f-47da-8ad4-d4c89165ba6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a5a85-7b88d78d8f-tvh8q" Sep 11 04:20:12.616900 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:12.616766 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4dfd252-ab7f-47da-8ad4-d4c89165ba6c-metrics-certs\") pod \"maas-api-e2e-ipp-b-7a5a85-7b88d78d8f-tvh8q\" (UID: \"a4dfd252-ab7f-47da-8ad4-d4c89165ba6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a5a85-7b88d78d8f-tvh8q" Sep 11 04:20:12.616900 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:12.616792 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjhmm\" (UniqueName: \"kubernetes.io/projected/a4dfd252-ab7f-47da-8ad4-d4c89165ba6c-kube-api-access-zjhmm\") pod \"maas-api-e2e-ipp-b-7a5a85-7b88d78d8f-tvh8q\" (UID: \"a4dfd252-ab7f-47da-8ad4-d4c89165ba6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a5a85-7b88d78d8f-tvh8q" Sep 11 04:20:12.617016 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:12.616912 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4dfd252-ab7f-47da-8ad4-d4c89165ba6c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7a5a85-7b88d78d8f-tvh8q\" (UID: \"a4dfd252-ab7f-47da-8ad4-d4c89165ba6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a5a85-7b88d78d8f-tvh8q" Sep 11 04:20:12.705075 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:12.705027 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:20:12.718238 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:12.718212 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjhmm\" (UniqueName: \"kubernetes.io/projected/a4dfd252-ab7f-47da-8ad4-d4c89165ba6c-kube-api-access-zjhmm\") pod \"maas-api-e2e-ipp-b-7a5a85-7b88d78d8f-tvh8q\" (UID: \"a4dfd252-ab7f-47da-8ad4-d4c89165ba6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a5a85-7b88d78d8f-tvh8q" Sep 11 04:20:12.718375 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:12.718281 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4dfd252-ab7f-47da-8ad4-d4c89165ba6c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7a5a85-7b88d78d8f-tvh8q\" (UID: \"a4dfd252-ab7f-47da-8ad4-d4c89165ba6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a5a85-7b88d78d8f-tvh8q" Sep 11 04:20:12.718375 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:12.718350 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4dfd252-ab7f-47da-8ad4-d4c89165ba6c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7a5a85-7b88d78d8f-tvh8q\" (UID: \"a4dfd252-ab7f-47da-8ad4-d4c89165ba6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a5a85-7b88d78d8f-tvh8q" Sep 11 04:20:12.718488 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:12.718385 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4dfd252-ab7f-47da-8ad4-d4c89165ba6c-metrics-certs\") pod \"maas-api-e2e-ipp-b-7a5a85-7b88d78d8f-tvh8q\" (UID: \"a4dfd252-ab7f-47da-8ad4-d4c89165ba6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a5a85-7b88d78d8f-tvh8q" Sep 11 04:20:12.718488 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:20:12.718453 2568 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a5a85-serving-cert: secret "maas-api-e2e-ipp-b-7a5a85-serving-cert" not found Sep 11 04:20:12.718614 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:20:12.718545 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4dfd252-ab7f-47da-8ad4-d4c89165ba6c-maas-api-tls podName:a4dfd252-ab7f-47da-8ad4-d4c89165ba6c nodeName:}" failed. No retries permitted until 2026-09-11 04:20:13.218523461 +0000 UTC m=+1627.972154856 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a4dfd252-ab7f-47da-8ad4-d4c89165ba6c-maas-api-tls") pod "maas-api-e2e-ipp-b-7a5a85-7b88d78d8f-tvh8q" (UID: "a4dfd252-ab7f-47da-8ad4-d4c89165ba6c") : secret "maas-api-e2e-ipp-b-7a5a85-serving-cert" not found Sep 11 04:20:12.718998 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:12.718975 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4dfd252-ab7f-47da-8ad4-d4c89165ba6c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-7a5a85-7b88d78d8f-tvh8q\" (UID: \"a4dfd252-ab7f-47da-8ad4-d4c89165ba6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a5a85-7b88d78d8f-tvh8q" Sep 11 04:20:12.720814 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:12.720793 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4dfd252-ab7f-47da-8ad4-d4c89165ba6c-metrics-certs\") pod \"maas-api-e2e-ipp-b-7a5a85-7b88d78d8f-tvh8q\" (UID: \"a4dfd252-ab7f-47da-8ad4-d4c89165ba6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a5a85-7b88d78d8f-tvh8q" Sep 11 04:20:12.728616 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:12.728596 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjhmm\" (UniqueName: \"kubernetes.io/projected/a4dfd252-ab7f-47da-8ad4-d4c89165ba6c-kube-api-access-zjhmm\") pod \"maas-api-e2e-ipp-b-7a5a85-7b88d78d8f-tvh8q\" (UID: \"a4dfd252-ab7f-47da-8ad4-d4c89165ba6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a5a85-7b88d78d8f-tvh8q" Sep 11 04:20:13.080796 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:13.080734 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:20:13.178555 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:13.178520 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:20:13.222836 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:13.222802 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4dfd252-ab7f-47da-8ad4-d4c89165ba6c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7a5a85-7b88d78d8f-tvh8q\" (UID: \"a4dfd252-ab7f-47da-8ad4-d4c89165ba6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a5a85-7b88d78d8f-tvh8q" Sep 11 04:20:13.225311 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:13.225285 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4dfd252-ab7f-47da-8ad4-d4c89165ba6c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-7a5a85-7b88d78d8f-tvh8q\" (UID: \"a4dfd252-ab7f-47da-8ad4-d4c89165ba6c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a5a85-7b88d78d8f-tvh8q" Sep 11 04:20:13.253596 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:13.253564 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7a5a85-86d55bdcdf-c5bfz"] Sep 11 04:20:13.257295 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:13.257280 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7a5a85-86d55bdcdf-c5bfz" Sep 11 04:20:13.261250 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:13.261202 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-7a5a85\"" Sep 11 04:20:13.261250 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:13.261209 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-7a5a85-dockercfg-l66p9\"" Sep 11 04:20:13.270575 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:13.270550 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7a5a85-86d55bdcdf-c5bfz"] Sep 11 04:20:13.280081 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:13.280019 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:20:13.323372 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:13.323325 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rllw\" (UniqueName: \"kubernetes.io/projected/8e7e8a64-d384-426f-83c1-d6e33595c7e5-kube-api-access-2rllw\") pod \"payload-processing-e2e-ipp-b-7a5a85-86d55bdcdf-c5bfz\" (UID: \"8e7e8a64-d384-426f-83c1-d6e33595c7e5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7a5a85-86d55bdcdf-c5bfz" Sep 11 04:20:13.323594 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:13.323403 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e7e8a64-d384-426f-83c1-d6e33595c7e5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7a5a85-86d55bdcdf-c5bfz\" (UID: \"8e7e8a64-d384-426f-83c1-d6e33595c7e5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7a5a85-86d55bdcdf-c5bfz" Sep 11 04:20:13.424318 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:13.424278 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e7e8a64-d384-426f-83c1-d6e33595c7e5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7a5a85-86d55bdcdf-c5bfz\" (UID: \"8e7e8a64-d384-426f-83c1-d6e33595c7e5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7a5a85-86d55bdcdf-c5bfz" Sep 11 04:20:13.424499 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:13.424382 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rllw\" (UniqueName: \"kubernetes.io/projected/8e7e8a64-d384-426f-83c1-d6e33595c7e5-kube-api-access-2rllw\") pod \"payload-processing-e2e-ipp-b-7a5a85-86d55bdcdf-c5bfz\" (UID: \"8e7e8a64-d384-426f-83c1-d6e33595c7e5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7a5a85-86d55bdcdf-c5bfz" Sep 11 04:20:13.424997 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:13.424978 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e7e8a64-d384-426f-83c1-d6e33595c7e5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7a5a85-86d55bdcdf-c5bfz\" (UID: \"8e7e8a64-d384-426f-83c1-d6e33595c7e5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7a5a85-86d55bdcdf-c5bfz" Sep 11 04:20:13.432332 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:13.432302 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rllw\" (UniqueName: \"kubernetes.io/projected/8e7e8a64-d384-426f-83c1-d6e33595c7e5-kube-api-access-2rllw\") pod \"payload-processing-e2e-ipp-b-7a5a85-86d55bdcdf-c5bfz\" (UID: \"8e7e8a64-d384-426f-83c1-d6e33595c7e5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7a5a85-86d55bdcdf-c5bfz" Sep 11 04:20:13.484619 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:13.484591 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a5a85-7b88d78d8f-tvh8q" Sep 11 04:20:13.568577 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:13.568544 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7a5a85-86d55bdcdf-c5bfz" Sep 11 04:20:13.756209 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:13.756099 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a5a85-7b88d78d8f-tvh8q"] Sep 11 04:20:13.758552 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:20:13.758521 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4dfd252_ab7f_47da_8ad4_d4c89165ba6c.slice/crio-f9da6df41d36fa84678eb298fa9264f03f8f63c820c18aee62021dbfdd959174 WatchSource:0}: Error finding container f9da6df41d36fa84678eb298fa9264f03f8f63c820c18aee62021dbfdd959174: Status 404 returned error can't find the container with id f9da6df41d36fa84678eb298fa9264f03f8f63c820c18aee62021dbfdd959174 Sep 11 04:20:13.841365 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:13.841332 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7a5a85-86d55bdcdf-c5bfz"] Sep 11 04:20:13.843964 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:20:13.843932 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e7e8a64_d384_426f_83c1_d6e33595c7e5.slice/crio-841e255ecbe468fd12bf9dea246ba7235b38370283583199d3d176d0ec11c5a9 WatchSource:0}: Error finding container 841e255ecbe468fd12bf9dea246ba7235b38370283583199d3d176d0ec11c5a9: Status 404 returned error can't find the container with id 841e255ecbe468fd12bf9dea246ba7235b38370283583199d3d176d0ec11c5a9 Sep 11 04:20:14.669106 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:14.669071 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a5a85-7b88d78d8f-tvh8q" event={"ID":"a4dfd252-ab7f-47da-8ad4-d4c89165ba6c","Type":"ContainerStarted","Data":"f9da6df41d36fa84678eb298fa9264f03f8f63c820c18aee62021dbfdd959174"} Sep 11 04:20:14.670462 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:14.670436 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7a5a85-86d55bdcdf-c5bfz" event={"ID":"8e7e8a64-d384-426f-83c1-d6e33595c7e5","Type":"ContainerStarted","Data":"b0fde5fb63ceab0cab4404d152b83cec0b89f6d90b9806df23438f2070455769"} Sep 11 04:20:14.670609 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:14.670485 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7a5a85-86d55bdcdf-c5bfz" event={"ID":"8e7e8a64-d384-426f-83c1-d6e33595c7e5","Type":"ContainerStarted","Data":"841e255ecbe468fd12bf9dea246ba7235b38370283583199d3d176d0ec11c5a9"} Sep 11 04:20:14.670609 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:14.670578 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7a5a85-86d55bdcdf-c5bfz" Sep 11 04:20:14.692564 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:14.692520 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-7a5a85-86d55bdcdf-c5bfz" podStartSLOduration=1.692506104 podStartE2EDuration="1.692506104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:20:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:20:14.690331965 +0000 UTC m=+1629.443963367" watchObservedRunningTime="2026-09-11 04:20:14.692506104 +0000 UTC m=+1629.446137505" Sep 11 04:20:15.678855 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:15.678819 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a5a85-7b88d78d8f-tvh8q" event={"ID":"a4dfd252-ab7f-47da-8ad4-d4c89165ba6c","Type":"ContainerStarted","Data":"bdd10d942a29182fc0d60b9ebaeb1f4616a9b168fbf9ea3416c5a327142b5275"} Sep 11 04:20:15.679304 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:15.678885 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a5a85-7b88d78d8f-tvh8q" Sep 11 04:20:15.695233 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:15.695181 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a5a85-7b88d78d8f-tvh8q" podStartSLOduration=2.157215972 podStartE2EDuration="3.695161149s" totalImagesPullingTime="1.537945177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:20:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:20:15.694583887 +0000 UTC m=+1630.448215291" watchObservedRunningTime="2026-09-11 04:20:15.695161149 +0000 UTC m=+1630.448792555" Sep 11 04:20:21.688872 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:21.688837 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a5a85-7b88d78d8f-tvh8q" Sep 11 04:20:25.681826 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:25.681796 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7a5a85-86d55bdcdf-c5bfz" Sep 11 04:20:36.501480 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:36.501447 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:20:41.759490 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:41.759453 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-7v7hl_879b00c8-d09a-4fc0-a80d-30261c01c32a/payload-processing/0.log" Sep 11 04:20:41.887224 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:41.887192 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-7a5a85-86d55bdcdf-c5bfz_8e7e8a64-d384-426f-83c1-d6e33595c7e5/payload-processing/0.log" Sep 11 04:20:47.184339 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:20:47.184301 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:21:06.066712 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:06.066656 2568 scope.go:117] "RemoveContainer" containerID="e49ebe998fb40dd210eb17a5877e56c00c97c672b73119ce70553edd70b29bf9" Sep 11 04:21:12.647063 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:12.647026 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-7v7hl_879b00c8-d09a-4fc0-a80d-30261c01c32a/payload-processing/0.log" Sep 11 04:21:12.769098 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:12.769064 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-7a5a85-86d55bdcdf-c5bfz_8e7e8a64-d384-426f-83c1-d6e33595c7e5/payload-processing/0.log" Sep 11 04:21:13.072670 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:13.072634 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:21:13.387296 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:13.387199 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:21:13.562907 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:13.562875 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:21:16.834762 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:16.834710 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:21:16.933544 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:16.933509 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:21:17.989924 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:17.989889 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-84b423-7dfc5f575-lw628"] Sep 11 04:21:17.993579 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:17.993557 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-84b423-7dfc5f575-lw628" Sep 11 04:21:17.997877 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:17.997847 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-84b423-serving-cert\"" Sep 11 04:21:18.006940 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:18.006845 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-84b423-7dfc5f575-lw628"] Sep 11 04:21:18.027574 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:18.027550 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:21:18.101719 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:18.101685 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:21:18.107827 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:18.107799 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d175486-6247-4c1f-926d-a4f0346fb7ea-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-84b423-7dfc5f575-lw628\" (UID: \"6d175486-6247-4c1f-926d-a4f0346fb7ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-84b423-7dfc5f575-lw628" Sep 11 04:21:18.107967 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:18.107832 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t64ll\" (UniqueName: \"kubernetes.io/projected/6d175486-6247-4c1f-926d-a4f0346fb7ea-kube-api-access-t64ll\") pod \"maas-api-e2e-ipp-cleanup-84b423-7dfc5f575-lw628\" (UID: \"6d175486-6247-4c1f-926d-a4f0346fb7ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-84b423-7dfc5f575-lw628" Sep 11 04:21:18.107967 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:18.107862 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d175486-6247-4c1f-926d-a4f0346fb7ea-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-84b423-7dfc5f575-lw628\" (UID: \"6d175486-6247-4c1f-926d-a4f0346fb7ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-84b423-7dfc5f575-lw628" Sep 11 04:21:18.107967 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:18.107939 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d175486-6247-4c1f-926d-a4f0346fb7ea-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-84b423-7dfc5f575-lw628\" (UID: \"6d175486-6247-4c1f-926d-a4f0346fb7ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-84b423-7dfc5f575-lw628" Sep 11 04:21:18.208973 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:18.208940 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d175486-6247-4c1f-926d-a4f0346fb7ea-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-84b423-7dfc5f575-lw628\" (UID: \"6d175486-6247-4c1f-926d-a4f0346fb7ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-84b423-7dfc5f575-lw628" Sep 11 04:21:18.208973 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:18.208974 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t64ll\" (UniqueName: \"kubernetes.io/projected/6d175486-6247-4c1f-926d-a4f0346fb7ea-kube-api-access-t64ll\") pod \"maas-api-e2e-ipp-cleanup-84b423-7dfc5f575-lw628\" (UID: \"6d175486-6247-4c1f-926d-a4f0346fb7ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-84b423-7dfc5f575-lw628" Sep 11 04:21:18.209221 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:18.209004 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d175486-6247-4c1f-926d-a4f0346fb7ea-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-84b423-7dfc5f575-lw628\" (UID: \"6d175486-6247-4c1f-926d-a4f0346fb7ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-84b423-7dfc5f575-lw628" Sep 11 04:21:18.209221 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:18.209075 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d175486-6247-4c1f-926d-a4f0346fb7ea-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-84b423-7dfc5f575-lw628\" (UID: \"6d175486-6247-4c1f-926d-a4f0346fb7ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-84b423-7dfc5f575-lw628" Sep 11 04:21:18.209683 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:18.209660 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d175486-6247-4c1f-926d-a4f0346fb7ea-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-84b423-7dfc5f575-lw628\" (UID: \"6d175486-6247-4c1f-926d-a4f0346fb7ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-84b423-7dfc5f575-lw628" Sep 11 04:21:18.211561 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:18.211537 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d175486-6247-4c1f-926d-a4f0346fb7ea-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-84b423-7dfc5f575-lw628\" (UID: \"6d175486-6247-4c1f-926d-a4f0346fb7ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-84b423-7dfc5f575-lw628" Sep 11 04:21:18.211641 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:18.211566 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d175486-6247-4c1f-926d-a4f0346fb7ea-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-84b423-7dfc5f575-lw628\" (UID: \"6d175486-6247-4c1f-926d-a4f0346fb7ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-84b423-7dfc5f575-lw628" Sep 11 04:21:18.217248 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:18.217228 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t64ll\" (UniqueName: \"kubernetes.io/projected/6d175486-6247-4c1f-926d-a4f0346fb7ea-kube-api-access-t64ll\") pod \"maas-api-e2e-ipp-cleanup-84b423-7dfc5f575-lw628\" (UID: \"6d175486-6247-4c1f-926d-a4f0346fb7ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-84b423-7dfc5f575-lw628" Sep 11 04:21:18.220659 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:18.220626 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:21:18.303765 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:18.303686 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-84b423-7dfc5f575-lw628" Sep 11 04:21:18.489972 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:18.489942 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:21:18.582219 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:18.582194 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-84b423-7dfc5f575-lw628"] Sep 11 04:21:18.585207 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:18.585141 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:21:18.585339 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:21:18.585321 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d175486_6247_4c1f_926d_a4f0346fb7ea.slice/crio-0f40ef9a5e655c3d3e5c55eff6b06f49a3fba158c03d8b7346c1c7435603c937 WatchSource:0}: Error finding container 0f40ef9a5e655c3d3e5c55eff6b06f49a3fba158c03d8b7346c1c7435603c937: Status 404 returned error can't find the container with id 0f40ef9a5e655c3d3e5c55eff6b06f49a3fba158c03d8b7346c1c7435603c937 Sep 11 04:21:18.998438 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:18.998288 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-84b423-7dfc5f575-lw628" event={"ID":"6d175486-6247-4c1f-926d-a4f0346fb7ea","Type":"ContainerStarted","Data":"0f40ef9a5e655c3d3e5c55eff6b06f49a3fba158c03d8b7346c1c7435603c937"} Sep 11 04:21:20.004605 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:20.004565 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-84b423-7dfc5f575-lw628" event={"ID":"6d175486-6247-4c1f-926d-a4f0346fb7ea","Type":"ContainerStarted","Data":"59d75a1f6266fda592af1eb1fa7880fb827d987eef6d8fe129d5c9aeb6560504"} Sep 11 04:21:20.005005 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:20.004635 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-84b423-7dfc5f575-lw628" Sep 11 04:21:20.024891 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:20.024844 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-84b423-7dfc5f575-lw628" podStartSLOduration=2.650505456 podStartE2EDuration="3.024827857s" totalImagesPullingTime="374.322401ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:21:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:21:20.022521661 +0000 UTC m=+1694.776153064" watchObservedRunningTime="2026-09-11 04:21:20.024827857 +0000 UTC m=+1694.778459259" Sep 11 04:21:26.016279 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:26.016247 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-84b423-7dfc5f575-lw628" Sep 11 04:21:41.902735 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:41.902634 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:21:48.472861 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:48.472828 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-84b423-7dfc5f575-lw628"] Sep 11 04:21:48.473353 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:48.473103 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-84b423-7dfc5f575-lw628" podUID="6d175486-6247-4c1f-926d-a4f0346fb7ea" containerName="maas-api" containerID="cri-o://59d75a1f6266fda592af1eb1fa7880fb827d987eef6d8fe129d5c9aeb6560504" gracePeriod=30 Sep 11 04:21:48.514079 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:48.514019 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:21:48.590701 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:48.590659 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:21:48.660973 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:48.660941 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:21:48.729902 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:48.729834 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-84b423-7dfc5f575-lw628" Sep 11 04:21:48.730429 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:48.730401 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:21:48.784346 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:48.784320 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d175486-6247-4c1f-926d-a4f0346fb7ea-openshift-service-ca\") pod \"6d175486-6247-4c1f-926d-a4f0346fb7ea\" (UID: \"6d175486-6247-4c1f-926d-a4f0346fb7ea\") " Sep 11 04:21:48.784485 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:48.784360 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d175486-6247-4c1f-926d-a4f0346fb7ea-metrics-certs\") pod \"6d175486-6247-4c1f-926d-a4f0346fb7ea\" (UID: \"6d175486-6247-4c1f-926d-a4f0346fb7ea\") " Sep 11 04:21:48.784485 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:48.784419 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d175486-6247-4c1f-926d-a4f0346fb7ea-maas-api-tls\") pod \"6d175486-6247-4c1f-926d-a4f0346fb7ea\" (UID: \"6d175486-6247-4c1f-926d-a4f0346fb7ea\") " Sep 11 04:21:48.784485 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:48.784467 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t64ll\" (UniqueName: \"kubernetes.io/projected/6d175486-6247-4c1f-926d-a4f0346fb7ea-kube-api-access-t64ll\") pod \"6d175486-6247-4c1f-926d-a4f0346fb7ea\" (UID: \"6d175486-6247-4c1f-926d-a4f0346fb7ea\") " Sep 11 04:21:48.784697 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:48.784669 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d175486-6247-4c1f-926d-a4f0346fb7ea-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6d175486-6247-4c1f-926d-a4f0346fb7ea" (UID: "6d175486-6247-4c1f-926d-a4f0346fb7ea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 04:21:48.786476 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:48.786449 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d175486-6247-4c1f-926d-a4f0346fb7ea-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6d175486-6247-4c1f-926d-a4f0346fb7ea" (UID: "6d175486-6247-4c1f-926d-a4f0346fb7ea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 04:21:48.786580 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:48.786499 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d175486-6247-4c1f-926d-a4f0346fb7ea-kube-api-access-t64ll" (OuterVolumeSpecName: "kube-api-access-t64ll") pod "6d175486-6247-4c1f-926d-a4f0346fb7ea" (UID: "6d175486-6247-4c1f-926d-a4f0346fb7ea"). InnerVolumeSpecName "kube-api-access-t64ll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:21:48.786580 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:48.786545 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d175486-6247-4c1f-926d-a4f0346fb7ea-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6d175486-6247-4c1f-926d-a4f0346fb7ea" (UID: "6d175486-6247-4c1f-926d-a4f0346fb7ea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 04:21:48.886153 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:48.886062 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6d175486-6247-4c1f-926d-a4f0346fb7ea-maas-api-tls\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:21:48.886153 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:48.886096 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t64ll\" (UniqueName: \"kubernetes.io/projected/6d175486-6247-4c1f-926d-a4f0346fb7ea-kube-api-access-t64ll\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:21:48.886153 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:48.886114 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d175486-6247-4c1f-926d-a4f0346fb7ea-openshift-service-ca\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:21:48.886153 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:48.886128 2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6d175486-6247-4c1f-926d-a4f0346fb7ea-metrics-certs\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:21:49.101772 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:49.101674 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:21:49.151329 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:49.151299 2568 generic.go:356] "Generic (PLEG): container finished" podID="6d175486-6247-4c1f-926d-a4f0346fb7ea" containerID="59d75a1f6266fda592af1eb1fa7880fb827d987eef6d8fe129d5c9aeb6560504" exitCode=0 Sep 11 04:21:49.151510 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:49.151357 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-84b423-7dfc5f575-lw628" Sep 11 04:21:49.151510 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:49.151394 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-84b423-7dfc5f575-lw628" event={"ID":"6d175486-6247-4c1f-926d-a4f0346fb7ea","Type":"ContainerDied","Data":"59d75a1f6266fda592af1eb1fa7880fb827d987eef6d8fe129d5c9aeb6560504"} Sep 11 04:21:49.151510 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:49.151454 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-84b423-7dfc5f575-lw628" event={"ID":"6d175486-6247-4c1f-926d-a4f0346fb7ea","Type":"ContainerDied","Data":"0f40ef9a5e655c3d3e5c55eff6b06f49a3fba158c03d8b7346c1c7435603c937"} Sep 11 04:21:49.151510 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:49.151481 2568 scope.go:117] "RemoveContainer" containerID="59d75a1f6266fda592af1eb1fa7880fb827d987eef6d8fe129d5c9aeb6560504" Sep 11 04:21:49.161470 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:49.161455 2568 scope.go:117] "RemoveContainer" containerID="59d75a1f6266fda592af1eb1fa7880fb827d987eef6d8fe129d5c9aeb6560504" Sep 11 04:21:49.161745 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:21:49.161724 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59d75a1f6266fda592af1eb1fa7880fb827d987eef6d8fe129d5c9aeb6560504\": container with ID starting with 59d75a1f6266fda592af1eb1fa7880fb827d987eef6d8fe129d5c9aeb6560504 not found: ID does not exist" containerID="59d75a1f6266fda592af1eb1fa7880fb827d987eef6d8fe129d5c9aeb6560504" Sep 11 04:21:49.161819 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:49.161752 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59d75a1f6266fda592af1eb1fa7880fb827d987eef6d8fe129d5c9aeb6560504"} err="failed to get container status \"59d75a1f6266fda592af1eb1fa7880fb827d987eef6d8fe129d5c9aeb6560504\": rpc error: code = NotFound desc = could not find container \"59d75a1f6266fda592af1eb1fa7880fb827d987eef6d8fe129d5c9aeb6560504\": container with ID starting with 59d75a1f6266fda592af1eb1fa7880fb827d987eef6d8fe129d5c9aeb6560504 not found: ID does not exist" Sep 11 04:21:49.175366 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:49.175343 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-84b423-7dfc5f575-lw628"] Sep 11 04:21:49.182201 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:49.182182 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-84b423-7dfc5f575-lw628"] Sep 11 04:21:49.768055 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:49.768014 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d175486-6247-4c1f-926d-a4f0346fb7ea" path="/var/lib/kubelet/pods/6d175486-6247-4c1f-926d-a4f0346fb7ea/volumes" Sep 11 04:21:55.129628 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:55.129589 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:21:55.230064 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:55.230017 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:21:55.403972 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:55.403937 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:21:59.433708 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:21:59.433672 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:22:06.653756 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:22:06.653716 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:22:06.754031 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:22:06.753985 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:22:14.742235 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:22:14.742196 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fdc690-65b64cbcc7-sd4zv"] Sep 11 04:22:14.742645 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:22:14.742468 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fdc690-65b64cbcc7-sd4zv" podUID="9c70a619-7132-4acf-a6b2-d8a9bb7a21b7" containerName="maas-api" containerID="cri-o://335ee90e9fec8c9ff0c915c3edf777e1d0d5b11358a8a75ba8aa27a1df47795e" gracePeriod=30 Sep 11 04:22:14.779588 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:22:14.779561 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:22:14.880422 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:22:14.880366 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:22:14.979860 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:22:14.979827 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:22:14.990290 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:22:14.990263 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fdc690-65b64cbcc7-sd4zv" Sep 11 04:22:15.105751 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:22:15.105662 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:22:15.124728 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:22:15.124694 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c70a619-7132-4acf-a6b2-d8a9bb7a21b7-maas-api-tls\") pod \"9c70a619-7132-4acf-a6b2-d8a9bb7a21b7\" (UID: \"9c70a619-7132-4acf-a6b2-d8a9bb7a21b7\") " Sep 11 04:22:15.124867 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:22:15.124754 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c70a619-7132-4acf-a6b2-d8a9bb7a21b7-openshift-service-ca\") pod \"9c70a619-7132-4acf-a6b2-d8a9bb7a21b7\" (UID: \"9c70a619-7132-4acf-a6b2-d8a9bb7a21b7\") " Sep 11 04:22:15.124867 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:22:15.124780 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlh5d\" (UniqueName: \"kubernetes.io/projected/9c70a619-7132-4acf-a6b2-d8a9bb7a21b7-kube-api-access-dlh5d\") pod \"9c70a619-7132-4acf-a6b2-d8a9bb7a21b7\" (UID: \"9c70a619-7132-4acf-a6b2-d8a9bb7a21b7\") " Sep 11 04:22:15.124867 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:22:15.124799 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c70a619-7132-4acf-a6b2-d8a9bb7a21b7-metrics-certs\") pod \"9c70a619-7132-4acf-a6b2-d8a9bb7a21b7\" (UID: \"9c70a619-7132-4acf-a6b2-d8a9bb7a21b7\") " Sep 11 04:22:15.125202 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:22:15.125173 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c70a619-7132-4acf-a6b2-d8a9bb7a21b7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9c70a619-7132-4acf-a6b2-d8a9bb7a21b7" (UID: "9c70a619-7132-4acf-a6b2-d8a9bb7a21b7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 04:22:15.127189 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:22:15.127157 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c70a619-7132-4acf-a6b2-d8a9bb7a21b7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9c70a619-7132-4acf-a6b2-d8a9bb7a21b7" (UID: "9c70a619-7132-4acf-a6b2-d8a9bb7a21b7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 04:22:15.127189 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:22:15.127176 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c70a619-7132-4acf-a6b2-d8a9bb7a21b7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9c70a619-7132-4acf-a6b2-d8a9bb7a21b7" (UID: "9c70a619-7132-4acf-a6b2-d8a9bb7a21b7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 04:22:15.127349 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:22:15.127193 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c70a619-7132-4acf-a6b2-d8a9bb7a21b7-kube-api-access-dlh5d" (OuterVolumeSpecName: "kube-api-access-dlh5d") pod "9c70a619-7132-4acf-a6b2-d8a9bb7a21b7" (UID: "9c70a619-7132-4acf-a6b2-d8a9bb7a21b7"). InnerVolumeSpecName "kube-api-access-dlh5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:22:15.225515 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:22:15.225485 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c70a619-7132-4acf-a6b2-d8a9bb7a21b7-maas-api-tls\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:22:15.225515 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:22:15.225513 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c70a619-7132-4acf-a6b2-d8a9bb7a21b7-openshift-service-ca\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:22:15.225515 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:22:15.225524 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlh5d\" (UniqueName: \"kubernetes.io/projected/9c70a619-7132-4acf-a6b2-d8a9bb7a21b7-kube-api-access-dlh5d\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:22:15.225762 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:22:15.225533 2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c70a619-7132-4acf-a6b2-d8a9bb7a21b7-metrics-certs\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:22:15.279077 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:22:15.279012 2568 generic.go:356] "Generic (PLEG): container finished" podID="9c70a619-7132-4acf-a6b2-d8a9bb7a21b7" containerID="335ee90e9fec8c9ff0c915c3edf777e1d0d5b11358a8a75ba8aa27a1df47795e" exitCode=0 Sep 11 04:22:15.279238 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:22:15.279092 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fdc690-65b64cbcc7-sd4zv" Sep 11 04:22:15.279238 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:22:15.279103 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fdc690-65b64cbcc7-sd4zv" event={"ID":"9c70a619-7132-4acf-a6b2-d8a9bb7a21b7","Type":"ContainerDied","Data":"335ee90e9fec8c9ff0c915c3edf777e1d0d5b11358a8a75ba8aa27a1df47795e"} Sep 11 04:22:15.279238 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:22:15.279193 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-fdc690-65b64cbcc7-sd4zv" event={"ID":"9c70a619-7132-4acf-a6b2-d8a9bb7a21b7","Type":"ContainerDied","Data":"d02d5fb5f7d401e98872b384c2852fa921269187595a7507c92836140abf1fb8"} Sep 11 04:22:15.279238 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:22:15.279219 2568 scope.go:117] "RemoveContainer" containerID="335ee90e9fec8c9ff0c915c3edf777e1d0d5b11358a8a75ba8aa27a1df47795e" Sep 11 04:22:15.288365 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:22:15.288348 2568 scope.go:117] "RemoveContainer" containerID="335ee90e9fec8c9ff0c915c3edf777e1d0d5b11358a8a75ba8aa27a1df47795e" Sep 11 04:22:15.288698 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:22:15.288675 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"335ee90e9fec8c9ff0c915c3edf777e1d0d5b11358a8a75ba8aa27a1df47795e\": container with ID starting with 335ee90e9fec8c9ff0c915c3edf777e1d0d5b11358a8a75ba8aa27a1df47795e not found: ID does not exist" containerID="335ee90e9fec8c9ff0c915c3edf777e1d0d5b11358a8a75ba8aa27a1df47795e" Sep 11 04:22:15.288801 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:22:15.288704 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"335ee90e9fec8c9ff0c915c3edf777e1d0d5b11358a8a75ba8aa27a1df47795e"} err="failed to get container status \"335ee90e9fec8c9ff0c915c3edf777e1d0d5b11358a8a75ba8aa27a1df47795e\": rpc error: code = NotFound desc = could not find container \"335ee90e9fec8c9ff0c915c3edf777e1d0d5b11358a8a75ba8aa27a1df47795e\": container with ID starting with 335ee90e9fec8c9ff0c915c3edf777e1d0d5b11358a8a75ba8aa27a1df47795e not found: ID does not exist" Sep 11 04:22:15.307681 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:22:15.307655 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fdc690-65b64cbcc7-sd4zv"] Sep 11 04:22:15.311305 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:22:15.311283 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-fdc690-65b64cbcc7-sd4zv"] Sep 11 04:22:15.463127 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:22:15.463093 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:22:15.769105 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:22:15.769025 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c70a619-7132-4acf-a6b2-d8a9bb7a21b7" path="/var/lib/kubelet/pods/9c70a619-7132-4acf-a6b2-d8a9bb7a21b7/volumes" Sep 11 04:22:20.486566 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:22:20.486522 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:22:20.589827 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:22:20.589791 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:22:47.044366 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:22:47.044327 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:22:56.570075 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:22:56.570011 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:22:57.708122 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:22:57.708080 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:23:03.364894 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:03.364858 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:23:03.465673 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:03.465641 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:23:03.653680 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:03.653644 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:23:04.287726 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:04.287633 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:23:05.849585 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:05.849550 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-5tpdc_9112557d-d0ef-44c7-8b2a-f6ecd0a980d6/console-operator/1.log" Sep 11 04:23:05.856672 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:05.856646 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-5tpdc_9112557d-d0ef-44c7-8b2a-f6ecd0a980d6/console-operator/1.log" Sep 11 04:23:14.561036 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:14.560997 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a5a85-7b88d78d8f-tvh8q"] Sep 11 04:23:14.561607 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:14.561271 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a5a85-7b88d78d8f-tvh8q" podUID="a4dfd252-ab7f-47da-8ad4-d4c89165ba6c" containerName="maas-api" containerID="cri-o://bdd10d942a29182fc0d60b9ebaeb1f4616a9b168fbf9ea3416c5a327142b5275" gracePeriod=30 Sep 11 04:23:14.589794 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:14.589759 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:23:14.614865 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:14.614832 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7a5a85-86d55bdcdf-c5bfz"] Sep 11 04:23:14.615093 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:14.615061 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-7a5a85-86d55bdcdf-c5bfz" podUID="8e7e8a64-d384-426f-83c1-d6e33595c7e5" containerName="payload-processing" containerID="cri-o://b0fde5fb63ceab0cab4404d152b83cec0b89f6d90b9806df23438f2070455769" gracePeriod=30 Sep 11 04:23:14.690889 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:14.690857 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:23:14.785793 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:14.785758 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:23:14.876673 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:14.876637 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a5a85-7b88d78d8f-tvh8q" Sep 11 04:23:14.902010 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:14.901716 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7a5a85-86d55bdcdf-c5bfz" Sep 11 04:23:14.938847 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:14.938810 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4dfd252-ab7f-47da-8ad4-d4c89165ba6c-openshift-service-ca\") pod \"a4dfd252-ab7f-47da-8ad4-d4c89165ba6c\" (UID: \"a4dfd252-ab7f-47da-8ad4-d4c89165ba6c\") " Sep 11 04:23:14.939041 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:14.938891 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4dfd252-ab7f-47da-8ad4-d4c89165ba6c-metrics-certs\") pod \"a4dfd252-ab7f-47da-8ad4-d4c89165ba6c\" (UID: \"a4dfd252-ab7f-47da-8ad4-d4c89165ba6c\") " Sep 11 04:23:14.939041 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:14.938920 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e7e8a64-d384-426f-83c1-d6e33595c7e5-plugins-config-volume\") pod \"8e7e8a64-d384-426f-83c1-d6e33595c7e5\" (UID: \"8e7e8a64-d384-426f-83c1-d6e33595c7e5\") " Sep 11 04:23:14.939041 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:14.938963 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4dfd252-ab7f-47da-8ad4-d4c89165ba6c-maas-api-tls\") pod \"a4dfd252-ab7f-47da-8ad4-d4c89165ba6c\" (UID: \"a4dfd252-ab7f-47da-8ad4-d4c89165ba6c\") " Sep 11 04:23:14.939311 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:14.939276 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4dfd252-ab7f-47da-8ad4-d4c89165ba6c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a4dfd252-ab7f-47da-8ad4-d4c89165ba6c" (UID: "a4dfd252-ab7f-47da-8ad4-d4c89165ba6c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 04:23:14.939460 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:14.939431 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjhmm\" (UniqueName: \"kubernetes.io/projected/a4dfd252-ab7f-47da-8ad4-d4c89165ba6c-kube-api-access-zjhmm\") pod \"a4dfd252-ab7f-47da-8ad4-d4c89165ba6c\" (UID: \"a4dfd252-ab7f-47da-8ad4-d4c89165ba6c\") " Sep 11 04:23:14.940019 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:14.939560 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rllw\" (UniqueName: \"kubernetes.io/projected/8e7e8a64-d384-426f-83c1-d6e33595c7e5-kube-api-access-2rllw\") pod \"8e7e8a64-d384-426f-83c1-d6e33595c7e5\" (UID: \"8e7e8a64-d384-426f-83c1-d6e33595c7e5\") " Sep 11 04:23:14.940019 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:14.939680 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e7e8a64-d384-426f-83c1-d6e33595c7e5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8e7e8a64-d384-426f-83c1-d6e33595c7e5" (UID: "8e7e8a64-d384-426f-83c1-d6e33595c7e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 04:23:14.940019 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:14.939867 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e7e8a64-d384-426f-83c1-d6e33595c7e5-plugins-config-volume\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:23:14.940019 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:14.939885 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4dfd252-ab7f-47da-8ad4-d4c89165ba6c-openshift-service-ca\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:23:14.941705 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:14.941671 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4dfd252-ab7f-47da-8ad4-d4c89165ba6c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a4dfd252-ab7f-47da-8ad4-d4c89165ba6c" (UID: "a4dfd252-ab7f-47da-8ad4-d4c89165ba6c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 04:23:14.942010 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:14.941986 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4dfd252-ab7f-47da-8ad4-d4c89165ba6c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a4dfd252-ab7f-47da-8ad4-d4c89165ba6c" (UID: "a4dfd252-ab7f-47da-8ad4-d4c89165ba6c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 04:23:14.942648 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:14.942615 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4dfd252-ab7f-47da-8ad4-d4c89165ba6c-kube-api-access-zjhmm" (OuterVolumeSpecName: "kube-api-access-zjhmm") pod "a4dfd252-ab7f-47da-8ad4-d4c89165ba6c" (UID: "a4dfd252-ab7f-47da-8ad4-d4c89165ba6c"). InnerVolumeSpecName "kube-api-access-zjhmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:23:14.942751 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:14.942649 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e7e8a64-d384-426f-83c1-d6e33595c7e5-kube-api-access-2rllw" (OuterVolumeSpecName: "kube-api-access-2rllw") pod "8e7e8a64-d384-426f-83c1-d6e33595c7e5" (UID: "8e7e8a64-d384-426f-83c1-d6e33595c7e5"). InnerVolumeSpecName "kube-api-access-2rllw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:23:15.041151 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:15.041113 2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4dfd252-ab7f-47da-8ad4-d4c89165ba6c-metrics-certs\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:23:15.041151 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:15.041148 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4dfd252-ab7f-47da-8ad4-d4c89165ba6c-maas-api-tls\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:23:15.041395 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:15.041163 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjhmm\" (UniqueName: \"kubernetes.io/projected/a4dfd252-ab7f-47da-8ad4-d4c89165ba6c-kube-api-access-zjhmm\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:23:15.041395 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:15.041182 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rllw\" (UniqueName: \"kubernetes.io/projected/8e7e8a64-d384-426f-83c1-d6e33595c7e5-kube-api-access-2rllw\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:23:15.375840 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:15.375791 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:23:15.475993 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:15.475950 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:23:15.575362 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:15.575320 2568 generic.go:356] "Generic (PLEG): container finished" podID="a4dfd252-ab7f-47da-8ad4-d4c89165ba6c" containerID="bdd10d942a29182fc0d60b9ebaeb1f4616a9b168fbf9ea3416c5a327142b5275" exitCode=0 Sep 11 04:23:15.575845 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:15.575365 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a5a85-7b88d78d8f-tvh8q" Sep 11 04:23:15.575845 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:15.575383 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a5a85-7b88d78d8f-tvh8q" event={"ID":"a4dfd252-ab7f-47da-8ad4-d4c89165ba6c","Type":"ContainerDied","Data":"bdd10d942a29182fc0d60b9ebaeb1f4616a9b168fbf9ea3416c5a327142b5275"} Sep 11 04:23:15.575845 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:15.575447 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a5a85-7b88d78d8f-tvh8q" event={"ID":"a4dfd252-ab7f-47da-8ad4-d4c89165ba6c","Type":"ContainerDied","Data":"f9da6df41d36fa84678eb298fa9264f03f8f63c820c18aee62021dbfdd959174"} Sep 11 04:23:15.575845 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:15.575476 2568 scope.go:117] "RemoveContainer" containerID="bdd10d942a29182fc0d60b9ebaeb1f4616a9b168fbf9ea3416c5a327142b5275" Sep 11 04:23:15.577075 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:15.576874 2568 generic.go:356] "Generic (PLEG): container finished" podID="8e7e8a64-d384-426f-83c1-d6e33595c7e5" containerID="b0fde5fb63ceab0cab4404d152b83cec0b89f6d90b9806df23438f2070455769" exitCode=0 Sep 11 04:23:15.577075 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:15.577000 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7a5a85-86d55bdcdf-c5bfz" event={"ID":"8e7e8a64-d384-426f-83c1-d6e33595c7e5","Type":"ContainerDied","Data":"b0fde5fb63ceab0cab4404d152b83cec0b89f6d90b9806df23438f2070455769"} Sep 11 04:23:15.577224 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:15.577161 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7a5a85-86d55bdcdf-c5bfz" event={"ID":"8e7e8a64-d384-426f-83c1-d6e33595c7e5","Type":"ContainerDied","Data":"841e255ecbe468fd12bf9dea246ba7235b38370283583199d3d176d0ec11c5a9"} Sep 11 04:23:15.577224 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:15.577204 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7a5a85-86d55bdcdf-c5bfz" Sep 11 04:23:15.586100 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:15.586076 2568 scope.go:117] "RemoveContainer" containerID="bdd10d942a29182fc0d60b9ebaeb1f4616a9b168fbf9ea3416c5a327142b5275" Sep 11 04:23:15.586470 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:23:15.586448 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdd10d942a29182fc0d60b9ebaeb1f4616a9b168fbf9ea3416c5a327142b5275\": container with ID starting with bdd10d942a29182fc0d60b9ebaeb1f4616a9b168fbf9ea3416c5a327142b5275 not found: ID does not exist" containerID="bdd10d942a29182fc0d60b9ebaeb1f4616a9b168fbf9ea3416c5a327142b5275" Sep 11 04:23:15.586534 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:15.586479 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdd10d942a29182fc0d60b9ebaeb1f4616a9b168fbf9ea3416c5a327142b5275"} err="failed to get container status \"bdd10d942a29182fc0d60b9ebaeb1f4616a9b168fbf9ea3416c5a327142b5275\": rpc error: code = NotFound desc = could not find container \"bdd10d942a29182fc0d60b9ebaeb1f4616a9b168fbf9ea3416c5a327142b5275\": container with ID starting with bdd10d942a29182fc0d60b9ebaeb1f4616a9b168fbf9ea3416c5a327142b5275 not found: ID does not exist" Sep 11 04:23:15.586534 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:15.586499 2568 scope.go:117] "RemoveContainer" containerID="b0fde5fb63ceab0cab4404d152b83cec0b89f6d90b9806df23438f2070455769" Sep 11 04:23:15.594501 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:15.594485 2568 scope.go:117] "RemoveContainer" containerID="b0fde5fb63ceab0cab4404d152b83cec0b89f6d90b9806df23438f2070455769" Sep 11 04:23:15.594761 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:23:15.594743 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0fde5fb63ceab0cab4404d152b83cec0b89f6d90b9806df23438f2070455769\": container with ID starting with b0fde5fb63ceab0cab4404d152b83cec0b89f6d90b9806df23438f2070455769 not found: ID does not exist" containerID="b0fde5fb63ceab0cab4404d152b83cec0b89f6d90b9806df23438f2070455769" Sep 11 04:23:15.594799 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:15.594768 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0fde5fb63ceab0cab4404d152b83cec0b89f6d90b9806df23438f2070455769"} err="failed to get container status \"b0fde5fb63ceab0cab4404d152b83cec0b89f6d90b9806df23438f2070455769\": rpc error: code = NotFound desc = could not find container \"b0fde5fb63ceab0cab4404d152b83cec0b89f6d90b9806df23438f2070455769\": container with ID starting with b0fde5fb63ceab0cab4404d152b83cec0b89f6d90b9806df23438f2070455769 not found: ID does not exist" Sep 11 04:23:15.598276 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:15.598251 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a5a85-7b88d78d8f-tvh8q"] Sep 11 04:23:15.601782 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:15.601758 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-7a5a85-7b88d78d8f-tvh8q"] Sep 11 04:23:15.611311 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:15.611287 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7a5a85-86d55bdcdf-c5bfz"] Sep 11 04:23:15.617362 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:15.617338 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7a5a85-86d55bdcdf-c5bfz"] Sep 11 04:23:15.768402 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:15.768371 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e7e8a64-d384-426f-83c1-d6e33595c7e5" path="/var/lib/kubelet/pods/8e7e8a64-d384-426f-83c1-d6e33595c7e5/volumes" Sep 11 04:23:15.768769 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:15.768756 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4dfd252-ab7f-47da-8ad4-d4c89165ba6c" path="/var/lib/kubelet/pods/a4dfd252-ab7f-47da-8ad4-d4c89165ba6c/volumes" Sep 11 04:23:20.929244 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:20.929204 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:23:21.029552 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:21.029502 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:23:21.231777 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:21.231688 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:23:21.687544 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:21.687501 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:23:32.024068 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:32.020094 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cc6708-56f88d9f5c-mjgd9"] Sep 11 04:23:32.024068 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:32.021196 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cc6708-56f88d9f5c-mjgd9" podUID="957ae4ff-94c3-4f2c-b224-852d4e742bfc" containerName="maas-api" containerID="cri-o://44ec23d777b901313243b30e472dc3a8fd9ccc168b2e955900b6fd5844f758b6" gracePeriod=30 Sep 11 04:23:32.048938 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:32.048905 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:23:32.082271 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:32.082180 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cc6708-7679d85c98-nz6rx"] Sep 11 04:23:32.082486 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:32.082420 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cc6708-7679d85c98-nz6rx" podUID="f09610e1-0ccd-4a16-8c9b-b670eaecb1e4" containerName="payload-pre-processing" containerID="cri-o://cc81f2115b68cf065a4dad0b8d38ec99a0c53287d5c3f2a58f5b1410b9813d51" gracePeriod=30 Sep 11 04:23:32.149205 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:32.149165 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:23:32.253655 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:32.253555 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:23:32.326031 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:32.326004 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cc6708-56f88d9f5c-mjgd9" Sep 11 04:23:32.366932 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:32.366904 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cc6708-7679d85c98-nz6rx" Sep 11 04:23:32.385520 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:32.385491 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/957ae4ff-94c3-4f2c-b224-852d4e742bfc-openshift-service-ca\") pod \"957ae4ff-94c3-4f2c-b224-852d4e742bfc\" (UID: \"957ae4ff-94c3-4f2c-b224-852d4e742bfc\") " Sep 11 04:23:32.385702 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:32.385546 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/957ae4ff-94c3-4f2c-b224-852d4e742bfc-metrics-certs\") pod \"957ae4ff-94c3-4f2c-b224-852d4e742bfc\" (UID: \"957ae4ff-94c3-4f2c-b224-852d4e742bfc\") " Sep 11 04:23:32.385702 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:32.385589 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbbjf\" (UniqueName: \"kubernetes.io/projected/957ae4ff-94c3-4f2c-b224-852d4e742bfc-kube-api-access-lbbjf\") pod \"957ae4ff-94c3-4f2c-b224-852d4e742bfc\" (UID: \"957ae4ff-94c3-4f2c-b224-852d4e742bfc\") " Sep 11 04:23:32.385702 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:32.385676 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/957ae4ff-94c3-4f2c-b224-852d4e742bfc-maas-api-tls\") pod \"957ae4ff-94c3-4f2c-b224-852d4e742bfc\" (UID: \"957ae4ff-94c3-4f2c-b224-852d4e742bfc\") " Sep 11 04:23:32.385942 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:32.385897 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/957ae4ff-94c3-4f2c-b224-852d4e742bfc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "957ae4ff-94c3-4f2c-b224-852d4e742bfc" (UID: "957ae4ff-94c3-4f2c-b224-852d4e742bfc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 04:23:32.387982 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:32.387958 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/957ae4ff-94c3-4f2c-b224-852d4e742bfc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "957ae4ff-94c3-4f2c-b224-852d4e742bfc" (UID: "957ae4ff-94c3-4f2c-b224-852d4e742bfc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 04:23:32.388379 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:32.388347 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/957ae4ff-94c3-4f2c-b224-852d4e742bfc-kube-api-access-lbbjf" (OuterVolumeSpecName: "kube-api-access-lbbjf") pod "957ae4ff-94c3-4f2c-b224-852d4e742bfc" (UID: "957ae4ff-94c3-4f2c-b224-852d4e742bfc"). InnerVolumeSpecName "kube-api-access-lbbjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:23:32.388477 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:32.388396 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/957ae4ff-94c3-4f2c-b224-852d4e742bfc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "957ae4ff-94c3-4f2c-b224-852d4e742bfc" (UID: "957ae4ff-94c3-4f2c-b224-852d4e742bfc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 04:23:32.487134 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:32.487095 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-42lfc\" (UniqueName: \"kubernetes.io/projected/f09610e1-0ccd-4a16-8c9b-b670eaecb1e4-kube-api-access-42lfc\") pod \"f09610e1-0ccd-4a16-8c9b-b670eaecb1e4\" (UID: \"f09610e1-0ccd-4a16-8c9b-b670eaecb1e4\") " Sep 11 04:23:32.487334 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:32.487223 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f09610e1-0ccd-4a16-8c9b-b670eaecb1e4-plugins-config-volume\") pod \"f09610e1-0ccd-4a16-8c9b-b670eaecb1e4\" (UID: \"f09610e1-0ccd-4a16-8c9b-b670eaecb1e4\") " Sep 11 04:23:32.487469 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:32.487454 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/957ae4ff-94c3-4f2c-b224-852d4e742bfc-maas-api-tls\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:23:32.487532 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:32.487474 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/957ae4ff-94c3-4f2c-b224-852d4e742bfc-openshift-service-ca\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:23:32.487532 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:32.487489 2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/957ae4ff-94c3-4f2c-b224-852d4e742bfc-metrics-certs\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:23:32.487532 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:32.487502 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbbjf\" (UniqueName: \"kubernetes.io/projected/957ae4ff-94c3-4f2c-b224-852d4e742bfc-kube-api-access-lbbjf\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:23:32.487672 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:32.487578 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f09610e1-0ccd-4a16-8c9b-b670eaecb1e4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f09610e1-0ccd-4a16-8c9b-b670eaecb1e4" (UID: "f09610e1-0ccd-4a16-8c9b-b670eaecb1e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 04:23:32.489368 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:32.489344 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f09610e1-0ccd-4a16-8c9b-b670eaecb1e4-kube-api-access-42lfc" (OuterVolumeSpecName: "kube-api-access-42lfc") pod "f09610e1-0ccd-4a16-8c9b-b670eaecb1e4" (UID: "f09610e1-0ccd-4a16-8c9b-b670eaecb1e4"). InnerVolumeSpecName "kube-api-access-42lfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:23:32.588578 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:32.588494 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42lfc\" (UniqueName: \"kubernetes.io/projected/f09610e1-0ccd-4a16-8c9b-b670eaecb1e4-kube-api-access-42lfc\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:23:32.588578 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:32.588523 2568 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f09610e1-0ccd-4a16-8c9b-b670eaecb1e4-plugins-config-volume\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:23:32.662357 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:32.662317 2568 generic.go:356] "Generic (PLEG): container finished" podID="957ae4ff-94c3-4f2c-b224-852d4e742bfc" containerID="44ec23d777b901313243b30e472dc3a8fd9ccc168b2e955900b6fd5844f758b6" exitCode=0 Sep 11 04:23:32.662554 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:32.662370 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cc6708-56f88d9f5c-mjgd9" Sep 11 04:23:32.662554 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:32.662400 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cc6708-56f88d9f5c-mjgd9" event={"ID":"957ae4ff-94c3-4f2c-b224-852d4e742bfc","Type":"ContainerDied","Data":"44ec23d777b901313243b30e472dc3a8fd9ccc168b2e955900b6fd5844f758b6"} Sep 11 04:23:32.662554 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:32.662452 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cc6708-56f88d9f5c-mjgd9" event={"ID":"957ae4ff-94c3-4f2c-b224-852d4e742bfc","Type":"ContainerDied","Data":"602e7f2d76de1510be1e1f66c16f57da3b24daf1181fc72c70f56fc8367c8ab7"} Sep 11 04:23:32.662554 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:32.662479 2568 scope.go:117] "RemoveContainer" containerID="44ec23d777b901313243b30e472dc3a8fd9ccc168b2e955900b6fd5844f758b6" Sep 11 04:23:32.663884 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:32.663861 2568 generic.go:356] "Generic (PLEG): container finished" podID="f09610e1-0ccd-4a16-8c9b-b670eaecb1e4" containerID="cc81f2115b68cf065a4dad0b8d38ec99a0c53287d5c3f2a58f5b1410b9813d51" exitCode=0 Sep 11 04:23:32.664005 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:32.663893 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cc6708-7679d85c98-nz6rx" event={"ID":"f09610e1-0ccd-4a16-8c9b-b670eaecb1e4","Type":"ContainerDied","Data":"cc81f2115b68cf065a4dad0b8d38ec99a0c53287d5c3f2a58f5b1410b9813d51"} Sep 11 04:23:32.664005 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:32.663903 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cc6708-7679d85c98-nz6rx" Sep 11 04:23:32.664005 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:32.663927 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cc6708-7679d85c98-nz6rx" event={"ID":"f09610e1-0ccd-4a16-8c9b-b670eaecb1e4","Type":"ContainerDied","Data":"11c5fe8a48643202afb8bf119320d39fdd52feaf3fef6ab3fad9b2834faf3f8b"} Sep 11 04:23:32.674818 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:32.674621 2568 scope.go:117] "RemoveContainer" containerID="44ec23d777b901313243b30e472dc3a8fd9ccc168b2e955900b6fd5844f758b6" Sep 11 04:23:32.675275 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:23:32.675243 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44ec23d777b901313243b30e472dc3a8fd9ccc168b2e955900b6fd5844f758b6\": container with ID starting with 44ec23d777b901313243b30e472dc3a8fd9ccc168b2e955900b6fd5844f758b6 not found: ID does not exist" containerID="44ec23d777b901313243b30e472dc3a8fd9ccc168b2e955900b6fd5844f758b6" Sep 11 04:23:32.675376 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:32.675290 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44ec23d777b901313243b30e472dc3a8fd9ccc168b2e955900b6fd5844f758b6"} err="failed to get container status \"44ec23d777b901313243b30e472dc3a8fd9ccc168b2e955900b6fd5844f758b6\": rpc error: code = NotFound desc = could not find container \"44ec23d777b901313243b30e472dc3a8fd9ccc168b2e955900b6fd5844f758b6\": container with ID starting with 44ec23d777b901313243b30e472dc3a8fd9ccc168b2e955900b6fd5844f758b6 not found: ID does not exist" Sep 11 04:23:32.675376 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:32.675314 2568 scope.go:117] "RemoveContainer" containerID="cc81f2115b68cf065a4dad0b8d38ec99a0c53287d5c3f2a58f5b1410b9813d51" Sep 11 04:23:32.682918 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:32.682896 2568 scope.go:117] "RemoveContainer" containerID="cc81f2115b68cf065a4dad0b8d38ec99a0c53287d5c3f2a58f5b1410b9813d51" Sep 11 04:23:32.683204 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:23:32.683185 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc81f2115b68cf065a4dad0b8d38ec99a0c53287d5c3f2a58f5b1410b9813d51\": container with ID starting with cc81f2115b68cf065a4dad0b8d38ec99a0c53287d5c3f2a58f5b1410b9813d51 not found: ID does not exist" containerID="cc81f2115b68cf065a4dad0b8d38ec99a0c53287d5c3f2a58f5b1410b9813d51" Sep 11 04:23:32.683282 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:32.683215 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc81f2115b68cf065a4dad0b8d38ec99a0c53287d5c3f2a58f5b1410b9813d51"} err="failed to get container status \"cc81f2115b68cf065a4dad0b8d38ec99a0c53287d5c3f2a58f5b1410b9813d51\": rpc error: code = NotFound desc = could not find container \"cc81f2115b68cf065a4dad0b8d38ec99a0c53287d5c3f2a58f5b1410b9813d51\": container with ID starting with cc81f2115b68cf065a4dad0b8d38ec99a0c53287d5c3f2a58f5b1410b9813d51 not found: ID does not exist" Sep 11 04:23:32.688133 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:32.688111 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cc6708-7679d85c98-nz6rx"] Sep 11 04:23:32.692656 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:32.692636 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-cc6708-7679d85c98-nz6rx"] Sep 11 04:23:32.705867 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:32.705837 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cc6708-56f88d9f5c-mjgd9"] Sep 11 04:23:32.709484 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:32.709463 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-cc6708-56f88d9f5c-mjgd9"] Sep 11 04:23:32.734891 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:32.734860 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:23:32.840378 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:32.840281 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:23:33.768041 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:33.768010 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="957ae4ff-94c3-4f2c-b224-852d4e742bfc" path="/var/lib/kubelet/pods/957ae4ff-94c3-4f2c-b224-852d4e742bfc/volumes" Sep 11 04:23:33.768417 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:33.768409 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f09610e1-0ccd-4a16-8c9b-b670eaecb1e4" path="/var/lib/kubelet/pods/f09610e1-0ccd-4a16-8c9b-b670eaecb1e4/volumes" Sep 11 04:23:38.489784 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:38.489737 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:23:38.563069 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:38.562998 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:23:38.741536 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:38.741424 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w"] Sep 11 04:23:38.741740 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:38.741705 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" podUID="8795982f-8f14-4d43-948c-f1966fc1be21" containerName="istio-proxy" containerID="cri-o://550342e0e85a719f2bd75ccb3fa0bf4c7ebde4f1862d93dee1a843bcb131def9" gracePeriod=30 Sep 11 04:23:38.764969 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:38.764941 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:23:39.112759 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:39.112664 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9342ed-plvz5"] Sep 11 04:23:39.113126 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:39.113107 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="957ae4ff-94c3-4f2c-b224-852d4e742bfc" containerName="maas-api" Sep 11 04:23:39.113233 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:39.113127 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="957ae4ff-94c3-4f2c-b224-852d4e742bfc" containerName="maas-api" Sep 11 04:23:39.113233 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:39.113152 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f09610e1-0ccd-4a16-8c9b-b670eaecb1e4" containerName="payload-pre-processing" Sep 11 04:23:39.113233 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:39.113160 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="f09610e1-0ccd-4a16-8c9b-b670eaecb1e4" containerName="payload-pre-processing" Sep 11 04:23:39.113233 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:39.113183 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c70a619-7132-4acf-a6b2-d8a9bb7a21b7" containerName="maas-api" Sep 11 04:23:39.113233 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:39.113192 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c70a619-7132-4acf-a6b2-d8a9bb7a21b7" containerName="maas-api" Sep 11 04:23:39.113233 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:39.113207 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4dfd252-ab7f-47da-8ad4-d4c89165ba6c" containerName="maas-api" Sep 11 04:23:39.113233 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:39.113216 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4dfd252-ab7f-47da-8ad4-d4c89165ba6c" containerName="maas-api" Sep 11 04:23:39.113233 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:39.113226 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e7e8a64-d384-426f-83c1-d6e33595c7e5" containerName="payload-processing" Sep 11 04:23:39.113233 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:39.113235 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e7e8a64-d384-426f-83c1-d6e33595c7e5" containerName="payload-processing" Sep 11 04:23:39.113677 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:39.113245 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d175486-6247-4c1f-926d-a4f0346fb7ea" containerName="maas-api" Sep 11 04:23:39.113677 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:39.113252 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d175486-6247-4c1f-926d-a4f0346fb7ea" containerName="maas-api" Sep 11 04:23:39.113677 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:39.113335 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d175486-6247-4c1f-926d-a4f0346fb7ea" containerName="maas-api" Sep 11 04:23:39.113677 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:39.113349 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e7e8a64-d384-426f-83c1-d6e33595c7e5" containerName="payload-processing" Sep 11 04:23:39.113677 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:39.113358 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="957ae4ff-94c3-4f2c-b224-852d4e742bfc" containerName="maas-api" Sep 11 04:23:39.113677 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:39.113368 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="f09610e1-0ccd-4a16-8c9b-b670eaecb1e4" containerName="payload-pre-processing" Sep 11 04:23:39.113677 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:39.113378 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c70a619-7132-4acf-a6b2-d8a9bb7a21b7" containerName="maas-api" Sep 11 04:23:39.113677 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:39.113386 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4dfd252-ab7f-47da-8ad4-d4c89165ba6c" containerName="maas-api" Sep 11 04:23:39.116413 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:39.116393 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9342ed-plvz5" Sep 11 04:23:39.119087 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:39.119067 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 04:23:39.119206 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:39.119143 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6brsr\"" Sep 11 04:23:39.124041 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:39.124015 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9342ed-plvz5"] Sep 11 04:23:39.250155 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:39.250118 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38802f54-40d4-4274-99d5-416a34ffcfd0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-9342ed-plvz5\" (UID: \"38802f54-40d4-4274-99d5-416a34ffcfd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9342ed-plvz5" Sep 11 04:23:39.303832 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:39.303797 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:23:39.350815 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:39.350782 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38802f54-40d4-4274-99d5-416a34ffcfd0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-9342ed-plvz5\" (UID: \"38802f54-40d4-4274-99d5-416a34ffcfd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9342ed-plvz5" Sep 11 04:23:39.351402 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:39.351383 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38802f54-40d4-4274-99d5-416a34ffcfd0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-9342ed-plvz5\" (UID: \"38802f54-40d4-4274-99d5-416a34ffcfd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9342ed-plvz5" Sep 11 04:23:39.428129 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:39.428092 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9342ed-plvz5" Sep 11 04:23:39.703844 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:39.703767 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9342ed-plvz5"] Sep 11 04:23:39.707196 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:23:39.707168 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38802f54_40d4_4274_99d5_416a34ffcfd0.slice/crio-a122f05c201c7cc23cc6341805ef2980728d6dc8cc73f2dc2216a37ac678baf1 WatchSource:0}: Error finding container a122f05c201c7cc23cc6341805ef2980728d6dc8cc73f2dc2216a37ac678baf1: Status 404 returned error can't find the container with id a122f05c201c7cc23cc6341805ef2980728d6dc8cc73f2dc2216a37ac678baf1 Sep 11 04:23:40.705607 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:40.705577 2568 generic.go:356] "Generic (PLEG): container finished" podID="38802f54-40d4-4274-99d5-416a34ffcfd0" containerID="06feb1e9343b0a1889a2486d999c535175ecea63beb62ba2208007fbcecdf32d" exitCode=0 Sep 11 04:23:40.705972 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:40.705618 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9342ed-plvz5" event={"ID":"38802f54-40d4-4274-99d5-416a34ffcfd0","Type":"ContainerDied","Data":"06feb1e9343b0a1889a2486d999c535175ecea63beb62ba2208007fbcecdf32d"} Sep 11 04:23:40.705972 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:40.705676 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9342ed-plvz5" event={"ID":"38802f54-40d4-4274-99d5-416a34ffcfd0","Type":"ContainerStarted","Data":"a122f05c201c7cc23cc6341805ef2980728d6dc8cc73f2dc2216a37ac678baf1"} Sep 11 04:23:41.842209 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:41.842184 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9342ed-plvz5" Sep 11 04:23:41.974982 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:41.974887 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38802f54-40d4-4274-99d5-416a34ffcfd0-maas-api-service-ca\") pod \"38802f54-40d4-4274-99d5-416a34ffcfd0\" (UID: \"38802f54-40d4-4274-99d5-416a34ffcfd0\") " Sep 11 04:23:41.975283 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:41.975252 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38802f54-40d4-4274-99d5-416a34ffcfd0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "38802f54-40d4-4274-99d5-416a34ffcfd0" (UID: "38802f54-40d4-4274-99d5-416a34ffcfd0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 04:23:42.076547 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:42.076504 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38802f54-40d4-4274-99d5-416a34ffcfd0-maas-api-service-ca\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:23:42.716460 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:42.716431 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9342ed-plvz5" Sep 11 04:23:42.716631 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:42.716461 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9342ed-plvz5" event={"ID":"38802f54-40d4-4274-99d5-416a34ffcfd0","Type":"ContainerDied","Data":"a122f05c201c7cc23cc6341805ef2980728d6dc8cc73f2dc2216a37ac678baf1"} Sep 11 04:23:42.716631 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:42.716531 2568 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a122f05c201c7cc23cc6341805ef2980728d6dc8cc73f2dc2216a37ac678baf1" Sep 11 04:23:43.987969 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:43.987941 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" Sep 11 04:23:44.094536 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:44.094446 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8795982f-8f14-4d43-948c-f1966fc1be21-istio-token\") pod \"8795982f-8f14-4d43-948c-f1966fc1be21\" (UID: \"8795982f-8f14-4d43-948c-f1966fc1be21\") " Sep 11 04:23:44.094536 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:44.094488 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjnt5\" (UniqueName: \"kubernetes.io/projected/8795982f-8f14-4d43-948c-f1966fc1be21-kube-api-access-rjnt5\") pod \"8795982f-8f14-4d43-948c-f1966fc1be21\" (UID: \"8795982f-8f14-4d43-948c-f1966fc1be21\") " Sep 11 04:23:44.094536 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:44.094519 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8795982f-8f14-4d43-948c-f1966fc1be21-istiod-ca-cert\") pod \"8795982f-8f14-4d43-948c-f1966fc1be21\" (UID: \"8795982f-8f14-4d43-948c-f1966fc1be21\") " Sep 11 04:23:44.094835 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:44.094570 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8795982f-8f14-4d43-948c-f1966fc1be21-istio-envoy\") pod \"8795982f-8f14-4d43-948c-f1966fc1be21\" (UID: \"8795982f-8f14-4d43-948c-f1966fc1be21\") " Sep 11 04:23:44.094835 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:44.094590 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8795982f-8f14-4d43-948c-f1966fc1be21-credential-socket\") pod \"8795982f-8f14-4d43-948c-f1966fc1be21\" (UID: \"8795982f-8f14-4d43-948c-f1966fc1be21\") " Sep 11 04:23:44.094835 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:44.094609 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8795982f-8f14-4d43-948c-f1966fc1be21-workload-socket\") pod \"8795982f-8f14-4d43-948c-f1966fc1be21\" (UID: \"8795982f-8f14-4d43-948c-f1966fc1be21\") " Sep 11 04:23:44.094835 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:44.094630 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8795982f-8f14-4d43-948c-f1966fc1be21-istio-podinfo\") pod \"8795982f-8f14-4d43-948c-f1966fc1be21\" (UID: \"8795982f-8f14-4d43-948c-f1966fc1be21\") " Sep 11 04:23:44.094835 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:44.094663 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8795982f-8f14-4d43-948c-f1966fc1be21-workload-certs\") pod \"8795982f-8f14-4d43-948c-f1966fc1be21\" (UID: \"8795982f-8f14-4d43-948c-f1966fc1be21\") " Sep 11 04:23:44.094835 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:44.094731 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8795982f-8f14-4d43-948c-f1966fc1be21-istio-data\") pod \"8795982f-8f14-4d43-948c-f1966fc1be21\" (UID: \"8795982f-8f14-4d43-948c-f1966fc1be21\") " Sep 11 04:23:44.095171 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:44.094899 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8795982f-8f14-4d43-948c-f1966fc1be21-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8795982f-8f14-4d43-948c-f1966fc1be21" (UID: "8795982f-8f14-4d43-948c-f1966fc1be21"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:23:44.095171 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:44.094925 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8795982f-8f14-4d43-948c-f1966fc1be21-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8795982f-8f14-4d43-948c-f1966fc1be21" (UID: "8795982f-8f14-4d43-948c-f1966fc1be21"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:23:44.095171 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:44.095082 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8795982f-8f14-4d43-948c-f1966fc1be21-credential-socket\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:23:44.095171 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:44.095102 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8795982f-8f14-4d43-948c-f1966fc1be21-workload-socket\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:23:44.095361 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:44.095179 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8795982f-8f14-4d43-948c-f1966fc1be21-istio-data" (OuterVolumeSpecName: "istio-data") pod "8795982f-8f14-4d43-948c-f1966fc1be21" (UID: "8795982f-8f14-4d43-948c-f1966fc1be21"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:23:44.095361 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:44.095197 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8795982f-8f14-4d43-948c-f1966fc1be21-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8795982f-8f14-4d43-948c-f1966fc1be21" (UID: "8795982f-8f14-4d43-948c-f1966fc1be21"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:23:44.095361 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:44.095303 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8795982f-8f14-4d43-948c-f1966fc1be21-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8795982f-8f14-4d43-948c-f1966fc1be21" (UID: "8795982f-8f14-4d43-948c-f1966fc1be21"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 04:23:44.096845 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:44.096819 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8795982f-8f14-4d43-948c-f1966fc1be21-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8795982f-8f14-4d43-948c-f1966fc1be21" (UID: "8795982f-8f14-4d43-948c-f1966fc1be21"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:23:44.096962 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:44.096889 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8795982f-8f14-4d43-948c-f1966fc1be21-kube-api-access-rjnt5" (OuterVolumeSpecName: "kube-api-access-rjnt5") pod "8795982f-8f14-4d43-948c-f1966fc1be21" (UID: "8795982f-8f14-4d43-948c-f1966fc1be21"). InnerVolumeSpecName "kube-api-access-rjnt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:23:44.097038 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:44.097018 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8795982f-8f14-4d43-948c-f1966fc1be21-istio-token" (OuterVolumeSpecName: "istio-token") pod "8795982f-8f14-4d43-948c-f1966fc1be21" (UID: "8795982f-8f14-4d43-948c-f1966fc1be21"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:23:44.097157 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:44.097134 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8795982f-8f14-4d43-948c-f1966fc1be21-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8795982f-8f14-4d43-948c-f1966fc1be21" (UID: "8795982f-8f14-4d43-948c-f1966fc1be21"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 04:23:44.195706 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:44.195672 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8795982f-8f14-4d43-948c-f1966fc1be21-istio-token\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:23:44.195706 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:44.195694 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjnt5\" (UniqueName: \"kubernetes.io/projected/8795982f-8f14-4d43-948c-f1966fc1be21-kube-api-access-rjnt5\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:23:44.195706 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:44.195706 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8795982f-8f14-4d43-948c-f1966fc1be21-istiod-ca-cert\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:23:44.195706 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:44.195715 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8795982f-8f14-4d43-948c-f1966fc1be21-istio-envoy\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:23:44.195944 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:44.195723 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8795982f-8f14-4d43-948c-f1966fc1be21-istio-podinfo\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:23:44.195944 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:44.195731 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8795982f-8f14-4d43-948c-f1966fc1be21-workload-certs\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:23:44.195944 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:44.195738 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8795982f-8f14-4d43-948c-f1966fc1be21-istio-data\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:23:44.726582 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:44.726549 2568 generic.go:356] "Generic (PLEG): container finished" podID="8795982f-8f14-4d43-948c-f1966fc1be21" containerID="550342e0e85a719f2bd75ccb3fa0bf4c7ebde4f1862d93dee1a843bcb131def9" exitCode=0 Sep 11 04:23:44.726759 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:44.726602 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" Sep 11 04:23:44.726759 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:44.726636 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" event={"ID":"8795982f-8f14-4d43-948c-f1966fc1be21","Type":"ContainerDied","Data":"550342e0e85a719f2bd75ccb3fa0bf4c7ebde4f1862d93dee1a843bcb131def9"} Sep 11 04:23:44.726759 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:44.726702 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w" event={"ID":"8795982f-8f14-4d43-948c-f1966fc1be21","Type":"ContainerDied","Data":"f19c98cba04478039a824e1872105fe0bb58e8069b43ab28998cbb1d68624512"} Sep 11 04:23:44.726759 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:44.726733 2568 scope.go:117] "RemoveContainer" containerID="550342e0e85a719f2bd75ccb3fa0bf4c7ebde4f1862d93dee1a843bcb131def9" Sep 11 04:23:44.735457 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:44.735442 2568 scope.go:117] "RemoveContainer" containerID="bd3f2e2a4696ac80c356e3dd8bb01319919b012d3302dcb41c50719c8cc9742c" Sep 11 04:23:44.743015 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:44.743000 2568 scope.go:117] "RemoveContainer" containerID="550342e0e85a719f2bd75ccb3fa0bf4c7ebde4f1862d93dee1a843bcb131def9" Sep 11 04:23:44.743259 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:23:44.743241 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"550342e0e85a719f2bd75ccb3fa0bf4c7ebde4f1862d93dee1a843bcb131def9\": container with ID starting with 550342e0e85a719f2bd75ccb3fa0bf4c7ebde4f1862d93dee1a843bcb131def9 not found: ID does not exist" containerID="550342e0e85a719f2bd75ccb3fa0bf4c7ebde4f1862d93dee1a843bcb131def9" Sep 11 04:23:44.743311 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:44.743267 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"550342e0e85a719f2bd75ccb3fa0bf4c7ebde4f1862d93dee1a843bcb131def9"} err="failed to get container status \"550342e0e85a719f2bd75ccb3fa0bf4c7ebde4f1862d93dee1a843bcb131def9\": rpc error: code = NotFound desc = could not find container \"550342e0e85a719f2bd75ccb3fa0bf4c7ebde4f1862d93dee1a843bcb131def9\": container with ID starting with 550342e0e85a719f2bd75ccb3fa0bf4c7ebde4f1862d93dee1a843bcb131def9 not found: ID does not exist" Sep 11 04:23:44.743311 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:44.743282 2568 scope.go:117] "RemoveContainer" containerID="bd3f2e2a4696ac80c356e3dd8bb01319919b012d3302dcb41c50719c8cc9742c" Sep 11 04:23:44.743529 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:23:44.743510 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd3f2e2a4696ac80c356e3dd8bb01319919b012d3302dcb41c50719c8cc9742c\": container with ID starting with bd3f2e2a4696ac80c356e3dd8bb01319919b012d3302dcb41c50719c8cc9742c not found: ID does not exist" containerID="bd3f2e2a4696ac80c356e3dd8bb01319919b012d3302dcb41c50719c8cc9742c" Sep 11 04:23:44.743580 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:44.743539 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd3f2e2a4696ac80c356e3dd8bb01319919b012d3302dcb41c50719c8cc9742c"} err="failed to get container status \"bd3f2e2a4696ac80c356e3dd8bb01319919b012d3302dcb41c50719c8cc9742c\": rpc error: code = NotFound desc = could not find container \"bd3f2e2a4696ac80c356e3dd8bb01319919b012d3302dcb41c50719c8cc9742c\": container with ID starting with bd3f2e2a4696ac80c356e3dd8bb01319919b012d3302dcb41c50719c8cc9742c not found: ID does not exist" Sep 11 04:23:44.747792 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:44.747769 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w"] Sep 11 04:23:44.754150 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:44.754125 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cc6708-openshift-default-6779bb4d7d-t7s5w"] Sep 11 04:23:45.767586 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:45.767552 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8795982f-8f14-4d43-948c-f1966fc1be21" path="/var/lib/kubelet/pods/8795982f-8f14-4d43-948c-f1966fc1be21/volumes" Sep 11 04:23:49.611743 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:49.611703 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:23:49.683269 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:49.683236 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:23:49.783002 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:49.782966 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:23:49.802557 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:49.802526 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9342ed-plvz5"] Sep 11 04:23:49.805688 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:49.805665 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-9342ed-plvz5"] Sep 11 04:23:49.838880 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:49.838851 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:23:50.189915 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:50.189873 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:23:51.767542 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:51.767505 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38802f54-40d4-4274-99d5-416a34ffcfd0" path="/var/lib/kubelet/pods/38802f54-40d4-4274-99d5-416a34ffcfd0/volumes" Sep 11 04:23:56.494738 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:56.494700 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:23:56.631535 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:56.631495 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:23:56.719386 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:56.719354 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj"] Sep 11 04:23:56.719636 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:56.719610 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" podUID="6eee5be4-2a65-4513-8c1e-47db274ca367" containerName="istio-proxy" containerID="cri-o://df8c80348d8d9ee05a0f37a4dfcc7119d2d1e6e12ffd5caec7a11c32ef786302" gracePeriod=30 Sep 11 04:23:56.736176 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:56.736147 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:23:58.990128 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:58.990090 2568 patch_prober.go:28] interesting pod/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 04:23:58.990515 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:23:58.990178 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" podUID="6eee5be4-2a65-4513-8c1e-47db274ca367" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 04:24:01.973012 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:24:01.972986 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" Sep 11 04:24:02.145227 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:24:02.145134 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eee5be4-2a65-4513-8c1e-47db274ca367-workload-socket\") pod \"6eee5be4-2a65-4513-8c1e-47db274ca367\" (UID: \"6eee5be4-2a65-4513-8c1e-47db274ca367\") " Sep 11 04:24:02.145227 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:24:02.145175 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eee5be4-2a65-4513-8c1e-47db274ca367-workload-certs\") pod \"6eee5be4-2a65-4513-8c1e-47db274ca367\" (UID: \"6eee5be4-2a65-4513-8c1e-47db274ca367\") " Sep 11 04:24:02.145227 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:24:02.145208 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eee5be4-2a65-4513-8c1e-47db274ca367-istio-envoy\") pod \"6eee5be4-2a65-4513-8c1e-47db274ca367\" (UID: \"6eee5be4-2a65-4513-8c1e-47db274ca367\") " Sep 11 04:24:02.145737 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:24:02.145236 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kppln\" (UniqueName: \"kubernetes.io/projected/6eee5be4-2a65-4513-8c1e-47db274ca367-kube-api-access-kppln\") pod \"6eee5be4-2a65-4513-8c1e-47db274ca367\" (UID: \"6eee5be4-2a65-4513-8c1e-47db274ca367\") " Sep 11 04:24:02.145737 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:24:02.145271 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eee5be4-2a65-4513-8c1e-47db274ca367-istio-token\") pod \"6eee5be4-2a65-4513-8c1e-47db274ca367\" (UID: \"6eee5be4-2a65-4513-8c1e-47db274ca367\") " Sep 11 04:24:02.145737 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:24:02.145299 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eee5be4-2a65-4513-8c1e-47db274ca367-credential-socket\") pod \"6eee5be4-2a65-4513-8c1e-47db274ca367\" (UID: \"6eee5be4-2a65-4513-8c1e-47db274ca367\") " Sep 11 04:24:02.145737 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:24:02.145326 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eee5be4-2a65-4513-8c1e-47db274ca367-istio-podinfo\") pod \"6eee5be4-2a65-4513-8c1e-47db274ca367\" (UID: \"6eee5be4-2a65-4513-8c1e-47db274ca367\") " Sep 11 04:24:02.145737 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:24:02.145382 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eee5be4-2a65-4513-8c1e-47db274ca367-istiod-ca-cert\") pod \"6eee5be4-2a65-4513-8c1e-47db274ca367\" (UID: \"6eee5be4-2a65-4513-8c1e-47db274ca367\") " Sep 11 04:24:02.145737 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:24:02.145411 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eee5be4-2a65-4513-8c1e-47db274ca367-istio-data\") pod \"6eee5be4-2a65-4513-8c1e-47db274ca367\" (UID: \"6eee5be4-2a65-4513-8c1e-47db274ca367\") " Sep 11 04:24:02.145737 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:24:02.145478 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eee5be4-2a65-4513-8c1e-47db274ca367-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6eee5be4-2a65-4513-8c1e-47db274ca367" (UID: "6eee5be4-2a65-4513-8c1e-47db274ca367"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:24:02.145737 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:24:02.145506 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eee5be4-2a65-4513-8c1e-47db274ca367-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6eee5be4-2a65-4513-8c1e-47db274ca367" (UID: "6eee5be4-2a65-4513-8c1e-47db274ca367"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:24:02.145737 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:24:02.145662 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eee5be4-2a65-4513-8c1e-47db274ca367-istio-data" (OuterVolumeSpecName: "istio-data") pod "6eee5be4-2a65-4513-8c1e-47db274ca367" (UID: "6eee5be4-2a65-4513-8c1e-47db274ca367"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:24:02.145737 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:24:02.145713 2568 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eee5be4-2a65-4513-8c1e-47db274ca367-workload-socket\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:24:02.145737 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:24:02.145714 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eee5be4-2a65-4513-8c1e-47db274ca367-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6eee5be4-2a65-4513-8c1e-47db274ca367" (UID: "6eee5be4-2a65-4513-8c1e-47db274ca367"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:24:02.145737 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:24:02.145733 2568 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eee5be4-2a65-4513-8c1e-47db274ca367-workload-certs\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:24:02.146399 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:24:02.146074 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eee5be4-2a65-4513-8c1e-47db274ca367-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6eee5be4-2a65-4513-8c1e-47db274ca367" (UID: "6eee5be4-2a65-4513-8c1e-47db274ca367"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 04:24:02.147687 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:24:02.147658 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eee5be4-2a65-4513-8c1e-47db274ca367-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6eee5be4-2a65-4513-8c1e-47db274ca367" (UID: "6eee5be4-2a65-4513-8c1e-47db274ca367"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:24:02.147857 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:24:02.147829 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6eee5be4-2a65-4513-8c1e-47db274ca367-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6eee5be4-2a65-4513-8c1e-47db274ca367" (UID: "6eee5be4-2a65-4513-8c1e-47db274ca367"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 04:24:02.148069 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:24:02.148029 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eee5be4-2a65-4513-8c1e-47db274ca367-istio-token" (OuterVolumeSpecName: "istio-token") pod "6eee5be4-2a65-4513-8c1e-47db274ca367" (UID: "6eee5be4-2a65-4513-8c1e-47db274ca367"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:24:02.148205 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:24:02.148185 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eee5be4-2a65-4513-8c1e-47db274ca367-kube-api-access-kppln" (OuterVolumeSpecName: "kube-api-access-kppln") pod "6eee5be4-2a65-4513-8c1e-47db274ca367" (UID: "6eee5be4-2a65-4513-8c1e-47db274ca367"). InnerVolumeSpecName "kube-api-access-kppln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:24:02.246804 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:24:02.246764 2568 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eee5be4-2a65-4513-8c1e-47db274ca367-istiod-ca-cert\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:24:02.246804 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:24:02.246799 2568 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eee5be4-2a65-4513-8c1e-47db274ca367-istio-data\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:24:02.246804 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:24:02.246812 2568 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eee5be4-2a65-4513-8c1e-47db274ca367-istio-envoy\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:24:02.247092 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:24:02.246823 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kppln\" (UniqueName: \"kubernetes.io/projected/6eee5be4-2a65-4513-8c1e-47db274ca367-kube-api-access-kppln\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:24:02.247092 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:24:02.246836 2568 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eee5be4-2a65-4513-8c1e-47db274ca367-istio-token\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:24:02.247092 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:24:02.246848 2568 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eee5be4-2a65-4513-8c1e-47db274ca367-credential-socket\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:24:02.247092 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:24:02.246860 2568 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eee5be4-2a65-4513-8c1e-47db274ca367-istio-podinfo\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:24:02.812196 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:24:02.812161 2568 generic.go:356] "Generic (PLEG): container finished" podID="6eee5be4-2a65-4513-8c1e-47db274ca367" containerID="df8c80348d8d9ee05a0f37a4dfcc7119d2d1e6e12ffd5caec7a11c32ef786302" exitCode=0 Sep 11 04:24:02.812363 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:24:02.812208 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" Sep 11 04:24:02.812363 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:24:02.812240 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" event={"ID":"6eee5be4-2a65-4513-8c1e-47db274ca367","Type":"ContainerDied","Data":"df8c80348d8d9ee05a0f37a4dfcc7119d2d1e6e12ffd5caec7a11c32ef786302"} Sep 11 04:24:02.812363 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:24:02.812296 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj" event={"ID":"6eee5be4-2a65-4513-8c1e-47db274ca367","Type":"ContainerDied","Data":"00e674f5fc8c119c3f689b96a13eac1435306e909e06f47933aaac12cbd477f4"} Sep 11 04:24:02.812363 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:24:02.812327 2568 scope.go:117] "RemoveContainer" containerID="df8c80348d8d9ee05a0f37a4dfcc7119d2d1e6e12ffd5caec7a11c32ef786302" Sep 11 04:24:02.821421 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:24:02.821405 2568 scope.go:117] "RemoveContainer" containerID="28b3f2b7dc4594fb91a56c8b3869cf01bc8cef34ce0eb47ab45af272d5dcf75d" Sep 11 04:24:02.829133 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:24:02.829115 2568 scope.go:117] "RemoveContainer" containerID="df8c80348d8d9ee05a0f37a4dfcc7119d2d1e6e12ffd5caec7a11c32ef786302" Sep 11 04:24:02.829393 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:24:02.829373 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df8c80348d8d9ee05a0f37a4dfcc7119d2d1e6e12ffd5caec7a11c32ef786302\": container with ID starting with df8c80348d8d9ee05a0f37a4dfcc7119d2d1e6e12ffd5caec7a11c32ef786302 not found: ID does not exist" containerID="df8c80348d8d9ee05a0f37a4dfcc7119d2d1e6e12ffd5caec7a11c32ef786302" Sep 11 04:24:02.829457 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:24:02.829400 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df8c80348d8d9ee05a0f37a4dfcc7119d2d1e6e12ffd5caec7a11c32ef786302"} err="failed to get container status \"df8c80348d8d9ee05a0f37a4dfcc7119d2d1e6e12ffd5caec7a11c32ef786302\": rpc error: code = NotFound desc = could not find container \"df8c80348d8d9ee05a0f37a4dfcc7119d2d1e6e12ffd5caec7a11c32ef786302\": container with ID starting with df8c80348d8d9ee05a0f37a4dfcc7119d2d1e6e12ffd5caec7a11c32ef786302 not found: ID does not exist" Sep 11 04:24:02.829457 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:24:02.829417 2568 scope.go:117] "RemoveContainer" containerID="28b3f2b7dc4594fb91a56c8b3869cf01bc8cef34ce0eb47ab45af272d5dcf75d" Sep 11 04:24:02.829667 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:24:02.829650 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28b3f2b7dc4594fb91a56c8b3869cf01bc8cef34ce0eb47ab45af272d5dcf75d\": container with ID starting with 28b3f2b7dc4594fb91a56c8b3869cf01bc8cef34ce0eb47ab45af272d5dcf75d not found: ID does not exist" containerID="28b3f2b7dc4594fb91a56c8b3869cf01bc8cef34ce0eb47ab45af272d5dcf75d" Sep 11 04:24:02.829719 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:24:02.829672 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28b3f2b7dc4594fb91a56c8b3869cf01bc8cef34ce0eb47ab45af272d5dcf75d"} err="failed to get container status \"28b3f2b7dc4594fb91a56c8b3869cf01bc8cef34ce0eb47ab45af272d5dcf75d\": rpc error: code = NotFound desc = could not find container \"28b3f2b7dc4594fb91a56c8b3869cf01bc8cef34ce0eb47ab45af272d5dcf75d\": container with ID starting with 28b3f2b7dc4594fb91a56c8b3869cf01bc8cef34ce0eb47ab45af272d5dcf75d not found: ID does not exist" Sep 11 04:24:02.832747 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:24:02.832725 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj"] Sep 11 04:24:02.836219 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:24:02.836199 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-9342ed-openshift-default-b985b9d4-bgqmj"] Sep 11 04:24:03.768407 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:24:03.768372 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6eee5be4-2a65-4513-8c1e-47db274ca367" path="/var/lib/kubelet/pods/6eee5be4-2a65-4513-8c1e-47db274ca367/volumes" Sep 11 04:24:12.244262 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:24:12.244218 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:24:13.396348 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:24:13.396306 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:24:45.858124 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:24:45.858022 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:25:14.585969 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:25:14.585926 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:25:14.644463 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:25:14.644426 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:25:22.179881 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:25:22.179824 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:25:22.771008 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:25:22.770967 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:25:24.433681 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:25:24.433646 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:26:36.567463 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:26:36.567427 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l9blm"] Sep 11 04:26:36.568023 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:26:36.567664 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l9blm" podUID="3fc8812f-fd87-492c-b430-973faacc06d8" containerName="manager" containerID="cri-o://8cbacb61806252760d034b1f19e5b40a1710cf19d187456c9295c60e6114c493" gracePeriod=10 Sep 11 04:26:36.811092 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:26:36.811066 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l9blm" Sep 11 04:26:36.889380 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:26:36.889295 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tgm6d\" (UniqueName: \"kubernetes.io/projected/3fc8812f-fd87-492c-b430-973faacc06d8-kube-api-access-tgm6d\") pod \"3fc8812f-fd87-492c-b430-973faacc06d8\" (UID: \"3fc8812f-fd87-492c-b430-973faacc06d8\") " Sep 11 04:26:36.889380 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:26:36.889346 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3fc8812f-fd87-492c-b430-973faacc06d8-extensions-socket-volume\") pod \"3fc8812f-fd87-492c-b430-973faacc06d8\" (UID: \"3fc8812f-fd87-492c-b430-973faacc06d8\") " Sep 11 04:26:36.889772 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:26:36.889739 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fc8812f-fd87-492c-b430-973faacc06d8-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3fc8812f-fd87-492c-b430-973faacc06d8" (UID: "3fc8812f-fd87-492c-b430-973faacc06d8"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:26:36.891423 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:26:36.891393 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fc8812f-fd87-492c-b430-973faacc06d8-kube-api-access-tgm6d" (OuterVolumeSpecName: "kube-api-access-tgm6d") pod "3fc8812f-fd87-492c-b430-973faacc06d8" (UID: "3fc8812f-fd87-492c-b430-973faacc06d8"). InnerVolumeSpecName "kube-api-access-tgm6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:26:36.990062 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:26:36.990017 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgm6d\" (UniqueName: \"kubernetes.io/projected/3fc8812f-fd87-492c-b430-973faacc06d8-kube-api-access-tgm6d\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:26:36.990062 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:26:36.990041 2568 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3fc8812f-fd87-492c-b430-973faacc06d8-extensions-socket-volume\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:26:37.503848 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:26:37.503809 2568 generic.go:356] "Generic (PLEG): container finished" podID="3fc8812f-fd87-492c-b430-973faacc06d8" containerID="8cbacb61806252760d034b1f19e5b40a1710cf19d187456c9295c60e6114c493" exitCode=0 Sep 11 04:26:37.504020 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:26:37.503861 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l9blm" Sep 11 04:26:37.504020 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:26:37.503894 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l9blm" event={"ID":"3fc8812f-fd87-492c-b430-973faacc06d8","Type":"ContainerDied","Data":"8cbacb61806252760d034b1f19e5b40a1710cf19d187456c9295c60e6114c493"} Sep 11 04:26:37.504020 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:26:37.503948 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l9blm" event={"ID":"3fc8812f-fd87-492c-b430-973faacc06d8","Type":"ContainerDied","Data":"a38a5013cc48169cda5fdce8419b22e92cb1b5870ce49d7c783d47b2593a4f4c"} Sep 11 04:26:37.504020 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:26:37.503970 2568 scope.go:117] "RemoveContainer" containerID="8cbacb61806252760d034b1f19e5b40a1710cf19d187456c9295c60e6114c493" Sep 11 04:26:37.513404 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:26:37.513360 2568 scope.go:117] "RemoveContainer" containerID="8cbacb61806252760d034b1f19e5b40a1710cf19d187456c9295c60e6114c493" Sep 11 04:26:37.513675 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:26:37.513653 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cbacb61806252760d034b1f19e5b40a1710cf19d187456c9295c60e6114c493\": container with ID starting with 8cbacb61806252760d034b1f19e5b40a1710cf19d187456c9295c60e6114c493 not found: ID does not exist" containerID="8cbacb61806252760d034b1f19e5b40a1710cf19d187456c9295c60e6114c493" Sep 11 04:26:37.513735 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:26:37.513684 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cbacb61806252760d034b1f19e5b40a1710cf19d187456c9295c60e6114c493"} err="failed to get container status \"8cbacb61806252760d034b1f19e5b40a1710cf19d187456c9295c60e6114c493\": rpc error: code = NotFound desc = could not find container \"8cbacb61806252760d034b1f19e5b40a1710cf19d187456c9295c60e6114c493\": container with ID starting with 8cbacb61806252760d034b1f19e5b40a1710cf19d187456c9295c60e6114c493 not found: ID does not exist" Sep 11 04:26:37.525542 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:26:37.525510 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l9blm"] Sep 11 04:26:37.529649 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:26:37.529620 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-l9blm"] Sep 11 04:26:37.767744 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:26:37.767658 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fc8812f-fd87-492c-b430-973faacc06d8" path="/var/lib/kubelet/pods/3fc8812f-fd87-492c-b430-973faacc06d8/volumes" Sep 11 04:27:12.645249 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:27:12.645200 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:27:37.284378 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:27:37.284292 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hsqnh"] Sep 11 04:27:37.284827 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:27:37.284600 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fc8812f-fd87-492c-b430-973faacc06d8" containerName="manager" Sep 11 04:27:37.284827 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:27:37.284611 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fc8812f-fd87-492c-b430-973faacc06d8" containerName="manager" Sep 11 04:27:37.284827 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:27:37.284629 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38802f54-40d4-4274-99d5-416a34ffcfd0" containerName="revoke-keys" Sep 11 04:27:37.284827 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:27:37.284635 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="38802f54-40d4-4274-99d5-416a34ffcfd0" containerName="revoke-keys" Sep 11 04:27:37.284827 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:27:37.284642 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6eee5be4-2a65-4513-8c1e-47db274ca367" containerName="istio-proxy" Sep 11 04:27:37.284827 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:27:37.284648 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6eee5be4-2a65-4513-8c1e-47db274ca367" containerName="istio-proxy" Sep 11 04:27:37.284827 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:27:37.284655 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8795982f-8f14-4d43-948c-f1966fc1be21" containerName="istio-proxy" Sep 11 04:27:37.284827 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:27:37.284660 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="8795982f-8f14-4d43-948c-f1966fc1be21" containerName="istio-proxy" Sep 11 04:27:37.284827 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:27:37.284671 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8795982f-8f14-4d43-948c-f1966fc1be21" containerName="istio-proxy" Sep 11 04:27:37.284827 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:27:37.284676 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="8795982f-8f14-4d43-948c-f1966fc1be21" containerName="istio-proxy" Sep 11 04:27:37.284827 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:27:37.284726 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="8795982f-8f14-4d43-948c-f1966fc1be21" containerName="istio-proxy" Sep 11 04:27:37.284827 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:27:37.284734 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="38802f54-40d4-4274-99d5-416a34ffcfd0" containerName="revoke-keys" Sep 11 04:27:37.284827 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:27:37.284742 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fc8812f-fd87-492c-b430-973faacc06d8" containerName="manager" Sep 11 04:27:37.284827 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:27:37.284751 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="6eee5be4-2a65-4513-8c1e-47db274ca367" containerName="istio-proxy" Sep 11 04:27:37.284827 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:27:37.284759 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="6eee5be4-2a65-4513-8c1e-47db274ca367" containerName="istio-proxy" Sep 11 04:27:37.287624 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:27:37.287606 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hsqnh" Sep 11 04:27:37.290207 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:27:37.290189 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-dzb9t\"" Sep 11 04:27:37.299564 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:27:37.299540 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hsqnh"] Sep 11 04:27:37.483068 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:27:37.483008 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dcca20bf-0991-4d77-baff-cea0bb250831-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-hsqnh\" (UID: \"dcca20bf-0991-4d77-baff-cea0bb250831\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hsqnh" Sep 11 04:27:37.483245 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:27:37.483142 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpn75\" (UniqueName: \"kubernetes.io/projected/dcca20bf-0991-4d77-baff-cea0bb250831-kube-api-access-dpn75\") pod \"kuadrant-operator-controller-manager-5769f99d9f-hsqnh\" (UID: \"dcca20bf-0991-4d77-baff-cea0bb250831\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hsqnh" Sep 11 04:27:37.583909 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:27:37.583804 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dcca20bf-0991-4d77-baff-cea0bb250831-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-hsqnh\" (UID: \"dcca20bf-0991-4d77-baff-cea0bb250831\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hsqnh" Sep 11 04:27:37.583909 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:27:37.583890 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpn75\" (UniqueName: \"kubernetes.io/projected/dcca20bf-0991-4d77-baff-cea0bb250831-kube-api-access-dpn75\") pod \"kuadrant-operator-controller-manager-5769f99d9f-hsqnh\" (UID: \"dcca20bf-0991-4d77-baff-cea0bb250831\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hsqnh" Sep 11 04:27:37.584223 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:27:37.584199 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dcca20bf-0991-4d77-baff-cea0bb250831-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-hsqnh\" (UID: \"dcca20bf-0991-4d77-baff-cea0bb250831\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hsqnh" Sep 11 04:27:37.595309 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:27:37.595284 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpn75\" (UniqueName: \"kubernetes.io/projected/dcca20bf-0991-4d77-baff-cea0bb250831-kube-api-access-dpn75\") pod \"kuadrant-operator-controller-manager-5769f99d9f-hsqnh\" (UID: \"dcca20bf-0991-4d77-baff-cea0bb250831\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hsqnh" Sep 11 04:27:37.597079 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:27:37.597042 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hsqnh" Sep 11 04:27:37.872294 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:27:37.872234 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hsqnh"] Sep 11 04:27:37.874711 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:27:37.874682 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcca20bf_0991_4d77_baff_cea0bb250831.slice/crio-f20a3466288e71554c288a08a3768cbad34585a10f0871db28baae7439102461 WatchSource:0}: Error finding container f20a3466288e71554c288a08a3768cbad34585a10f0871db28baae7439102461: Status 404 returned error can't find the container with id f20a3466288e71554c288a08a3768cbad34585a10f0871db28baae7439102461 Sep 11 04:27:38.785116 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:27:38.785079 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hsqnh" event={"ID":"dcca20bf-0991-4d77-baff-cea0bb250831","Type":"ContainerStarted","Data":"0000e882eb44743c9fd2f30196a125809bf0b1957b82125e56116340f0b4b12b"} Sep 11 04:27:38.785495 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:27:38.785141 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hsqnh" event={"ID":"dcca20bf-0991-4d77-baff-cea0bb250831","Type":"ContainerStarted","Data":"f20a3466288e71554c288a08a3768cbad34585a10f0871db28baae7439102461"} Sep 11 04:27:38.785495 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:27:38.785176 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hsqnh" Sep 11 04:27:38.806409 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:27:38.806348 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hsqnh" podStartSLOduration=1.806326897 podStartE2EDuration="1.806326897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:27:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:27:38.804570721 +0000 UTC m=+2073.558202125" watchObservedRunningTime="2026-09-11 04:27:38.806326897 +0000 UTC m=+2073.559958303" Sep 11 04:27:49.790942 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:27:49.790906 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-hsqnh" Sep 11 04:28:05.880798 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:05.880771 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-5tpdc_9112557d-d0ef-44c7-8b2a-f6ecd0a980d6/console-operator/1.log" Sep 11 04:28:05.886662 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:05.886642 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-5tpdc_9112557d-d0ef-44c7-8b2a-f6ecd0a980d6/console-operator/1.log" Sep 11 04:28:23.192257 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:23.192218 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f"] Sep 11 04:28:23.194653 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:23.192722 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6eee5be4-2a65-4513-8c1e-47db274ca367" containerName="istio-proxy" Sep 11 04:28:23.194653 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:23.192740 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="6eee5be4-2a65-4513-8c1e-47db274ca367" containerName="istio-proxy" Sep 11 04:28:23.194653 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:23.192844 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="8795982f-8f14-4d43-948c-f1966fc1be21" containerName="istio-proxy" Sep 11 04:28:23.195861 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:23.195847 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f" Sep 11 04:28:23.198417 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:23.198394 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 11 04:28:23.199274 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:23.199256 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-97jpl\"" Sep 11 04:28:23.199361 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:23.199283 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-b8ef2310-kserve-self-signed-certs\"" Sep 11 04:28:23.199361 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:23.199302 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 11 04:28:23.206005 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:23.205980 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f"] Sep 11 04:28:23.241673 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:23.241635 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:28:23.361220 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:23.361181 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b80918c0-877f-414e-a9d6-e017c0debf1f-tls-certs\") pod \"e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f\" (UID: \"b80918c0-877f-414e-a9d6-e017c0debf1f\") " pod="llm/e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f" Sep 11 04:28:23.361410 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:23.361228 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b80918c0-877f-414e-a9d6-e017c0debf1f-dshm\") pod \"e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f\" (UID: \"b80918c0-877f-414e-a9d6-e017c0debf1f\") " pod="llm/e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f" Sep 11 04:28:23.361410 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:23.361311 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b80918c0-877f-414e-a9d6-e017c0debf1f-tmp-dir\") pod \"e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f\" (UID: \"b80918c0-877f-414e-a9d6-e017c0debf1f\") " pod="llm/e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f" Sep 11 04:28:23.361410 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:23.361350 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b80918c0-877f-414e-a9d6-e017c0debf1f-home\") pod \"e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f\" (UID: \"b80918c0-877f-414e-a9d6-e017c0debf1f\") " pod="llm/e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f" Sep 11 04:28:23.361410 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:23.361377 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncvtx\" (UniqueName: \"kubernetes.io/projected/b80918c0-877f-414e-a9d6-e017c0debf1f-kube-api-access-ncvtx\") pod \"e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f\" (UID: \"b80918c0-877f-414e-a9d6-e017c0debf1f\") " pod="llm/e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f" Sep 11 04:28:23.361564 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:23.361433 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b80918c0-877f-414e-a9d6-e017c0debf1f-model-cache\") pod \"e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f\" (UID: \"b80918c0-877f-414e-a9d6-e017c0debf1f\") " pod="llm/e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f" Sep 11 04:28:23.462603 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:23.462510 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b80918c0-877f-414e-a9d6-e017c0debf1f-tmp-dir\") pod \"e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f\" (UID: \"b80918c0-877f-414e-a9d6-e017c0debf1f\") " pod="llm/e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f" Sep 11 04:28:23.462603 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:23.462555 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b80918c0-877f-414e-a9d6-e017c0debf1f-home\") pod \"e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f\" (UID: \"b80918c0-877f-414e-a9d6-e017c0debf1f\") " pod="llm/e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f" Sep 11 04:28:23.462603 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:23.462591 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncvtx\" (UniqueName: \"kubernetes.io/projected/b80918c0-877f-414e-a9d6-e017c0debf1f-kube-api-access-ncvtx\") pod \"e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f\" (UID: \"b80918c0-877f-414e-a9d6-e017c0debf1f\") " pod="llm/e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f" Sep 11 04:28:23.462879 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:23.462621 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b80918c0-877f-414e-a9d6-e017c0debf1f-model-cache\") pod \"e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f\" (UID: \"b80918c0-877f-414e-a9d6-e017c0debf1f\") " pod="llm/e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f" Sep 11 04:28:23.462879 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:23.462666 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b80918c0-877f-414e-a9d6-e017c0debf1f-tls-certs\") pod \"e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f\" (UID: \"b80918c0-877f-414e-a9d6-e017c0debf1f\") " pod="llm/e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f" Sep 11 04:28:23.462879 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:23.462704 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b80918c0-877f-414e-a9d6-e017c0debf1f-dshm\") pod \"e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f\" (UID: \"b80918c0-877f-414e-a9d6-e017c0debf1f\") " pod="llm/e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f" Sep 11 04:28:23.463029 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:23.462993 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b80918c0-877f-414e-a9d6-e017c0debf1f-tmp-dir\") pod \"e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f\" (UID: \"b80918c0-877f-414e-a9d6-e017c0debf1f\") " pod="llm/e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f" Sep 11 04:28:23.463114 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:23.463032 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b80918c0-877f-414e-a9d6-e017c0debf1f-home\") pod \"e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f\" (UID: \"b80918c0-877f-414e-a9d6-e017c0debf1f\") " pod="llm/e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f" Sep 11 04:28:23.463114 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:23.463087 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b80918c0-877f-414e-a9d6-e017c0debf1f-model-cache\") pod \"e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f\" (UID: \"b80918c0-877f-414e-a9d6-e017c0debf1f\") " pod="llm/e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f" Sep 11 04:28:23.464988 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:23.464962 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b80918c0-877f-414e-a9d6-e017c0debf1f-dshm\") pod \"e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f\" (UID: \"b80918c0-877f-414e-a9d6-e017c0debf1f\") " pod="llm/e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f" Sep 11 04:28:23.465218 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:23.465203 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b80918c0-877f-414e-a9d6-e017c0debf1f-tls-certs\") pod \"e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f\" (UID: \"b80918c0-877f-414e-a9d6-e017c0debf1f\") " pod="llm/e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f" Sep 11 04:28:23.471370 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:23.471348 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncvtx\" (UniqueName: \"kubernetes.io/projected/b80918c0-877f-414e-a9d6-e017c0debf1f-kube-api-access-ncvtx\") pod \"e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f\" (UID: \"b80918c0-877f-414e-a9d6-e017c0debf1f\") " pod="llm/e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f" Sep 11 04:28:23.505902 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:23.505881 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f" Sep 11 04:28:23.778026 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:23.777907 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f"] Sep 11 04:28:23.780295 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:28:23.780268 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb80918c0_877f_414e_a9d6_e017c0debf1f.slice/crio-9ff91941e55f07476fb6c49484c342f8aa03d5d1046d783486ac51e52ae263d7 WatchSource:0}: Error finding container 9ff91941e55f07476fb6c49484c342f8aa03d5d1046d783486ac51e52ae263d7: Status 404 returned error can't find the container with id 9ff91941e55f07476fb6c49484c342f8aa03d5d1046d783486ac51e52ae263d7 Sep 11 04:28:23.988714 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:23.988672 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f" event={"ID":"b80918c0-877f-414e-a9d6-e017c0debf1f","Type":"ContainerStarted","Data":"79ad9d49fc3c8d309dc3ef7f60caa1a320c13b6b331c96143121f5c83288879f"} Sep 11 04:28:23.988714 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:23.988720 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f" event={"ID":"b80918c0-877f-414e-a9d6-e017c0debf1f","Type":"ContainerStarted","Data":"9ff91941e55f07476fb6c49484c342f8aa03d5d1046d783486ac51e52ae263d7"} Sep 11 04:28:24.006730 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:24.006682 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f" podStartSLOduration=1.006665898 podStartE2EDuration="1.006665898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:28:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:28:24.005667782 +0000 UTC m=+2118.759299177" watchObservedRunningTime="2026-09-11 04:28:24.006665898 +0000 UTC m=+2118.760297299" Sep 11 04:28:33.507059 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:33.507020 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f" Sep 11 04:28:33.507573 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:33.507107 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f" Sep 11 04:28:33.519641 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:33.519619 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f" Sep 11 04:28:34.045134 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:34.045104 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f" Sep 11 04:28:37.277780 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:37.277745 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq"] Sep 11 04:28:37.281315 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:37.281293 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq" Sep 11 04:28:37.283547 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:37.283526 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-b8ef2310-kserve-self-signed-certs\"" Sep 11 04:28:37.284963 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:37.284939 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58535024-911f-44b4-b1e0-d2a49f38a312-dshm\") pod \"e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq\" (UID: \"58535024-911f-44b4-b1e0-d2a49f38a312\") " pod="llm/e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq" Sep 11 04:28:37.285042 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:37.285006 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58535024-911f-44b4-b1e0-d2a49f38a312-home\") pod \"e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq\" (UID: \"58535024-911f-44b4-b1e0-d2a49f38a312\") " pod="llm/e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq" Sep 11 04:28:37.285042 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:37.285025 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58535024-911f-44b4-b1e0-d2a49f38a312-tmp-dir\") pod \"e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq\" (UID: \"58535024-911f-44b4-b1e0-d2a49f38a312\") " pod="llm/e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq" Sep 11 04:28:37.285132 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:37.285042 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtpr5\" (UniqueName: \"kubernetes.io/projected/58535024-911f-44b4-b1e0-d2a49f38a312-kube-api-access-vtpr5\") pod \"e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq\" (UID: \"58535024-911f-44b4-b1e0-d2a49f38a312\") " pod="llm/e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq" Sep 11 04:28:37.285168 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:37.285134 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58535024-911f-44b4-b1e0-d2a49f38a312-model-cache\") pod \"e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq\" (UID: \"58535024-911f-44b4-b1e0-d2a49f38a312\") " pod="llm/e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq" Sep 11 04:28:37.285232 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:37.285212 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58535024-911f-44b4-b1e0-d2a49f38a312-tls-certs\") pod \"e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq\" (UID: \"58535024-911f-44b4-b1e0-d2a49f38a312\") " pod="llm/e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq" Sep 11 04:28:37.292923 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:37.292900 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq"] Sep 11 04:28:37.385706 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:37.385670 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58535024-911f-44b4-b1e0-d2a49f38a312-home\") pod \"e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq\" (UID: \"58535024-911f-44b4-b1e0-d2a49f38a312\") " pod="llm/e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq" Sep 11 04:28:37.385706 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:37.385711 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58535024-911f-44b4-b1e0-d2a49f38a312-tmp-dir\") pod \"e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq\" (UID: \"58535024-911f-44b4-b1e0-d2a49f38a312\") " pod="llm/e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq" Sep 11 04:28:37.385951 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:37.385728 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtpr5\" (UniqueName: \"kubernetes.io/projected/58535024-911f-44b4-b1e0-d2a49f38a312-kube-api-access-vtpr5\") pod \"e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq\" (UID: \"58535024-911f-44b4-b1e0-d2a49f38a312\") " pod="llm/e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq" Sep 11 04:28:37.385951 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:37.385750 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58535024-911f-44b4-b1e0-d2a49f38a312-model-cache\") pod \"e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq\" (UID: \"58535024-911f-44b4-b1e0-d2a49f38a312\") " pod="llm/e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq" Sep 11 04:28:37.385951 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:37.385807 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58535024-911f-44b4-b1e0-d2a49f38a312-tls-certs\") pod \"e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq\" (UID: \"58535024-911f-44b4-b1e0-d2a49f38a312\") " pod="llm/e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq" Sep 11 04:28:37.385951 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:37.385851 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58535024-911f-44b4-b1e0-d2a49f38a312-dshm\") pod \"e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq\" (UID: \"58535024-911f-44b4-b1e0-d2a49f38a312\") " pod="llm/e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq" Sep 11 04:28:37.386181 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:37.386159 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58535024-911f-44b4-b1e0-d2a49f38a312-home\") pod \"e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq\" (UID: \"58535024-911f-44b4-b1e0-d2a49f38a312\") " pod="llm/e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq" Sep 11 04:28:37.386246 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:37.386225 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58535024-911f-44b4-b1e0-d2a49f38a312-model-cache\") pod \"e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq\" (UID: \"58535024-911f-44b4-b1e0-d2a49f38a312\") " pod="llm/e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq" Sep 11 04:28:37.386284 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:37.386242 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58535024-911f-44b4-b1e0-d2a49f38a312-tmp-dir\") pod \"e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq\" (UID: \"58535024-911f-44b4-b1e0-d2a49f38a312\") " pod="llm/e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq" Sep 11 04:28:37.388110 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:37.388090 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58535024-911f-44b4-b1e0-d2a49f38a312-dshm\") pod \"e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq\" (UID: \"58535024-911f-44b4-b1e0-d2a49f38a312\") " pod="llm/e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq" Sep 11 04:28:37.388376 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:37.388356 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58535024-911f-44b4-b1e0-d2a49f38a312-tls-certs\") pod \"e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq\" (UID: \"58535024-911f-44b4-b1e0-d2a49f38a312\") " pod="llm/e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq" Sep 11 04:28:37.401967 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:37.401946 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtpr5\" (UniqueName: \"kubernetes.io/projected/58535024-911f-44b4-b1e0-d2a49f38a312-kube-api-access-vtpr5\") pod \"e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq\" (UID: \"58535024-911f-44b4-b1e0-d2a49f38a312\") " pod="llm/e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq" Sep 11 04:28:37.403719 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:37.403689 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:28:37.591345 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:37.591255 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq" Sep 11 04:28:37.869633 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:37.869608 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq"] Sep 11 04:28:37.871988 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:28:37.871958 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58535024_911f_44b4_b1e0_d2a49f38a312.slice/crio-93172a47544c54af56b23029fc5bed04a4a9d5c4482aaa1065b778e56549c290 WatchSource:0}: Error finding container 93172a47544c54af56b23029fc5bed04a4a9d5c4482aaa1065b778e56549c290: Status 404 returned error can't find the container with id 93172a47544c54af56b23029fc5bed04a4a9d5c4482aaa1065b778e56549c290 Sep 11 04:28:38.055524 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:38.055479 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq" event={"ID":"58535024-911f-44b4-b1e0-d2a49f38a312","Type":"ContainerStarted","Data":"90a92309722d815f2f8a76f0c7f5f694557db06fbfd11f19b37f7712ccff5649"} Sep 11 04:28:38.055524 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:38.055529 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq" event={"ID":"58535024-911f-44b4-b1e0-d2a49f38a312","Type":"ContainerStarted","Data":"93172a47544c54af56b23029fc5bed04a4a9d5c4482aaa1065b778e56549c290"} Sep 11 04:28:38.083837 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:38.083788 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq" podStartSLOduration=1.083772463 podStartE2EDuration="1.083772463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:28:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:28:38.08287315 +0000 UTC m=+2132.836504553" watchObservedRunningTime="2026-09-11 04:28:38.083772463 +0000 UTC m=+2132.837403902" Sep 11 04:28:47.591564 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:47.591526 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq" Sep 11 04:28:47.591564 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:47.591572 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq" Sep 11 04:28:47.604129 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:47.604100 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq" Sep 11 04:28:48.111205 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:28:48.111166 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq" Sep 11 04:29:05.051841 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:05.051805 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f"] Sep 11 04:29:05.072642 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:05.072607 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:29:06.804639 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:06.804603 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq"] Sep 11 04:29:06.831931 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:06.831892 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:29:20.071402 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:20.071330 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f" podUID="b80918c0-877f-414e-a9d6-e017c0debf1f" containerName="main" containerID="cri-o://79ad9d49fc3c8d309dc3ef7f60caa1a320c13b6b331c96143121f5c83288879f" gracePeriod=45 Sep 11 04:29:20.250489 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:20.250451 2568 generic.go:356] "Generic (PLEG): container finished" podID="b80918c0-877f-414e-a9d6-e017c0debf1f" containerID="79ad9d49fc3c8d309dc3ef7f60caa1a320c13b6b331c96143121f5c83288879f" exitCode=0 Sep 11 04:29:20.250653 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:20.250520 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f" event={"ID":"b80918c0-877f-414e-a9d6-e017c0debf1f","Type":"ContainerDied","Data":"79ad9d49fc3c8d309dc3ef7f60caa1a320c13b6b331c96143121f5c83288879f"} Sep 11 04:29:20.329997 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:20.329937 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f" Sep 11 04:29:20.452697 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:20.452664 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b80918c0-877f-414e-a9d6-e017c0debf1f-home\") pod \"b80918c0-877f-414e-a9d6-e017c0debf1f\" (UID: \"b80918c0-877f-414e-a9d6-e017c0debf1f\") " Sep 11 04:29:20.452869 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:20.452711 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b80918c0-877f-414e-a9d6-e017c0debf1f-dshm\") pod \"b80918c0-877f-414e-a9d6-e017c0debf1f\" (UID: \"b80918c0-877f-414e-a9d6-e017c0debf1f\") " Sep 11 04:29:20.452869 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:20.452732 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b80918c0-877f-414e-a9d6-e017c0debf1f-tls-certs\") pod \"b80918c0-877f-414e-a9d6-e017c0debf1f\" (UID: \"b80918c0-877f-414e-a9d6-e017c0debf1f\") " Sep 11 04:29:20.452869 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:20.452755 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b80918c0-877f-414e-a9d6-e017c0debf1f-tmp-dir\") pod \"b80918c0-877f-414e-a9d6-e017c0debf1f\" (UID: \"b80918c0-877f-414e-a9d6-e017c0debf1f\") " Sep 11 04:29:20.452869 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:20.452847 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b80918c0-877f-414e-a9d6-e017c0debf1f-model-cache\") pod \"b80918c0-877f-414e-a9d6-e017c0debf1f\" (UID: \"b80918c0-877f-414e-a9d6-e017c0debf1f\") " Sep 11 04:29:20.453131 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:20.452906 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ncvtx\" (UniqueName: \"kubernetes.io/projected/b80918c0-877f-414e-a9d6-e017c0debf1f-kube-api-access-ncvtx\") pod \"b80918c0-877f-414e-a9d6-e017c0debf1f\" (UID: \"b80918c0-877f-414e-a9d6-e017c0debf1f\") " Sep 11 04:29:20.453131 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:20.452960 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b80918c0-877f-414e-a9d6-e017c0debf1f-home" (OuterVolumeSpecName: "home") pod "b80918c0-877f-414e-a9d6-e017c0debf1f" (UID: "b80918c0-877f-414e-a9d6-e017c0debf1f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:29:20.453131 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:20.453019 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b80918c0-877f-414e-a9d6-e017c0debf1f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b80918c0-877f-414e-a9d6-e017c0debf1f" (UID: "b80918c0-877f-414e-a9d6-e017c0debf1f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:29:20.453298 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:20.453150 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b80918c0-877f-414e-a9d6-e017c0debf1f-model-cache" (OuterVolumeSpecName: "model-cache") pod "b80918c0-877f-414e-a9d6-e017c0debf1f" (UID: "b80918c0-877f-414e-a9d6-e017c0debf1f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:29:20.453298 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:20.453184 2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b80918c0-877f-414e-a9d6-e017c0debf1f-home\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:29:20.453298 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:20.453198 2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b80918c0-877f-414e-a9d6-e017c0debf1f-tmp-dir\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:29:20.454952 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:20.454928 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b80918c0-877f-414e-a9d6-e017c0debf1f-dshm" (OuterVolumeSpecName: "dshm") pod "b80918c0-877f-414e-a9d6-e017c0debf1f" (UID: "b80918c0-877f-414e-a9d6-e017c0debf1f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:29:20.455109 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:20.455014 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b80918c0-877f-414e-a9d6-e017c0debf1f-kube-api-access-ncvtx" (OuterVolumeSpecName: "kube-api-access-ncvtx") pod "b80918c0-877f-414e-a9d6-e017c0debf1f" (UID: "b80918c0-877f-414e-a9d6-e017c0debf1f"). InnerVolumeSpecName "kube-api-access-ncvtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:29:20.455109 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:20.455069 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b80918c0-877f-414e-a9d6-e017c0debf1f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b80918c0-877f-414e-a9d6-e017c0debf1f" (UID: "b80918c0-877f-414e-a9d6-e017c0debf1f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 04:29:20.554176 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:20.554145 2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b80918c0-877f-414e-a9d6-e017c0debf1f-model-cache\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:29:20.554176 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:20.554175 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncvtx\" (UniqueName: \"kubernetes.io/projected/b80918c0-877f-414e-a9d6-e017c0debf1f-kube-api-access-ncvtx\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:29:20.554341 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:20.554189 2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b80918c0-877f-414e-a9d6-e017c0debf1f-dshm\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:29:20.554341 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:20.554201 2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b80918c0-877f-414e-a9d6-e017c0debf1f-tls-certs\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:29:21.256404 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:21.256369 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f" Sep 11 04:29:21.256823 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:21.256380 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f" event={"ID":"b80918c0-877f-414e-a9d6-e017c0debf1f","Type":"ContainerDied","Data":"9ff91941e55f07476fb6c49484c342f8aa03d5d1046d783486ac51e52ae263d7"} Sep 11 04:29:21.256823 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:21.256497 2568 scope.go:117] "RemoveContainer" containerID="79ad9d49fc3c8d309dc3ef7f60caa1a320c13b6b331c96143121f5c83288879f" Sep 11 04:29:21.276894 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:21.276870 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f"] Sep 11 04:29:21.280913 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:21.280891 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-b8ef2310-kserve-c8fbf7d7f-rsp2f"] Sep 11 04:29:21.768381 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:21.768347 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b80918c0-877f-414e-a9d6-e017c0debf1f" path="/var/lib/kubelet/pods/b80918c0-877f-414e-a9d6-e017c0debf1f/volumes" Sep 11 04:29:21.823679 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:21.823643 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq" podUID="58535024-911f-44b4-b1e0-d2a49f38a312" containerName="main" containerID="cri-o://90a92309722d815f2f8a76f0c7f5f694557db06fbfd11f19b37f7712ccff5649" gracePeriod=45 Sep 11 04:29:22.069901 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:22.069875 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq" Sep 11 04:29:22.262935 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:22.262894 2568 generic.go:356] "Generic (PLEG): container finished" podID="58535024-911f-44b4-b1e0-d2a49f38a312" containerID="90a92309722d815f2f8a76f0c7f5f694557db06fbfd11f19b37f7712ccff5649" exitCode=0 Sep 11 04:29:22.262935 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:22.262939 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq" Sep 11 04:29:22.263425 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:22.262950 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq" event={"ID":"58535024-911f-44b4-b1e0-d2a49f38a312","Type":"ContainerDied","Data":"90a92309722d815f2f8a76f0c7f5f694557db06fbfd11f19b37f7712ccff5649"} Sep 11 04:29:22.263425 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:22.262987 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq" event={"ID":"58535024-911f-44b4-b1e0-d2a49f38a312","Type":"ContainerDied","Data":"93172a47544c54af56b23029fc5bed04a4a9d5c4482aaa1065b778e56549c290"} Sep 11 04:29:22.263425 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:22.263009 2568 scope.go:117] "RemoveContainer" containerID="90a92309722d815f2f8a76f0c7f5f694557db06fbfd11f19b37f7712ccff5649" Sep 11 04:29:22.266262 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:22.266239 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58535024-911f-44b4-b1e0-d2a49f38a312-tmp-dir\") pod \"58535024-911f-44b4-b1e0-d2a49f38a312\" (UID: \"58535024-911f-44b4-b1e0-d2a49f38a312\") " Sep 11 04:29:22.266391 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:22.266278 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58535024-911f-44b4-b1e0-d2a49f38a312-dshm\") pod \"58535024-911f-44b4-b1e0-d2a49f38a312\" (UID: \"58535024-911f-44b4-b1e0-d2a49f38a312\") " Sep 11 04:29:22.266391 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:22.266317 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58535024-911f-44b4-b1e0-d2a49f38a312-home\") pod \"58535024-911f-44b4-b1e0-d2a49f38a312\" (UID: \"58535024-911f-44b4-b1e0-d2a49f38a312\") " Sep 11 04:29:22.266391 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:22.266345 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtpr5\" (UniqueName: \"kubernetes.io/projected/58535024-911f-44b4-b1e0-d2a49f38a312-kube-api-access-vtpr5\") pod \"58535024-911f-44b4-b1e0-d2a49f38a312\" (UID: \"58535024-911f-44b4-b1e0-d2a49f38a312\") " Sep 11 04:29:22.266556 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:22.266419 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58535024-911f-44b4-b1e0-d2a49f38a312-model-cache\") pod \"58535024-911f-44b4-b1e0-d2a49f38a312\" (UID: \"58535024-911f-44b4-b1e0-d2a49f38a312\") " Sep 11 04:29:22.266556 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:22.266446 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58535024-911f-44b4-b1e0-d2a49f38a312-tls-certs\") pod \"58535024-911f-44b4-b1e0-d2a49f38a312\" (UID: \"58535024-911f-44b4-b1e0-d2a49f38a312\") " Sep 11 04:29:22.266556 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:22.266536 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58535024-911f-44b4-b1e0-d2a49f38a312-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "58535024-911f-44b4-b1e0-d2a49f38a312" (UID: "58535024-911f-44b4-b1e0-d2a49f38a312"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:29:22.266727 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:22.266591 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58535024-911f-44b4-b1e0-d2a49f38a312-home" (OuterVolumeSpecName: "home") pod "58535024-911f-44b4-b1e0-d2a49f38a312" (UID: "58535024-911f-44b4-b1e0-d2a49f38a312"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:29:22.266842 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:22.266817 2568 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58535024-911f-44b4-b1e0-d2a49f38a312-tmp-dir\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:29:22.266963 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:22.266844 2568 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58535024-911f-44b4-b1e0-d2a49f38a312-home\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:29:22.266963 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:22.266950 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58535024-911f-44b4-b1e0-d2a49f38a312-model-cache" (OuterVolumeSpecName: "model-cache") pod "58535024-911f-44b4-b1e0-d2a49f38a312" (UID: "58535024-911f-44b4-b1e0-d2a49f38a312"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:29:22.268660 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:22.268630 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58535024-911f-44b4-b1e0-d2a49f38a312-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "58535024-911f-44b4-b1e0-d2a49f38a312" (UID: "58535024-911f-44b4-b1e0-d2a49f38a312"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 04:29:22.268748 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:22.268716 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58535024-911f-44b4-b1e0-d2a49f38a312-kube-api-access-vtpr5" (OuterVolumeSpecName: "kube-api-access-vtpr5") pod "58535024-911f-44b4-b1e0-d2a49f38a312" (UID: "58535024-911f-44b4-b1e0-d2a49f38a312"). InnerVolumeSpecName "kube-api-access-vtpr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:29:22.268792 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:22.268744 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58535024-911f-44b4-b1e0-d2a49f38a312-dshm" (OuterVolumeSpecName: "dshm") pod "58535024-911f-44b4-b1e0-d2a49f38a312" (UID: "58535024-911f-44b4-b1e0-d2a49f38a312"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 04:29:22.273154 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:22.273137 2568 scope.go:117] "RemoveContainer" containerID="90a92309722d815f2f8a76f0c7f5f694557db06fbfd11f19b37f7712ccff5649" Sep 11 04:29:22.273455 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:29:22.273427 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90a92309722d815f2f8a76f0c7f5f694557db06fbfd11f19b37f7712ccff5649\": container with ID starting with 90a92309722d815f2f8a76f0c7f5f694557db06fbfd11f19b37f7712ccff5649 not found: ID does not exist" containerID="90a92309722d815f2f8a76f0c7f5f694557db06fbfd11f19b37f7712ccff5649" Sep 11 04:29:22.273512 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:22.273455 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90a92309722d815f2f8a76f0c7f5f694557db06fbfd11f19b37f7712ccff5649"} err="failed to get container status \"90a92309722d815f2f8a76f0c7f5f694557db06fbfd11f19b37f7712ccff5649\": rpc error: code = NotFound desc = could not find container \"90a92309722d815f2f8a76f0c7f5f694557db06fbfd11f19b37f7712ccff5649\": container with ID starting with 90a92309722d815f2f8a76f0c7f5f694557db06fbfd11f19b37f7712ccff5649 not found: ID does not exist" Sep 11 04:29:22.367581 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:22.367513 2568 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58535024-911f-44b4-b1e0-d2a49f38a312-dshm\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:29:22.367581 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:22.367540 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtpr5\" (UniqueName: \"kubernetes.io/projected/58535024-911f-44b4-b1e0-d2a49f38a312-kube-api-access-vtpr5\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:29:22.367581 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:22.367555 2568 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58535024-911f-44b4-b1e0-d2a49f38a312-model-cache\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:29:22.367581 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:22.367570 2568 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58535024-911f-44b4-b1e0-d2a49f38a312-tls-certs\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:29:22.584876 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:22.584846 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq"] Sep 11 04:29:22.588591 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:22.588562 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-b8ef2310-kserve-fbdc68b8c-xl4gq"] Sep 11 04:29:23.093388 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:23.093351 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7ff8fbcd4f-ftx5g"] Sep 11 04:29:23.093694 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:23.093680 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b80918c0-877f-414e-a9d6-e017c0debf1f" containerName="main" Sep 11 04:29:23.093741 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:23.093695 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b80918c0-877f-414e-a9d6-e017c0debf1f" containerName="main" Sep 11 04:29:23.093741 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:23.093704 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58535024-911f-44b4-b1e0-d2a49f38a312" containerName="main" Sep 11 04:29:23.093741 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:23.093709 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="58535024-911f-44b4-b1e0-d2a49f38a312" containerName="main" Sep 11 04:29:23.093837 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:23.093758 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="b80918c0-877f-414e-a9d6-e017c0debf1f" containerName="main" Sep 11 04:29:23.093837 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:23.093769 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="58535024-911f-44b4-b1e0-d2a49f38a312" containerName="main" Sep 11 04:29:23.098019 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:23.098001 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7ff8fbcd4f-ftx5g" Sep 11 04:29:23.100502 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:23.100482 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 11 04:29:23.101439 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:23.101419 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 11 04:29:23.101531 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:23.101419 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-b8w8q\"" Sep 11 04:29:23.101531 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:23.101423 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 04:29:23.101531 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:23.101472 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 11 04:29:23.107559 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:23.107538 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7ff8fbcd4f-ftx5g"] Sep 11 04:29:23.113765 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:23.113742 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7f866d8d77-l5j55"] Sep 11 04:29:23.116894 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:23.116880 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f866d8d77-l5j55" Sep 11 04:29:23.126152 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:23.126127 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f866d8d77-l5j55"] Sep 11 04:29:23.174108 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:23.174075 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7cfb3a1-4e53-4dcf-9a57-3a1a44680aaa-maas-api-tls\") pod \"maas-api-7f866d8d77-l5j55\" (UID: \"c7cfb3a1-4e53-4dcf-9a57-3a1a44680aaa\") " pod="odh-ai-gateway-infra/maas-api-7f866d8d77-l5j55" Sep 11 04:29:23.174267 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:23.174114 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttw57\" (UniqueName: \"kubernetes.io/projected/c7cfb3a1-4e53-4dcf-9a57-3a1a44680aaa-kube-api-access-ttw57\") pod \"maas-api-7f866d8d77-l5j55\" (UID: \"c7cfb3a1-4e53-4dcf-9a57-3a1a44680aaa\") " pod="odh-ai-gateway-infra/maas-api-7f866d8d77-l5j55" Sep 11 04:29:23.174267 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:23.174142 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7cfb3a1-4e53-4dcf-9a57-3a1a44680aaa-openshift-service-ca\") pod \"maas-api-7f866d8d77-l5j55\" (UID: \"c7cfb3a1-4e53-4dcf-9a57-3a1a44680aaa\") " pod="odh-ai-gateway-infra/maas-api-7f866d8d77-l5j55" Sep 11 04:29:23.174267 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:23.174180 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7cfb3a1-4e53-4dcf-9a57-3a1a44680aaa-metrics-certs\") pod \"maas-api-7f866d8d77-l5j55\" (UID: \"c7cfb3a1-4e53-4dcf-9a57-3a1a44680aaa\") " pod="odh-ai-gateway-infra/maas-api-7f866d8d77-l5j55" Sep 11 04:29:23.275168 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:23.275132 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7cfb3a1-4e53-4dcf-9a57-3a1a44680aaa-openshift-service-ca\") pod \"maas-api-7f866d8d77-l5j55\" (UID: \"c7cfb3a1-4e53-4dcf-9a57-3a1a44680aaa\") " pod="odh-ai-gateway-infra/maas-api-7f866d8d77-l5j55" Sep 11 04:29:23.275640 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:23.275195 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/424cc6cc-5f07-44fc-a7e0-4d4f73390623-metrics-certs\") pod \"maas-api-7ff8fbcd4f-ftx5g\" (UID: \"424cc6cc-5f07-44fc-a7e0-4d4f73390623\") " pod="odh-ai-gateway-infra/maas-api-7ff8fbcd4f-ftx5g" Sep 11 04:29:23.275640 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:23.275225 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7cfb3a1-4e53-4dcf-9a57-3a1a44680aaa-metrics-certs\") pod \"maas-api-7f866d8d77-l5j55\" (UID: \"c7cfb3a1-4e53-4dcf-9a57-3a1a44680aaa\") " pod="odh-ai-gateway-infra/maas-api-7f866d8d77-l5j55" Sep 11 04:29:23.275640 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:23.275246 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9tcx\" (UniqueName: \"kubernetes.io/projected/424cc6cc-5f07-44fc-a7e0-4d4f73390623-kube-api-access-q9tcx\") pod \"maas-api-7ff8fbcd4f-ftx5g\" (UID: \"424cc6cc-5f07-44fc-a7e0-4d4f73390623\") " pod="odh-ai-gateway-infra/maas-api-7ff8fbcd4f-ftx5g" Sep 11 04:29:23.275640 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:23.275388 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7cfb3a1-4e53-4dcf-9a57-3a1a44680aaa-maas-api-tls\") pod \"maas-api-7f866d8d77-l5j55\" (UID: \"c7cfb3a1-4e53-4dcf-9a57-3a1a44680aaa\") " pod="odh-ai-gateway-infra/maas-api-7f866d8d77-l5j55" Sep 11 04:29:23.275640 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:23.275426 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/424cc6cc-5f07-44fc-a7e0-4d4f73390623-maas-api-tls\") pod \"maas-api-7ff8fbcd4f-ftx5g\" (UID: \"424cc6cc-5f07-44fc-a7e0-4d4f73390623\") " pod="odh-ai-gateway-infra/maas-api-7ff8fbcd4f-ftx5g" Sep 11 04:29:23.275640 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:23.275463 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/424cc6cc-5f07-44fc-a7e0-4d4f73390623-openshift-service-ca\") pod \"maas-api-7ff8fbcd4f-ftx5g\" (UID: \"424cc6cc-5f07-44fc-a7e0-4d4f73390623\") " pod="odh-ai-gateway-infra/maas-api-7ff8fbcd4f-ftx5g" Sep 11 04:29:23.275640 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:23.275492 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttw57\" (UniqueName: \"kubernetes.io/projected/c7cfb3a1-4e53-4dcf-9a57-3a1a44680aaa-kube-api-access-ttw57\") pod \"maas-api-7f866d8d77-l5j55\" (UID: \"c7cfb3a1-4e53-4dcf-9a57-3a1a44680aaa\") " pod="odh-ai-gateway-infra/maas-api-7f866d8d77-l5j55" Sep 11 04:29:23.275985 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:23.275920 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7cfb3a1-4e53-4dcf-9a57-3a1a44680aaa-openshift-service-ca\") pod \"maas-api-7f866d8d77-l5j55\" (UID: \"c7cfb3a1-4e53-4dcf-9a57-3a1a44680aaa\") " pod="odh-ai-gateway-infra/maas-api-7f866d8d77-l5j55" Sep 11 04:29:23.277867 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:23.277842 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7cfb3a1-4e53-4dcf-9a57-3a1a44680aaa-maas-api-tls\") pod \"maas-api-7f866d8d77-l5j55\" (UID: \"c7cfb3a1-4e53-4dcf-9a57-3a1a44680aaa\") " pod="odh-ai-gateway-infra/maas-api-7f866d8d77-l5j55" Sep 11 04:29:23.277867 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:23.277853 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7cfb3a1-4e53-4dcf-9a57-3a1a44680aaa-metrics-certs\") pod \"maas-api-7f866d8d77-l5j55\" (UID: \"c7cfb3a1-4e53-4dcf-9a57-3a1a44680aaa\") " pod="odh-ai-gateway-infra/maas-api-7f866d8d77-l5j55" Sep 11 04:29:23.283519 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:23.283495 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttw57\" (UniqueName: \"kubernetes.io/projected/c7cfb3a1-4e53-4dcf-9a57-3a1a44680aaa-kube-api-access-ttw57\") pod \"maas-api-7f866d8d77-l5j55\" (UID: \"c7cfb3a1-4e53-4dcf-9a57-3a1a44680aaa\") " pod="odh-ai-gateway-infra/maas-api-7f866d8d77-l5j55" Sep 11 04:29:23.376510 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:23.376414 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/424cc6cc-5f07-44fc-a7e0-4d4f73390623-maas-api-tls\") pod \"maas-api-7ff8fbcd4f-ftx5g\" (UID: \"424cc6cc-5f07-44fc-a7e0-4d4f73390623\") " pod="odh-ai-gateway-infra/maas-api-7ff8fbcd4f-ftx5g" Sep 11 04:29:23.376510 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:23.376456 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/424cc6cc-5f07-44fc-a7e0-4d4f73390623-openshift-service-ca\") pod \"maas-api-7ff8fbcd4f-ftx5g\" (UID: \"424cc6cc-5f07-44fc-a7e0-4d4f73390623\") " pod="odh-ai-gateway-infra/maas-api-7ff8fbcd4f-ftx5g" Sep 11 04:29:23.376510 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:23.376505 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/424cc6cc-5f07-44fc-a7e0-4d4f73390623-metrics-certs\") pod \"maas-api-7ff8fbcd4f-ftx5g\" (UID: \"424cc6cc-5f07-44fc-a7e0-4d4f73390623\") " pod="odh-ai-gateway-infra/maas-api-7ff8fbcd4f-ftx5g" Sep 11 04:29:23.376803 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:23.376539 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9tcx\" (UniqueName: \"kubernetes.io/projected/424cc6cc-5f07-44fc-a7e0-4d4f73390623-kube-api-access-q9tcx\") pod \"maas-api-7ff8fbcd4f-ftx5g\" (UID: \"424cc6cc-5f07-44fc-a7e0-4d4f73390623\") " pod="odh-ai-gateway-infra/maas-api-7ff8fbcd4f-ftx5g" Sep 11 04:29:23.377193 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:23.377172 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/424cc6cc-5f07-44fc-a7e0-4d4f73390623-openshift-service-ca\") pod \"maas-api-7ff8fbcd4f-ftx5g\" (UID: \"424cc6cc-5f07-44fc-a7e0-4d4f73390623\") " pod="odh-ai-gateway-infra/maas-api-7ff8fbcd4f-ftx5g" Sep 11 04:29:23.379027 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:23.379003 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/424cc6cc-5f07-44fc-a7e0-4d4f73390623-maas-api-tls\") pod \"maas-api-7ff8fbcd4f-ftx5g\" (UID: \"424cc6cc-5f07-44fc-a7e0-4d4f73390623\") " pod="odh-ai-gateway-infra/maas-api-7ff8fbcd4f-ftx5g" Sep 11 04:29:23.379128 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:23.379059 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/424cc6cc-5f07-44fc-a7e0-4d4f73390623-metrics-certs\") pod \"maas-api-7ff8fbcd4f-ftx5g\" (UID: \"424cc6cc-5f07-44fc-a7e0-4d4f73390623\") " pod="odh-ai-gateway-infra/maas-api-7ff8fbcd4f-ftx5g" Sep 11 04:29:23.383573 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:23.383551 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9tcx\" (UniqueName: \"kubernetes.io/projected/424cc6cc-5f07-44fc-a7e0-4d4f73390623-kube-api-access-q9tcx\") pod \"maas-api-7ff8fbcd4f-ftx5g\" (UID: \"424cc6cc-5f07-44fc-a7e0-4d4f73390623\") " pod="odh-ai-gateway-infra/maas-api-7ff8fbcd4f-ftx5g" Sep 11 04:29:23.408865 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:23.408838 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7ff8fbcd4f-ftx5g" Sep 11 04:29:23.427774 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:23.427754 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f866d8d77-l5j55" Sep 11 04:29:23.679518 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:23.679491 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7ff8fbcd4f-ftx5g"] Sep 11 04:29:23.681450 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:29:23.681421 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod424cc6cc_5f07_44fc_a7e0_4d4f73390623.slice/crio-c0ce62fbe94116581d92a589846acce6c30c2be97010282771b76af96a9148b4 WatchSource:0}: Error finding container c0ce62fbe94116581d92a589846acce6c30c2be97010282771b76af96a9148b4: Status 404 returned error can't find the container with id c0ce62fbe94116581d92a589846acce6c30c2be97010282771b76af96a9148b4 Sep 11 04:29:23.682721 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:23.682702 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 04:29:23.707140 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:23.707113 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f866d8d77-l5j55"] Sep 11 04:29:23.709013 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:29:23.708988 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7cfb3a1_4e53_4dcf_9a57_3a1a44680aaa.slice/crio-835ba572ecbafe0902b676230783c86186cf72f3f5d86d0e16db96006c0ec394 WatchSource:0}: Error finding container 835ba572ecbafe0902b676230783c86186cf72f3f5d86d0e16db96006c0ec394: Status 404 returned error can't find the container with id 835ba572ecbafe0902b676230783c86186cf72f3f5d86d0e16db96006c0ec394 Sep 11 04:29:23.768219 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:23.768182 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58535024-911f-44b4-b1e0-d2a49f38a312" path="/var/lib/kubelet/pods/58535024-911f-44b4-b1e0-d2a49f38a312/volumes" Sep 11 04:29:24.275194 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:24.275159 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f866d8d77-l5j55" event={"ID":"c7cfb3a1-4e53-4dcf-9a57-3a1a44680aaa","Type":"ContainerStarted","Data":"e3cd5635b476b36ca93e279ef8309bedb18ee7fdcfc424d867c869ffe0780a25"} Sep 11 04:29:24.275642 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:24.275260 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7f866d8d77-l5j55" Sep 11 04:29:24.275642 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:24.275285 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f866d8d77-l5j55" event={"ID":"c7cfb3a1-4e53-4dcf-9a57-3a1a44680aaa","Type":"ContainerStarted","Data":"835ba572ecbafe0902b676230783c86186cf72f3f5d86d0e16db96006c0ec394"} Sep 11 04:29:24.276937 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:24.276908 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7ff8fbcd4f-ftx5g" event={"ID":"424cc6cc-5f07-44fc-a7e0-4d4f73390623","Type":"ContainerStarted","Data":"9a62c50c311df57ebbd2467ba22846aba10570cdfe189cf2f8a20deabb21f284"} Sep 11 04:29:24.277067 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:24.276952 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7ff8fbcd4f-ftx5g" event={"ID":"424cc6cc-5f07-44fc-a7e0-4d4f73390623","Type":"ContainerStarted","Data":"c0ce62fbe94116581d92a589846acce6c30c2be97010282771b76af96a9148b4"} Sep 11 04:29:24.277067 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:24.276987 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7ff8fbcd4f-ftx5g" Sep 11 04:29:24.292554 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:24.292500 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7f866d8d77-l5j55" podStartSLOduration=0.865728523 podStartE2EDuration="1.292480788s" totalImagesPullingTime="426.752265ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:29:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:29:24.290408533 +0000 UTC m=+2179.044039935" watchObservedRunningTime="2026-09-11 04:29:24.292480788 +0000 UTC m=+2179.046112191" Sep 11 04:29:24.306948 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:24.306908 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7ff8fbcd4f-ftx5g" podStartSLOduration=0.926390984 podStartE2EDuration="1.306894497s" totalImagesPullingTime="380.503513ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:29:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:29:24.305550629 +0000 UTC m=+2179.059182032" watchObservedRunningTime="2026-09-11 04:29:24.306894497 +0000 UTC m=+2179.060525918" Sep 11 04:29:30.287324 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:30.287289 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7f866d8d77-l5j55" Sep 11 04:29:30.287768 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:30.287670 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7ff8fbcd4f-ftx5g" Sep 11 04:29:30.350386 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:30.350355 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7ff8fbcd4f-ftx5g"] Sep 11 04:29:30.350570 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:30.350547 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7ff8fbcd4f-ftx5g" podUID="424cc6cc-5f07-44fc-a7e0-4d4f73390623" containerName="maas-api" containerID="cri-o://9a62c50c311df57ebbd2467ba22846aba10570cdfe189cf2f8a20deabb21f284" gracePeriod=30 Sep 11 04:29:30.600524 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:30.600500 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7ff8fbcd4f-ftx5g" Sep 11 04:29:30.641069 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:30.641020 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/424cc6cc-5f07-44fc-a7e0-4d4f73390623-openshift-service-ca\") pod \"424cc6cc-5f07-44fc-a7e0-4d4f73390623\" (UID: \"424cc6cc-5f07-44fc-a7e0-4d4f73390623\") " Sep 11 04:29:30.641231 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:30.641088 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/424cc6cc-5f07-44fc-a7e0-4d4f73390623-metrics-certs\") pod \"424cc6cc-5f07-44fc-a7e0-4d4f73390623\" (UID: \"424cc6cc-5f07-44fc-a7e0-4d4f73390623\") " Sep 11 04:29:30.641231 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:30.641117 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/424cc6cc-5f07-44fc-a7e0-4d4f73390623-maas-api-tls\") pod \"424cc6cc-5f07-44fc-a7e0-4d4f73390623\" (UID: \"424cc6cc-5f07-44fc-a7e0-4d4f73390623\") " Sep 11 04:29:30.641231 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:30.641194 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9tcx\" (UniqueName: \"kubernetes.io/projected/424cc6cc-5f07-44fc-a7e0-4d4f73390623-kube-api-access-q9tcx\") pod \"424cc6cc-5f07-44fc-a7e0-4d4f73390623\" (UID: \"424cc6cc-5f07-44fc-a7e0-4d4f73390623\") " Sep 11 04:29:30.641481 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:30.641451 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/424cc6cc-5f07-44fc-a7e0-4d4f73390623-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "424cc6cc-5f07-44fc-a7e0-4d4f73390623" (UID: "424cc6cc-5f07-44fc-a7e0-4d4f73390623"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 04:29:30.643576 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:30.643539 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/424cc6cc-5f07-44fc-a7e0-4d4f73390623-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "424cc6cc-5f07-44fc-a7e0-4d4f73390623" (UID: "424cc6cc-5f07-44fc-a7e0-4d4f73390623"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 04:29:30.643887 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:30.643865 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/424cc6cc-5f07-44fc-a7e0-4d4f73390623-kube-api-access-q9tcx" (OuterVolumeSpecName: "kube-api-access-q9tcx") pod "424cc6cc-5f07-44fc-a7e0-4d4f73390623" (UID: "424cc6cc-5f07-44fc-a7e0-4d4f73390623"). InnerVolumeSpecName "kube-api-access-q9tcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 04:29:30.643887 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:30.643870 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/424cc6cc-5f07-44fc-a7e0-4d4f73390623-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "424cc6cc-5f07-44fc-a7e0-4d4f73390623" (UID: "424cc6cc-5f07-44fc-a7e0-4d4f73390623"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 04:29:30.742221 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:30.742175 2568 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/424cc6cc-5f07-44fc-a7e0-4d4f73390623-metrics-certs\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:29:30.742221 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:30.742211 2568 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/424cc6cc-5f07-44fc-a7e0-4d4f73390623-maas-api-tls\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:29:30.742221 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:30.742225 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9tcx\" (UniqueName: \"kubernetes.io/projected/424cc6cc-5f07-44fc-a7e0-4d4f73390623-kube-api-access-q9tcx\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:29:30.742470 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:30.742238 2568 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/424cc6cc-5f07-44fc-a7e0-4d4f73390623-openshift-service-ca\") on node \"ip-10-0-140-30.ec2.internal\" DevicePath \"\"" Sep 11 04:29:31.311517 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:31.311478 2568 generic.go:356] "Generic (PLEG): container finished" podID="424cc6cc-5f07-44fc-a7e0-4d4f73390623" containerID="9a62c50c311df57ebbd2467ba22846aba10570cdfe189cf2f8a20deabb21f284" exitCode=0 Sep 11 04:29:31.312000 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:31.311535 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7ff8fbcd4f-ftx5g" Sep 11 04:29:31.312000 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:31.311566 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7ff8fbcd4f-ftx5g" event={"ID":"424cc6cc-5f07-44fc-a7e0-4d4f73390623","Type":"ContainerDied","Data":"9a62c50c311df57ebbd2467ba22846aba10570cdfe189cf2f8a20deabb21f284"} Sep 11 04:29:31.312000 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:31.311645 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7ff8fbcd4f-ftx5g" event={"ID":"424cc6cc-5f07-44fc-a7e0-4d4f73390623","Type":"ContainerDied","Data":"c0ce62fbe94116581d92a589846acce6c30c2be97010282771b76af96a9148b4"} Sep 11 04:29:31.312000 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:31.311669 2568 scope.go:117] "RemoveContainer" containerID="9a62c50c311df57ebbd2467ba22846aba10570cdfe189cf2f8a20deabb21f284" Sep 11 04:29:31.320904 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:31.320885 2568 scope.go:117] "RemoveContainer" containerID="9a62c50c311df57ebbd2467ba22846aba10570cdfe189cf2f8a20deabb21f284" Sep 11 04:29:31.321184 ip-10-0-140-30 kubenswrapper[2568]: E0911 04:29:31.321164 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a62c50c311df57ebbd2467ba22846aba10570cdfe189cf2f8a20deabb21f284\": container with ID starting with 9a62c50c311df57ebbd2467ba22846aba10570cdfe189cf2f8a20deabb21f284 not found: ID does not exist" containerID="9a62c50c311df57ebbd2467ba22846aba10570cdfe189cf2f8a20deabb21f284" Sep 11 04:29:31.321239 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:31.321192 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a62c50c311df57ebbd2467ba22846aba10570cdfe189cf2f8a20deabb21f284"} err="failed to get container status \"9a62c50c311df57ebbd2467ba22846aba10570cdfe189cf2f8a20deabb21f284\": rpc error: code = NotFound desc = could not find container \"9a62c50c311df57ebbd2467ba22846aba10570cdfe189cf2f8a20deabb21f284\": container with ID starting with 9a62c50c311df57ebbd2467ba22846aba10570cdfe189cf2f8a20deabb21f284 not found: ID does not exist" Sep 11 04:29:31.336331 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:31.336312 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7ff8fbcd4f-ftx5g"] Sep 11 04:29:31.341716 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:31.341692 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7ff8fbcd4f-ftx5g"] Sep 11 04:29:31.768041 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:31.768009 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="424cc6cc-5f07-44fc-a7e0-4d4f73390623" path="/var/lib/kubelet/pods/424cc6cc-5f07-44fc-a7e0-4d4f73390623/volumes" Sep 11 04:29:53.135751 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:53.135711 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6b59f59dfb-ntnj2_861f1c75-8df2-4d2a-97ac-ae57e5478404/authorino/0.log" Sep 11 04:29:58.688903 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:58.688865 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-plffv_586a64b0-ea72-407e-8cac-fb88d47f146d/manager/0.log" Sep 11 04:29:58.816024 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:58.815987 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-mxmqw_2970cd01-adab-426f-9dbb-96fc7affff82/manager/0.log" Sep 11 04:29:59.080964 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:59.080887 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-v24qv_b0f7ac31-8a89-46b9-b805-4c19ecc6eb8f/server/0.log" Sep 11 04:29:59.204831 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:59.204797 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-f6fj4_be71696e-0b7d-4584-9761-4d5a7cf3bd5f/manager/0.log" Sep 11 04:29:59.319200 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:59.319176 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-65b869c898-d5nnh_786a4185-ee93-4ab3-9b70-a64bfd995e1b/copy-manifests/0.log" Sep 11 04:29:59.337361 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:29:59.337287 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-65b869c898-d5nnh_786a4185-ee93-4ab3-9b70-a64bfd995e1b/manager/0.log" Sep 11 04:30:00.060188 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:00.060158 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7f866d8d77-l5j55_c7cfb3a1-4e53-4dcf-9a57-3a1a44680aaa/maas-api/0.log" Sep 11 04:30:01.086000 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:01.085967 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6b59f59dfb-ntnj2_861f1c75-8df2-4d2a-97ac-ae57e5478404/authorino/0.log" Sep 11 04:30:01.431333 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:01.431307 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-ddlnd_413cad97-0870-48a3-a7ff-07bba8820ce5/kuadrant-console-plugin/0.log" Sep 11 04:30:01.554195 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:01.554165 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-hsqnh_dcca20bf-0991-4d77-baff-cea0bb250831/manager/0.log" Sep 11 04:30:01.792232 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:01.792149 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-46rbd_b5b8e568-1d15-43e6-a1fb-44ba5128d5c0/manager/0.log" Sep 11 04:30:02.901212 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:02.901178 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-7v7hl_879b00c8-d09a-4fc0-a80d-30261c01c32a/payload-processing/0.log" Sep 11 04:30:06.170488 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:06.170453 2568 scope.go:117] "RemoveContainer" containerID="06feb1e9343b0a1889a2486d999c535175ecea63beb62ba2208007fbcecdf32d" Sep 11 04:30:10.135387 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:10.135354 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dxk5v/must-gather-g9hdp"] Sep 11 04:30:10.135769 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:10.135684 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="424cc6cc-5f07-44fc-a7e0-4d4f73390623" containerName="maas-api" Sep 11 04:30:10.135769 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:10.135696 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="424cc6cc-5f07-44fc-a7e0-4d4f73390623" containerName="maas-api" Sep 11 04:30:10.135769 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:10.135760 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="424cc6cc-5f07-44fc-a7e0-4d4f73390623" containerName="maas-api" Sep 11 04:30:10.140408 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:10.140388 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dxk5v/must-gather-g9hdp" Sep 11 04:30:10.144106 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:10.144076 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dxk5v\"/\"openshift-service-ca.crt\"" Sep 11 04:30:10.144249 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:10.144218 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dxk5v\"/\"default-dockercfg-gxcvx\"" Sep 11 04:30:10.145004 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:10.144985 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dxk5v\"/\"kube-root-ca.crt\"" Sep 11 04:30:10.149364 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:10.149340 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dxk5v/must-gather-g9hdp"] Sep 11 04:30:10.157814 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:10.157788 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:30:10.265243 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:10.265211 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn2m8\" (UniqueName: \"kubernetes.io/projected/23172446-9283-4ae0-957e-c86dd9a5b0be-kube-api-access-vn2m8\") pod \"must-gather-g9hdp\" (UID: \"23172446-9283-4ae0-957e-c86dd9a5b0be\") " pod="openshift-must-gather-dxk5v/must-gather-g9hdp" Sep 11 04:30:10.265424 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:10.265336 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/23172446-9283-4ae0-957e-c86dd9a5b0be-must-gather-output\") pod \"must-gather-g9hdp\" (UID: \"23172446-9283-4ae0-957e-c86dd9a5b0be\") " pod="openshift-must-gather-dxk5v/must-gather-g9hdp" Sep 11 04:30:10.366451 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:10.366406 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/23172446-9283-4ae0-957e-c86dd9a5b0be-must-gather-output\") pod \"must-gather-g9hdp\" (UID: \"23172446-9283-4ae0-957e-c86dd9a5b0be\") " pod="openshift-must-gather-dxk5v/must-gather-g9hdp" Sep 11 04:30:10.366674 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:10.366464 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn2m8\" (UniqueName: \"kubernetes.io/projected/23172446-9283-4ae0-957e-c86dd9a5b0be-kube-api-access-vn2m8\") pod \"must-gather-g9hdp\" (UID: \"23172446-9283-4ae0-957e-c86dd9a5b0be\") " pod="openshift-must-gather-dxk5v/must-gather-g9hdp" Sep 11 04:30:10.366839 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:10.366815 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/23172446-9283-4ae0-957e-c86dd9a5b0be-must-gather-output\") pod \"must-gather-g9hdp\" (UID: \"23172446-9283-4ae0-957e-c86dd9a5b0be\") " pod="openshift-must-gather-dxk5v/must-gather-g9hdp" Sep 11 04:30:10.376893 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:10.376867 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn2m8\" (UniqueName: \"kubernetes.io/projected/23172446-9283-4ae0-957e-c86dd9a5b0be-kube-api-access-vn2m8\") pod \"must-gather-g9hdp\" (UID: \"23172446-9283-4ae0-957e-c86dd9a5b0be\") " pod="openshift-must-gather-dxk5v/must-gather-g9hdp" Sep 11 04:30:10.451094 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:10.451067 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dxk5v/must-gather-g9hdp" Sep 11 04:30:10.726165 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:10.726140 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dxk5v/must-gather-g9hdp"] Sep 11 04:30:10.731480 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:30:10.731451 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23172446_9283_4ae0_957e_c86dd9a5b0be.slice/crio-55f9e38bead441c7d26ece4ee8c6c72cc453e00d76311136cd01f89796bedae7 WatchSource:0}: Error finding container 55f9e38bead441c7d26ece4ee8c6c72cc453e00d76311136cd01f89796bedae7: Status 404 returned error can't find the container with id 55f9e38bead441c7d26ece4ee8c6c72cc453e00d76311136cd01f89796bedae7 Sep 11 04:30:11.278948 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:11.278909 2568 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" limit=1024 Sep 11 04:30:11.496832 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:11.496793 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dxk5v/must-gather-g9hdp" event={"ID":"23172446-9283-4ae0-957e-c86dd9a5b0be","Type":"ContainerStarted","Data":"86983e293cd21478f70d90747e4b56bc7b60d08a31f98ef4952dc5a0068adafa"} Sep 11 04:30:11.496981 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:11.496837 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dxk5v/must-gather-g9hdp" event={"ID":"23172446-9283-4ae0-957e-c86dd9a5b0be","Type":"ContainerStarted","Data":"55f9e38bead441c7d26ece4ee8c6c72cc453e00d76311136cd01f89796bedae7"} Sep 11 04:30:12.503452 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:12.503417 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dxk5v/must-gather-g9hdp" event={"ID":"23172446-9283-4ae0-957e-c86dd9a5b0be","Type":"ContainerStarted","Data":"28a2d4044f0e100d63ae41d44047a4b1496c1c6fe26d831d93ffc464a987c35c"} Sep 11 04:30:12.523378 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:12.523331 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dxk5v/must-gather-g9hdp" podStartSLOduration=1.928240974 podStartE2EDuration="2.523314473s" totalImagesPullingTime="595.073499ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 04:30:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:30:12.521369628 +0000 UTC m=+2227.275001031" watchObservedRunningTime="2026-09-11 04:30:12.523314473 +0000 UTC m=+2227.276945874" Sep 11 04:30:13.051151 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:13.051111 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-t5fb2_9fc48d3a-bf43-44f7-b142-cd4937cea66b/global-pull-secret-syncer/0.log" Sep 11 04:30:13.113141 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:13.113106 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-c5drs_aa66b45c-400f-46fa-b216-0e063413439b/konnectivity-agent/0.log" Sep 11 04:30:13.257658 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:13.257609 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-30.ec2.internal_f90f3e1f70d21477b755e9266cecc76b/haproxy/0.log" Sep 11 04:30:17.599068 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:17.598409 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6b59f59dfb-ntnj2_861f1c75-8df2-4d2a-97ac-ae57e5478404/authorino/0.log" Sep 11 04:30:17.693451 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:17.693422 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-ddlnd_413cad97-0870-48a3-a7ff-07bba8820ce5/kuadrant-console-plugin/0.log" Sep 11 04:30:17.756974 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:17.756937 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-hsqnh_dcca20bf-0991-4d77-baff-cea0bb250831/manager/0.log" Sep 11 04:30:17.871518 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:17.871429 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-46rbd_b5b8e568-1d15-43e6-a1fb-44ba5128d5c0/manager/0.log" Sep 11 04:30:19.785557 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:19.785516 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kdmjj_86ba6125-3e14-4a45-ba63-dc428059e6ea/node-exporter/0.log" Sep 11 04:30:19.811301 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:19.811265 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kdmjj_86ba6125-3e14-4a45-ba63-dc428059e6ea/kube-rbac-proxy/0.log" Sep 11 04:30:19.839726 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:19.839692 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kdmjj_86ba6125-3e14-4a45-ba63-dc428059e6ea/init-textfile/0.log" Sep 11 04:30:22.735260 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:22.735230 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-5tpdc_9112557d-d0ef-44c7-8b2a-f6ecd0a980d6/console-operator/1.log" Sep 11 04:30:22.740295 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:22.740249 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-5tpdc_9112557d-d0ef-44c7-8b2a-f6ecd0a980d6/console-operator/2.log" Sep 11 04:30:23.173022 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:23.172979 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dxk5v/perf-node-gather-daemonset-qlgcg"] Sep 11 04:30:23.181206 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:23.181172 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dxk5v/perf-node-gather-daemonset-qlgcg" Sep 11 04:30:23.193850 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:23.193812 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dxk5v/perf-node-gather-daemonset-qlgcg"] Sep 11 04:30:23.200190 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:23.200142 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bf03c6a7-82ac-446b-b60f-b9f6d2d2f4b5-lib-modules\") pod \"perf-node-gather-daemonset-qlgcg\" (UID: \"bf03c6a7-82ac-446b-b60f-b9f6d2d2f4b5\") " pod="openshift-must-gather-dxk5v/perf-node-gather-daemonset-qlgcg" Sep 11 04:30:23.200359 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:23.200217 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bf03c6a7-82ac-446b-b60f-b9f6d2d2f4b5-proc\") pod \"perf-node-gather-daemonset-qlgcg\" (UID: \"bf03c6a7-82ac-446b-b60f-b9f6d2d2f4b5\") " pod="openshift-must-gather-dxk5v/perf-node-gather-daemonset-qlgcg" Sep 11 04:30:23.200359 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:23.200260 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bf03c6a7-82ac-446b-b60f-b9f6d2d2f4b5-podres\") pod \"perf-node-gather-daemonset-qlgcg\" (UID: \"bf03c6a7-82ac-446b-b60f-b9f6d2d2f4b5\") " pod="openshift-must-gather-dxk5v/perf-node-gather-daemonset-qlgcg" Sep 11 04:30:23.200359 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:23.200318 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-846qq\" (UniqueName: \"kubernetes.io/projected/bf03c6a7-82ac-446b-b60f-b9f6d2d2f4b5-kube-api-access-846qq\") pod \"perf-node-gather-daemonset-qlgcg\" (UID: \"bf03c6a7-82ac-446b-b60f-b9f6d2d2f4b5\") " pod="openshift-must-gather-dxk5v/perf-node-gather-daemonset-qlgcg" Sep 11 04:30:23.200359 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:23.200351 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf03c6a7-82ac-446b-b60f-b9f6d2d2f4b5-sys\") pod \"perf-node-gather-daemonset-qlgcg\" (UID: \"bf03c6a7-82ac-446b-b60f-b9f6d2d2f4b5\") " pod="openshift-must-gather-dxk5v/perf-node-gather-daemonset-qlgcg" Sep 11 04:30:23.301845 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:23.301794 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-846qq\" (UniqueName: \"kubernetes.io/projected/bf03c6a7-82ac-446b-b60f-b9f6d2d2f4b5-kube-api-access-846qq\") pod \"perf-node-gather-daemonset-qlgcg\" (UID: \"bf03c6a7-82ac-446b-b60f-b9f6d2d2f4b5\") " pod="openshift-must-gather-dxk5v/perf-node-gather-daemonset-qlgcg" Sep 11 04:30:23.301845 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:23.301841 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf03c6a7-82ac-446b-b60f-b9f6d2d2f4b5-sys\") pod \"perf-node-gather-daemonset-qlgcg\" (UID: \"bf03c6a7-82ac-446b-b60f-b9f6d2d2f4b5\") " pod="openshift-must-gather-dxk5v/perf-node-gather-daemonset-qlgcg" Sep 11 04:30:23.302131 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:23.301889 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bf03c6a7-82ac-446b-b60f-b9f6d2d2f4b5-lib-modules\") pod \"perf-node-gather-daemonset-qlgcg\" (UID: \"bf03c6a7-82ac-446b-b60f-b9f6d2d2f4b5\") " pod="openshift-must-gather-dxk5v/perf-node-gather-daemonset-qlgcg" Sep 11 04:30:23.302131 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:23.301924 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bf03c6a7-82ac-446b-b60f-b9f6d2d2f4b5-proc\") pod \"perf-node-gather-daemonset-qlgcg\" (UID: \"bf03c6a7-82ac-446b-b60f-b9f6d2d2f4b5\") " pod="openshift-must-gather-dxk5v/perf-node-gather-daemonset-qlgcg" Sep 11 04:30:23.302131 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:23.302004 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bf03c6a7-82ac-446b-b60f-b9f6d2d2f4b5-proc\") pod \"perf-node-gather-daemonset-qlgcg\" (UID: \"bf03c6a7-82ac-446b-b60f-b9f6d2d2f4b5\") " pod="openshift-must-gather-dxk5v/perf-node-gather-daemonset-qlgcg" Sep 11 04:30:23.302131 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:23.302017 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bf03c6a7-82ac-446b-b60f-b9f6d2d2f4b5-sys\") pod \"perf-node-gather-daemonset-qlgcg\" (UID: \"bf03c6a7-82ac-446b-b60f-b9f6d2d2f4b5\") " pod="openshift-must-gather-dxk5v/perf-node-gather-daemonset-qlgcg" Sep 11 04:30:23.302131 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:23.302024 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bf03c6a7-82ac-446b-b60f-b9f6d2d2f4b5-podres\") pod \"perf-node-gather-daemonset-qlgcg\" (UID: \"bf03c6a7-82ac-446b-b60f-b9f6d2d2f4b5\") " pod="openshift-must-gather-dxk5v/perf-node-gather-daemonset-qlgcg" Sep 11 04:30:23.302131 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:23.302095 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bf03c6a7-82ac-446b-b60f-b9f6d2d2f4b5-lib-modules\") pod \"perf-node-gather-daemonset-qlgcg\" (UID: \"bf03c6a7-82ac-446b-b60f-b9f6d2d2f4b5\") " pod="openshift-must-gather-dxk5v/perf-node-gather-daemonset-qlgcg" Sep 11 04:30:23.302476 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:23.302146 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bf03c6a7-82ac-446b-b60f-b9f6d2d2f4b5-podres\") pod \"perf-node-gather-daemonset-qlgcg\" (UID: \"bf03c6a7-82ac-446b-b60f-b9f6d2d2f4b5\") " pod="openshift-must-gather-dxk5v/perf-node-gather-daemonset-qlgcg" Sep 11 04:30:23.311123 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:23.311088 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-846qq\" (UniqueName: \"kubernetes.io/projected/bf03c6a7-82ac-446b-b60f-b9f6d2d2f4b5-kube-api-access-846qq\") pod \"perf-node-gather-daemonset-qlgcg\" (UID: \"bf03c6a7-82ac-446b-b60f-b9f6d2d2f4b5\") " pod="openshift-must-gather-dxk5v/perf-node-gather-daemonset-qlgcg" Sep 11 04:30:23.513636 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:23.513539 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dxk5v/perf-node-gather-daemonset-qlgcg" Sep 11 04:30:23.798313 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:23.795893 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dxk5v/perf-node-gather-daemonset-qlgcg"] Sep 11 04:30:23.800103 ip-10-0-140-30 kubenswrapper[2568]: W0911 04:30:23.799694 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbf03c6a7_82ac_446b_b60f_b9f6d2d2f4b5.slice/crio-77b9f2017d2b1dad185f7a236cad5131cb3555033d280c70861b2771855370b6 WatchSource:0}: Error finding container 77b9f2017d2b1dad185f7a236cad5131cb3555033d280c70861b2771855370b6: Status 404 returned error can't find the container with id 77b9f2017d2b1dad185f7a236cad5131cb3555033d280c70861b2771855370b6 Sep 11 04:30:24.068363 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:24.068286 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-gqfwl_03b84729-9693-4100-b064-2cfb5e385273/volume-data-source-validator/0.log" Sep 11 04:30:24.588660 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:24.588615 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dxk5v/perf-node-gather-daemonset-qlgcg" event={"ID":"bf03c6a7-82ac-446b-b60f-b9f6d2d2f4b5","Type":"ContainerStarted","Data":"92e1684adce51fb1307278dce50f92413081691d4b9c77df55e5885dddcf6baf"} Sep 11 04:30:24.588858 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:24.588681 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dxk5v/perf-node-gather-daemonset-qlgcg" event={"ID":"bf03c6a7-82ac-446b-b60f-b9f6d2d2f4b5","Type":"ContainerStarted","Data":"77b9f2017d2b1dad185f7a236cad5131cb3555033d280c70861b2771855370b6"} Sep 11 04:30:24.588858 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:24.588837 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dxk5v/perf-node-gather-daemonset-qlgcg" Sep 11 04:30:24.606608 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:24.606542 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dxk5v/perf-node-gather-daemonset-qlgcg" podStartSLOduration=1.606522542 podStartE2EDuration="1.606522542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 04:30:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 04:30:24.605024788 +0000 UTC m=+2239.358656191" watchObservedRunningTime="2026-09-11 04:30:24.606522542 +0000 UTC m=+2239.360153945" Sep 11 04:30:25.330714 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:25.330684 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ssrwd_11fca7d2-d884-4012-9431-b3c085b8339c/dns/0.log" Sep 11 04:30:25.354037 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:25.354000 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ssrwd_11fca7d2-d884-4012-9431-b3c085b8339c/kube-rbac-proxy/0.log" Sep 11 04:30:25.427754 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:25.427727 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7f284_a48e2404-e37c-43d5-8902-b139cb2e07d9/dns-node-resolver/0.log" Sep 11 04:30:26.030886 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:26.030857 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gmdt4_db235f27-7ffe-4c12-9d60-53fb8b810d97/node-ca/0.log" Sep 11 04:30:27.410111 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:27.410078 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-7v7hl_879b00c8-d09a-4fc0-a80d-30261c01c32a/payload-processing/0.log" Sep 11 04:30:28.322452 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:28.322346 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nq6qk_badea905-61e6-47e1-8f68-505d04824821/serve-healthcheck-canary/0.log" Sep 11 04:30:28.835424 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:28.835364 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-4w9k5_7be9d187-475e-4a85-afc1-2f385d5a262e/insights-operator/0.log" Sep 11 04:30:28.836588 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:28.836563 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5844b856db-4w9k5_7be9d187-475e-4a85-afc1-2f385d5a262e/insights-operator/1.log" Sep 11 04:30:28.934747 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:28.934717 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-phn2f_c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f/kube-rbac-proxy/0.log" Sep 11 04:30:28.957010 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:28.956983 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-phn2f_c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f/exporter/0.log" Sep 11 04:30:28.982550 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:28.982512 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-phn2f_c6f6c1ac-998f-40f9-a13a-8da25c5e6c0f/extractor/0.log" Sep 11 04:30:30.608408 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:30.608368 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dxk5v/perf-node-gather-daemonset-qlgcg" Sep 11 04:30:31.176027 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:31.175995 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-plffv_586a64b0-ea72-407e-8cac-fb88d47f146d/manager/0.log" Sep 11 04:30:31.508291 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:31.507416 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-mxmqw_2970cd01-adab-426f-9dbb-96fc7affff82/manager/0.log" Sep 11 04:30:31.580598 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:31.580562 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-v24qv_b0f7ac31-8a89-46b9-b805-4c19ecc6eb8f/server/0.log" Sep 11 04:30:31.719558 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:31.719522 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-f6fj4_be71696e-0b7d-4584-9761-4d5a7cf3bd5f/manager/0.log" Sep 11 04:30:31.861821 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:31.859737 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-65b869c898-d5nnh_786a4185-ee93-4ab3-9b70-a64bfd995e1b/manager/0.log" Sep 11 04:30:31.879441 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:31.879415 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-65b869c898-d5nnh_786a4185-ee93-4ab3-9b70-a64bfd995e1b/copy-manifests/0.log" Sep 11 04:30:33.494680 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:33.494649 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5f8f45d65c-cn2gs_b0f71181-cd15-4756-bd67-94d3334dc883/manager/0.log" Sep 11 04:30:33.957747 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:33.957718 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-7v7hl_879b00c8-d09a-4fc0-a80d-30261c01c32a/payload-processing/0.log" Sep 11 04:30:35.808899 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:35.808848 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-7v7hl_879b00c8-d09a-4fc0-a80d-30261c01c32a/payload-processing/0.log" Sep 11 04:30:38.519801 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:38.519763 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-z28cw_83e53968-6fe7-4c74-a859-61fd72e01de4/migrator/0.log" Sep 11 04:30:38.542454 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:38.542421 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-z28cw_83e53968-6fe7-4c74-a859-61fd72e01de4/graceful-termination/0.log" Sep 11 04:30:38.897133 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:38.897024 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-b67kh_320f0913-5ebe-45b4-a60d-8359a83bd24b/kube-storage-version-migrator-operator/1.log" Sep 11 04:30:38.898925 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:38.898894 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-b67kh_320f0913-5ebe-45b4-a60d-8359a83bd24b/kube-storage-version-migrator-operator/0.log" Sep 11 04:30:40.139779 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:40.139745 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9r65j_d6906bf8-7095-4f6a-8969-d6f28167548b/kube-multus-additional-cni-plugins/0.log" Sep 11 04:30:40.164865 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:40.164831 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9r65j_d6906bf8-7095-4f6a-8969-d6f28167548b/egress-router-binary-copy/0.log" Sep 11 04:30:40.208260 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:40.208233 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9r65j_d6906bf8-7095-4f6a-8969-d6f28167548b/cni-plugins/0.log" Sep 11 04:30:40.247167 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:40.247133 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9r65j_d6906bf8-7095-4f6a-8969-d6f28167548b/bond-cni-plugin/0.log" Sep 11 04:30:40.270476 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:40.270436 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9r65j_d6906bf8-7095-4f6a-8969-d6f28167548b/routeoverride-cni/0.log" Sep 11 04:30:40.296030 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:40.295999 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9r65j_d6906bf8-7095-4f6a-8969-d6f28167548b/whereabouts-cni-bincopy/0.log" Sep 11 04:30:40.318184 ip-10-0-140-30 kubenswrapper[2568]: I0911 04:30:40.318159 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9r65j_d6906bf8-7095-4f6a-8969-d6f28167548b/whereabouts-cni/0.log"