Sep 16 07:33:48.316389 ip-10-0-142-170 systemd[1]: Starting Kubernetes Kubelet... Sep 16 07:33:48.706300 ip-10-0-142-170 kubenswrapper[2552]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 07:33:48.706300 ip-10-0-142-170 kubenswrapper[2552]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 16 07:33:48.706300 ip-10-0-142-170 kubenswrapper[2552]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 07:33:48.706300 ip-10-0-142-170 kubenswrapper[2552]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 16 07:33:48.706300 ip-10-0-142-170 kubenswrapper[2552]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 16 07:33:48.708431 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.708349 2552 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 16 07:33:48.713269 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713254 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 07:33:48.713269 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713268 2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 07:33:48.713337 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713273 2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 07:33:48.713337 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713276 2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 07:33:48.713337 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713281 2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 07:33:48.713337 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713284 2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 07:33:48.713337 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713287 2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 07:33:48.713337 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713291 2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 07:33:48.713337 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713294 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 07:33:48.713337 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713298 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 07:33:48.713337 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713301 2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 07:33:48.713337 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713303 2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 07:33:48.713337 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713306 2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 07:33:48.713337 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713309 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 07:33:48.713337 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713312 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 07:33:48.713337 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713314 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 07:33:48.713337 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713317 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 07:33:48.713337 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713320 2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 07:33:48.713337 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713322 2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 07:33:48.713337 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713325 2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 07:33:48.713337 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713328 2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 07:33:48.713790 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713334 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 07:33:48.713790 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713337 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 07:33:48.713790 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713341 2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 07:33:48.713790 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713343 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 07:33:48.713790 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713346 2552 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 07:33:48.713790 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713349 2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 07:33:48.713790 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713352 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 07:33:48.713790 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713355 2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 07:33:48.713790 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713357 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 07:33:48.713790 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713360 2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 07:33:48.713790 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713362 2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 07:33:48.713790 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713365 2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 07:33:48.713790 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713367 2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 07:33:48.713790 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713370 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 07:33:48.713790 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713372 2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 07:33:48.713790 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713375 2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 07:33:48.713790 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713377 2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 07:33:48.713790 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713386 2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 07:33:48.713790 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713389 2552 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 07:33:48.713790 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713392 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 07:33:48.714288 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713394 2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 07:33:48.714288 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713397 2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 07:33:48.714288 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713402 2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 07:33:48.714288 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713405 2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 07:33:48.714288 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713409 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 07:33:48.714288 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713412 2552 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 07:33:48.714288 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713415 2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 07:33:48.714288 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713418 2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 07:33:48.714288 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713421 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 07:33:48.714288 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713423 2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 07:33:48.714288 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713427 2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 07:33:48.714288 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713430 2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 07:33:48.714288 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713435 2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 07:33:48.714288 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713439 2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 07:33:48.714288 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713444 2552 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 07:33:48.714288 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713448 2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 07:33:48.714288 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713451 2552 feature_gate.go:328] unrecognized feature gate: Example Sep 16 07:33:48.714288 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713453 2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 07:33:48.714288 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713456 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 07:33:48.714760 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713459 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 07:33:48.714760 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713461 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 07:33:48.714760 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713464 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 07:33:48.714760 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713466 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 07:33:48.714760 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713469 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 07:33:48.714760 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713472 2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 07:33:48.714760 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713474 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 07:33:48.714760 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713477 2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 07:33:48.714760 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713479 2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 07:33:48.714760 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713482 2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 07:33:48.714760 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713484 2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 07:33:48.714760 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713487 2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 07:33:48.714760 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713489 2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 07:33:48.714760 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713492 2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 07:33:48.714760 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713495 2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 07:33:48.714760 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713497 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 07:33:48.714760 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713500 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 07:33:48.714760 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713503 2552 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 07:33:48.714760 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713506 2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 07:33:48.714760 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713508 2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 07:33:48.715264 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713511 2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 07:33:48.715264 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713513 2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 07:33:48.715264 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713516 2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 07:33:48.715264 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713519 2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 07:33:48.715264 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713522 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 07:33:48.715264 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713525 2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 07:33:48.715264 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713527 2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 07:33:48.715264 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713530 2552 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 07:33:48.715264 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713533 2552 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 07:33:48.715264 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713536 2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 07:33:48.715264 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713539 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 07:33:48.715264 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713568 2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 07:33:48.715264 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713571 2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 07:33:48.715264 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713575 2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 07:33:48.715264 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713578 2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 07:33:48.715264 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713581 2552 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 07:33:48.715264 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713584 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 07:33:48.715264 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713586 2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 07:33:48.715264 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.713590 2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 07:33:48.715264 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714254 2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 07:33:48.715754 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714260 2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 07:33:48.715754 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714264 2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 07:33:48.715754 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714266 2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 07:33:48.715754 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714269 2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 07:33:48.715754 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714271 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 07:33:48.715754 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714274 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 07:33:48.715754 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714277 2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 07:33:48.715754 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714280 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 07:33:48.715754 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714283 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 07:33:48.715754 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714286 2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 07:33:48.715754 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714288 2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 07:33:48.715754 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714292 2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 07:33:48.715754 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714295 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 07:33:48.715754 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714297 2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 07:33:48.715754 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714300 2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 07:33:48.715754 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714303 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 07:33:48.715754 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714305 2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 07:33:48.715754 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714309 2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 07:33:48.715754 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714312 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 07:33:48.715754 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714315 2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 07:33:48.716256 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714318 2552 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 07:33:48.716256 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714323 2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 07:33:48.716256 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714327 2552 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 07:33:48.716256 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714330 2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 07:33:48.716256 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714333 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 07:33:48.716256 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714336 2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 07:33:48.716256 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714338 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 07:33:48.716256 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714341 2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 07:33:48.716256 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714343 2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 07:33:48.716256 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714346 2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 07:33:48.716256 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714350 2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 07:33:48.716256 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714353 2552 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 07:33:48.716256 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714356 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 07:33:48.716256 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714359 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 07:33:48.716256 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714362 2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 07:33:48.716256 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714365 2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 07:33:48.716256 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714367 2552 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 07:33:48.716256 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714370 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 07:33:48.716256 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714372 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 07:33:48.716727 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714375 2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 07:33:48.716727 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714378 2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 07:33:48.716727 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714380 2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 07:33:48.716727 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714383 2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 07:33:48.716727 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714386 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 07:33:48.716727 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714389 2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 07:33:48.716727 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714391 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 07:33:48.716727 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714394 2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 07:33:48.716727 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714396 2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 07:33:48.716727 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714400 2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 07:33:48.716727 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714403 2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 07:33:48.716727 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714406 2552 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 07:33:48.716727 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714408 2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 07:33:48.716727 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714411 2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 07:33:48.716727 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714418 2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 07:33:48.716727 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714421 2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 07:33:48.716727 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714423 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 07:33:48.716727 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714426 2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 07:33:48.716727 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714428 2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 07:33:48.717206 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714433 2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 07:33:48.717206 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714436 2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 07:33:48.717206 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714439 2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 07:33:48.717206 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714442 2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 07:33:48.717206 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714445 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 07:33:48.717206 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714448 2552 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 07:33:48.717206 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714451 2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 07:33:48.717206 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714454 2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 07:33:48.717206 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714457 2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 07:33:48.717206 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714459 2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 07:33:48.717206 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714462 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 07:33:48.717206 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714464 2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 07:33:48.717206 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714467 2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 07:33:48.717206 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714469 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 07:33:48.717206 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714472 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 07:33:48.717206 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714475 2552 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 07:33:48.717206 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714478 2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 07:33:48.717206 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714482 2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 07:33:48.717206 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714485 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 07:33:48.717665 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714487 2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 07:33:48.717665 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714490 2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 07:33:48.717665 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714493 2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 07:33:48.717665 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714496 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 07:33:48.717665 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714499 2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 07:33:48.717665 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714501 2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 07:33:48.717665 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714504 2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 07:33:48.717665 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714506 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 07:33:48.717665 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714509 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 07:33:48.717665 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714512 2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 07:33:48.717665 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714515 2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 07:33:48.717665 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714517 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 07:33:48.717665 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714520 2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 07:33:48.717665 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714522 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 07:33:48.717665 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714525 2552 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 07:33:48.717665 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714527 2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 07:33:48.717665 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714530 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 07:33:48.717665 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714533 2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 07:33:48.717665 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714535 2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 07:33:48.717665 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714538 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 07:33:48.718170 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714540 2552 feature_gate.go:328] unrecognized feature gate: Example Sep 16 07:33:48.718170 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714897 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 07:33:48.718170 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714902 2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 07:33:48.718170 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714905 2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 07:33:48.718170 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714908 2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 07:33:48.718170 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714911 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 07:33:48.718170 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714914 2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 07:33:48.718170 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714917 2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 07:33:48.718170 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714919 2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 07:33:48.718170 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714922 2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 07:33:48.718170 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714925 2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 07:33:48.718170 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714928 2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 07:33:48.718170 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714931 2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 07:33:48.718170 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714933 2552 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 07:33:48.718170 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714936 2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 07:33:48.718170 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714938 2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 07:33:48.718170 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714941 2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 07:33:48.718170 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714944 2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 07:33:48.718170 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714947 2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 07:33:48.718656 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714951 2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 07:33:48.718656 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714954 2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 07:33:48.718656 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714958 2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 07:33:48.718656 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714960 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 07:33:48.718656 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714963 2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 07:33:48.718656 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714966 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 07:33:48.718656 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714968 2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 07:33:48.718656 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714971 2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 07:33:48.718656 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714973 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 07:33:48.718656 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714976 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 07:33:48.718656 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714978 2552 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 07:33:48.718656 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714981 2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 07:33:48.718656 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714983 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 07:33:48.718656 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714986 2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 07:33:48.718656 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714989 2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 07:33:48.718656 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714991 2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 07:33:48.718656 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714993 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 07:33:48.718656 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714996 2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 07:33:48.718656 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.714998 2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 07:33:48.718656 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715001 2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 07:33:48.719150 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715003 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 07:33:48.719150 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715006 2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 07:33:48.719150 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715011 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 07:33:48.719150 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715014 2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 07:33:48.719150 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715017 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 07:33:48.719150 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715019 2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 07:33:48.719150 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715022 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 07:33:48.719150 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715024 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 07:33:48.719150 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715027 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 07:33:48.719150 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715029 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 07:33:48.719150 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715032 2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 07:33:48.719150 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715034 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 07:33:48.719150 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715037 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 07:33:48.719150 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715040 2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 07:33:48.719150 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715043 2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 07:33:48.719150 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715046 2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 07:33:48.719150 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715050 2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 07:33:48.719150 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715054 2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 07:33:48.719150 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715057 2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 07:33:48.719620 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715060 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 07:33:48.719620 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715063 2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 07:33:48.719620 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715066 2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 07:33:48.719620 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715069 2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 07:33:48.719620 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715071 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 07:33:48.719620 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715074 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 07:33:48.719620 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715077 2552 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 07:33:48.719620 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715080 2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 07:33:48.719620 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715083 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 07:33:48.719620 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715085 2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 07:33:48.719620 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715088 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 07:33:48.719620 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715090 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 07:33:48.719620 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715093 2552 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 07:33:48.719620 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715111 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 07:33:48.719620 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715116 2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 07:33:48.719620 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715125 2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 07:33:48.719620 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715131 2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 07:33:48.719620 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715134 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 07:33:48.719620 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715137 2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 07:33:48.720130 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715140 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 07:33:48.720130 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715143 2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 07:33:48.720130 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715146 2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 07:33:48.720130 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715148 2552 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 07:33:48.720130 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715151 2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 07:33:48.720130 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715154 2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 07:33:48.720130 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715157 2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 07:33:48.720130 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715159 2552 feature_gate.go:328] unrecognized feature gate: Example Sep 16 07:33:48.720130 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715162 2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 07:33:48.720130 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715164 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 07:33:48.720130 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715168 2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 07:33:48.720130 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715171 2552 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 07:33:48.720130 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715173 2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 07:33:48.720130 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715176 2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 07:33:48.720130 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715178 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 07:33:48.720130 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715181 2552 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 07:33:48.720130 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715184 2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 07:33:48.720130 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715186 2552 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 07:33:48.720130 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715189 2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 07:33:48.720130 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715191 2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 07:33:48.720130 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715194 2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 07:33:48.720643 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715197 2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 07:33:48.720643 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715199 2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 07:33:48.720643 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715274 2552 flags.go:64] FLAG: --address="0.0.0.0" Sep 16 07:33:48.720643 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715304 2552 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 16 07:33:48.720643 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715310 2552 flags.go:64] FLAG: --anonymous-auth="true" Sep 16 07:33:48.720643 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715314 2552 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 16 07:33:48.720643 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715318 2552 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 16 07:33:48.720643 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715324 2552 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 16 07:33:48.720643 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715330 2552 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 16 07:33:48.720643 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715334 2552 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 16 07:33:48.720643 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715338 2552 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 16 07:33:48.720643 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715340 2552 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 16 07:33:48.720643 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715344 2552 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 16 07:33:48.720643 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715347 2552 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 16 07:33:48.720643 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715350 2552 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 16 07:33:48.720643 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715353 2552 flags.go:64] FLAG: --cgroup-root="" Sep 16 07:33:48.720643 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715356 2552 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 16 07:33:48.720643 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715359 2552 flags.go:64] FLAG: --client-ca-file="" Sep 16 07:33:48.720643 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715362 2552 flags.go:64] FLAG: --cloud-provider="external" Sep 16 07:33:48.720643 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715365 2552 flags.go:64] FLAG: --cluster-dns="[]" Sep 16 07:33:48.720643 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715370 2552 flags.go:64] FLAG: --cluster-domain="" Sep 16 07:33:48.720643 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715373 2552 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 16 07:33:48.720643 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715377 2552 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 16 07:33:48.721500 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715380 2552 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 16 07:33:48.721500 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715383 2552 flags.go:64] FLAG: --container-log-max-files="5" Sep 16 07:33:48.721500 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715387 2552 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 16 07:33:48.721500 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715390 2552 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 16 07:33:48.721500 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715393 2552 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 16 07:33:48.721500 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715396 2552 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 16 07:33:48.721500 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715399 2552 flags.go:64] FLAG: --contention-profiling="false" Sep 16 07:33:48.721500 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715402 2552 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 16 07:33:48.721500 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715405 2552 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 16 07:33:48.721500 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715408 2552 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 16 07:33:48.721500 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715411 2552 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 16 07:33:48.721500 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715415 2552 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 16 07:33:48.721500 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715418 2552 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 16 07:33:48.721500 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715421 2552 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 16 07:33:48.721500 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715424 2552 flags.go:64] FLAG: --enable-load-reader="false" Sep 16 07:33:48.721500 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715427 2552 flags.go:64] FLAG: --enable-server="true" Sep 16 07:33:48.721500 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715432 2552 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 16 07:33:48.721500 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715438 2552 flags.go:64] FLAG: --event-burst="100" Sep 16 07:33:48.721500 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715441 2552 flags.go:64] FLAG: --event-qps="50" Sep 16 07:33:48.721500 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715444 2552 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 16 07:33:48.721500 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715447 2552 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 16 07:33:48.721500 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715450 2552 flags.go:64] FLAG: --eviction-hard="" Sep 16 07:33:48.721500 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715453 2552 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 16 07:33:48.721500 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715456 2552 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 16 07:33:48.722184 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715459 2552 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 16 07:33:48.722184 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715462 2552 flags.go:64] FLAG: --eviction-soft="" Sep 16 07:33:48.722184 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715465 2552 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 16 07:33:48.722184 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715467 2552 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 16 07:33:48.722184 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715470 2552 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 16 07:33:48.722184 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715473 2552 flags.go:64] FLAG: --experimental-mounter-path="" Sep 16 07:33:48.722184 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715476 2552 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 16 07:33:48.722184 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715479 2552 flags.go:64] FLAG: --fail-swap-on="true" Sep 16 07:33:48.722184 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715482 2552 flags.go:64] FLAG: --feature-gates="" Sep 16 07:33:48.722184 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715486 2552 flags.go:64] FLAG: --file-check-frequency="20s" Sep 16 07:33:48.722184 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715489 2552 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 16 07:33:48.722184 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715492 2552 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 16 07:33:48.722184 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715495 2552 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 16 07:33:48.722184 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715498 2552 flags.go:64] FLAG: --healthz-port="10248" Sep 16 07:33:48.722184 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715501 2552 flags.go:64] FLAG: --help="false" Sep 16 07:33:48.722184 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715504 2552 flags.go:64] FLAG: --hostname-override="ip-10-0-142-170.ec2.internal" Sep 16 07:33:48.722184 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715507 2552 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 16 07:33:48.722184 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715510 2552 flags.go:64] FLAG: --http-check-frequency="20s" Sep 16 07:33:48.722184 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715513 2552 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 16 07:33:48.722184 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715516 2552 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 16 07:33:48.722184 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715519 2552 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 16 07:33:48.722184 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715522 2552 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 16 07:33:48.722184 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715525 2552 flags.go:64] FLAG: --image-service-endpoint="" Sep 16 07:33:48.722762 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715527 2552 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 16 07:33:48.722762 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715531 2552 flags.go:64] FLAG: --kube-api-burst="100" Sep 16 07:33:48.722762 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715536 2552 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 16 07:33:48.722762 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715539 2552 flags.go:64] FLAG: --kube-api-qps="50" Sep 16 07:33:48.722762 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715542 2552 flags.go:64] FLAG: --kube-reserved="" Sep 16 07:33:48.722762 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715544 2552 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 16 07:33:48.722762 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715547 2552 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 16 07:33:48.722762 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715550 2552 flags.go:64] FLAG: --kubelet-cgroups="" Sep 16 07:33:48.722762 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715553 2552 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 16 07:33:48.722762 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715556 2552 flags.go:64] FLAG: --lock-file="" Sep 16 07:33:48.722762 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715559 2552 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 16 07:33:48.722762 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715562 2552 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 16 07:33:48.722762 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715565 2552 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 16 07:33:48.722762 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715570 2552 flags.go:64] FLAG: --log-json-split-stream="false" Sep 16 07:33:48.722762 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715573 2552 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 16 07:33:48.722762 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715576 2552 flags.go:64] FLAG: --log-text-split-stream="false" Sep 16 07:33:48.722762 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715578 2552 flags.go:64] FLAG: --logging-format="text" Sep 16 07:33:48.722762 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715581 2552 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 16 07:33:48.722762 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715585 2552 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 16 07:33:48.722762 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715588 2552 flags.go:64] FLAG: --manifest-url="" Sep 16 07:33:48.722762 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715591 2552 flags.go:64] FLAG: --manifest-url-header="" Sep 16 07:33:48.722762 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715595 2552 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 16 07:33:48.722762 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715597 2552 flags.go:64] FLAG: --max-open-files="1000000" Sep 16 07:33:48.722762 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715601 2552 flags.go:64] FLAG: --max-pods="110" Sep 16 07:33:48.722762 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715604 2552 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 16 07:33:48.723646 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715607 2552 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 16 07:33:48.723646 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715610 2552 flags.go:64] FLAG: --memory-manager-policy="None" Sep 16 07:33:48.723646 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715612 2552 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 16 07:33:48.723646 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715615 2552 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 16 07:33:48.723646 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715618 2552 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 16 07:33:48.723646 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715621 2552 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 16 07:33:48.723646 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715628 2552 flags.go:64] FLAG: --node-status-max-images="50" Sep 16 07:33:48.723646 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715631 2552 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 16 07:33:48.723646 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715635 2552 flags.go:64] FLAG: --oom-score-adj="-999" Sep 16 07:33:48.723646 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715640 2552 flags.go:64] FLAG: --pod-cidr="" Sep 16 07:33:48.723646 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715643 2552 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 16 07:33:48.723646 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715648 2552 flags.go:64] FLAG: --pod-manifest-path="" Sep 16 07:33:48.723646 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715651 2552 flags.go:64] FLAG: --pod-max-pids="-1" Sep 16 07:33:48.723646 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715654 2552 flags.go:64] FLAG: --pods-per-core="0" Sep 16 07:33:48.723646 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715657 2552 flags.go:64] FLAG: --port="10250" Sep 16 07:33:48.723646 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715660 2552 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 16 07:33:48.723646 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715663 2552 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c84a30c618cceb75" Sep 16 07:33:48.723646 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715666 2552 flags.go:64] FLAG: --qos-reserved="" Sep 16 07:33:48.723646 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715668 2552 flags.go:64] FLAG: --read-only-port="10255" Sep 16 07:33:48.723646 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715672 2552 flags.go:64] FLAG: --register-node="true" Sep 16 07:33:48.723646 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715674 2552 flags.go:64] FLAG: --register-with-taints="" Sep 16 07:33:48.723646 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715678 2552 flags.go:64] FLAG: --registry-burst="10" Sep 16 07:33:48.723646 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715681 2552 flags.go:64] FLAG: --registry-qps="5" Sep 16 07:33:48.723646 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715684 2552 flags.go:64] FLAG: --reserved-cpus="" Sep 16 07:33:48.724449 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715687 2552 flags.go:64] FLAG: --reserved-memory="" Sep 16 07:33:48.724449 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715691 2552 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 16 07:33:48.724449 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715694 2552 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 16 07:33:48.724449 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715697 2552 flags.go:64] FLAG: --rotate-certificates="false" Sep 16 07:33:48.724449 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715700 2552 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 16 07:33:48.724449 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715702 2552 flags.go:64] FLAG: --runonce="false" Sep 16 07:33:48.724449 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715705 2552 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 16 07:33:48.724449 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715708 2552 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 16 07:33:48.724449 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715711 2552 flags.go:64] FLAG: --seccomp-default="false" Sep 16 07:33:48.724449 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715714 2552 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 16 07:33:48.724449 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715717 2552 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 16 07:33:48.724449 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715720 2552 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 16 07:33:48.724449 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715723 2552 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 16 07:33:48.724449 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715726 2552 flags.go:64] FLAG: --storage-driver-password="root" Sep 16 07:33:48.724449 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715729 2552 flags.go:64] FLAG: --storage-driver-secure="false" Sep 16 07:33:48.724449 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715731 2552 flags.go:64] FLAG: --storage-driver-table="stats" Sep 16 07:33:48.724449 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715734 2552 flags.go:64] FLAG: --storage-driver-user="root" Sep 16 07:33:48.724449 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715739 2552 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 16 07:33:48.724449 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715742 2552 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 16 07:33:48.724449 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715745 2552 flags.go:64] FLAG: --system-cgroups="" Sep 16 07:33:48.724449 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715748 2552 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 16 07:33:48.724449 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715753 2552 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 16 07:33:48.724449 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715757 2552 flags.go:64] FLAG: --tls-cert-file="" Sep 16 07:33:48.724449 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715760 2552 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 16 07:33:48.724449 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715764 2552 flags.go:64] FLAG: --tls-min-version="" Sep 16 07:33:48.725121 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715767 2552 flags.go:64] FLAG: --tls-private-key-file="" Sep 16 07:33:48.725121 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715770 2552 flags.go:64] FLAG: --topology-manager-policy="none" Sep 16 07:33:48.725121 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715773 2552 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 16 07:33:48.725121 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715776 2552 flags.go:64] FLAG: --topology-manager-scope="container" Sep 16 07:33:48.725121 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715779 2552 flags.go:64] FLAG: --v="2" Sep 16 07:33:48.725121 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715782 2552 flags.go:64] FLAG: --version="false" Sep 16 07:33:48.725121 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715786 2552 flags.go:64] FLAG: --vmodule="" Sep 16 07:33:48.725121 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715790 2552 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 16 07:33:48.725121 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.715793 2552 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 16 07:33:48.725121 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715900 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 07:33:48.725121 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715904 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 07:33:48.725121 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715907 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 07:33:48.725121 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715910 2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 07:33:48.725121 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715913 2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 07:33:48.725121 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715916 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 07:33:48.725121 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715918 2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 07:33:48.725121 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715921 2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 07:33:48.725121 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715924 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 07:33:48.725121 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715927 2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 07:33:48.725121 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715930 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 07:33:48.725121 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715933 2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 07:33:48.725121 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715938 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 07:33:48.725681 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715940 2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 07:33:48.725681 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715943 2552 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 07:33:48.725681 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715948 2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 07:33:48.725681 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715951 2552 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 07:33:48.725681 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715953 2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 07:33:48.725681 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715956 2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 07:33:48.725681 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715959 2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 07:33:48.725681 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715962 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 07:33:48.725681 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715964 2552 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 07:33:48.725681 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715968 2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 07:33:48.725681 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715972 2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 07:33:48.725681 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715975 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 07:33:48.725681 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715978 2552 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 07:33:48.725681 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715981 2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 07:33:48.725681 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715984 2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 07:33:48.725681 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715987 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 07:33:48.725681 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715990 2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 07:33:48.725681 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715992 2552 feature_gate.go:328] unrecognized feature gate: Example Sep 16 07:33:48.725681 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715995 2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 07:33:48.726171 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.715998 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 07:33:48.726171 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716001 2552 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 07:33:48.726171 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716005 2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 07:33:48.726171 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716008 2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 07:33:48.726171 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716010 2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 07:33:48.726171 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716013 2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 07:33:48.726171 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716015 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 07:33:48.726171 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716018 2552 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 07:33:48.726171 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716021 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 07:33:48.726171 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716024 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 07:33:48.726171 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716027 2552 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 07:33:48.726171 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716030 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 07:33:48.726171 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716032 2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 07:33:48.726171 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716035 2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 07:33:48.726171 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716038 2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 07:33:48.726171 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716042 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 07:33:48.726171 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716046 2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 07:33:48.726171 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716048 2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 07:33:48.726171 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716051 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 07:33:48.726171 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716054 2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 07:33:48.726655 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716056 2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 07:33:48.726655 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716059 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 07:33:48.726655 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716061 2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 07:33:48.726655 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716064 2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 07:33:48.726655 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716066 2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 07:33:48.726655 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716069 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 07:33:48.726655 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716072 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 07:33:48.726655 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716075 2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 07:33:48.726655 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716077 2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 07:33:48.726655 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716080 2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 07:33:48.726655 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716083 2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 07:33:48.726655 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716085 2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 07:33:48.726655 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716088 2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 07:33:48.726655 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716090 2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 07:33:48.726655 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716093 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 07:33:48.726655 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716108 2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 07:33:48.726655 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716111 2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 07:33:48.726655 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716114 2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 07:33:48.726655 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716116 2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 07:33:48.727120 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716120 2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 07:33:48.727120 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716124 2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 07:33:48.727120 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716127 2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 07:33:48.727120 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716129 2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 07:33:48.727120 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716132 2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 07:33:48.727120 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716135 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 07:33:48.727120 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716138 2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 07:33:48.727120 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716140 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 07:33:48.727120 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716145 2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 07:33:48.727120 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716148 2552 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 07:33:48.727120 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716151 2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 07:33:48.727120 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716153 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 07:33:48.727120 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716156 2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 07:33:48.727120 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716159 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 07:33:48.727120 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716162 2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 07:33:48.727120 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716164 2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 07:33:48.727120 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716167 2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 07:33:48.727120 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716169 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 07:33:48.727120 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716172 2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 07:33:48.727590 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716174 2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 07:33:48.727590 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716177 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 07:33:48.727590 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716180 2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 07:33:48.727590 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716183 2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 07:33:48.727590 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716185 2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 07:33:48.727590 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716188 2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 07:33:48.727590 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716190 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 07:33:48.727590 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716193 2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 07:33:48.727590 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.716196 2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 07:33:48.727590 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.716903 2552 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 07:33:48.727590 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.725513 2552 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 16 07:33:48.727590 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.725528 2552 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 16 07:33:48.727590 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.726472 2552 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 16 07:33:48.727590 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.726479 2552 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 16 07:33:48.727590 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726535 2552 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 07:33:48.727967 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726540 2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 07:33:48.727967 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726543 2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 07:33:48.727967 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726548 2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 07:33:48.727967 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726552 2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 07:33:48.727967 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726556 2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 07:33:48.727967 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726558 2552 feature_gate.go:328] unrecognized feature gate: Example Sep 16 07:33:48.727967 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726561 2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 07:33:48.727967 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726564 2552 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 07:33:48.727967 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726567 2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 07:33:48.727967 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726570 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 07:33:48.727967 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726573 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 07:33:48.727967 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726575 2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 07:33:48.727967 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726578 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 07:33:48.727967 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726582 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 07:33:48.727967 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726584 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 07:33:48.727967 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726587 2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 07:33:48.727967 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726589 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 07:33:48.727967 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726592 2552 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 07:33:48.727967 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726595 2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 07:33:48.727967 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726597 2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 07:33:48.728473 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726600 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 07:33:48.728473 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726604 2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 07:33:48.728473 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726606 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 07:33:48.728473 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726609 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 07:33:48.728473 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726612 2552 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 07:33:48.728473 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726615 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 07:33:48.728473 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726618 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 07:33:48.728473 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726620 2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 07:33:48.728473 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726624 2552 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 07:33:48.728473 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726626 2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 07:33:48.728473 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726629 2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 07:33:48.728473 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726632 2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 07:33:48.728473 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726635 2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 07:33:48.728473 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726638 2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 07:33:48.728473 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726641 2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 07:33:48.728473 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726643 2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 07:33:48.728473 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726646 2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 07:33:48.728473 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726648 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 07:33:48.728473 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726651 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 07:33:48.728473 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726654 2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 07:33:48.728998 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726656 2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 07:33:48.728998 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726661 2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 07:33:48.728998 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726664 2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 07:33:48.728998 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726667 2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 07:33:48.728998 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726670 2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 07:33:48.728998 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726674 2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 07:33:48.728998 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726676 2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 07:33:48.728998 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726679 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 07:33:48.728998 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726682 2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 07:33:48.728998 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726684 2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 07:33:48.728998 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726687 2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 07:33:48.728998 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726690 2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 07:33:48.728998 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726692 2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 07:33:48.728998 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726695 2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 07:33:48.728998 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726698 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 07:33:48.728998 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726701 2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 07:33:48.728998 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726704 2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 07:33:48.728998 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726706 2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 07:33:48.728998 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726709 2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 07:33:48.729573 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726712 2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 07:33:48.729573 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726715 2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 07:33:48.729573 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726718 2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 07:33:48.729573 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726721 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 07:33:48.729573 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726724 2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 07:33:48.729573 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726727 2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 07:33:48.729573 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726729 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 07:33:48.729573 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726732 2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 07:33:48.729573 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726736 2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 07:33:48.729573 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726740 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 07:33:48.729573 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726743 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 07:33:48.729573 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726746 2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 07:33:48.729573 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726748 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 07:33:48.729573 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726751 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 07:33:48.729573 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726753 2552 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 07:33:48.729573 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726756 2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 07:33:48.729573 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726759 2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 07:33:48.729573 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726761 2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 07:33:48.729573 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726764 2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 07:33:48.730020 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726767 2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 07:33:48.730020 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726770 2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 07:33:48.730020 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726772 2552 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 07:33:48.730020 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726774 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 07:33:48.730020 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726777 2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 07:33:48.730020 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726780 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 07:33:48.730020 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726783 2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 07:33:48.730020 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726785 2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 07:33:48.730020 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726789 2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 07:33:48.730020 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726791 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 07:33:48.730020 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726794 2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 07:33:48.730020 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726796 2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 07:33:48.730020 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726799 2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 07:33:48.730020 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726802 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 07:33:48.730020 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726805 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 07:33:48.730020 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726807 2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 07:33:48.730020 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726810 2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 07:33:48.730020 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726813 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 07:33:48.730020 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726815 2552 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 07:33:48.730020 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726818 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 07:33:48.730508 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.726823 2552 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 07:33:48.730508 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726956 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 16 07:33:48.730508 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726962 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 16 07:33:48.730508 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726965 2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 16 07:33:48.730508 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726968 2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 16 07:33:48.730508 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726971 2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 16 07:33:48.730508 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726974 2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 16 07:33:48.730508 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726976 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 16 07:33:48.730508 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726979 2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 16 07:33:48.730508 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726982 2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 16 07:33:48.730508 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726984 2552 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 16 07:33:48.730508 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726987 2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 16 07:33:48.730508 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726989 2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 16 07:33:48.730508 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726993 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 16 07:33:48.730508 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726996 2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 16 07:33:48.730873 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.726999 2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 16 07:33:48.730873 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727001 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 16 07:33:48.730873 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727004 2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 16 07:33:48.730873 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727007 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 16 07:33:48.730873 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727011 2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 16 07:33:48.730873 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727013 2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 16 07:33:48.730873 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727016 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 16 07:33:48.730873 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727019 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 16 07:33:48.730873 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727021 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 16 07:33:48.730873 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727024 2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 16 07:33:48.730873 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727026 2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 16 07:33:48.730873 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727029 2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 16 07:33:48.730873 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727032 2552 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 16 07:33:48.730873 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727035 2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 16 07:33:48.730873 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727037 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 16 07:33:48.730873 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727040 2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 16 07:33:48.730873 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727042 2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 16 07:33:48.730873 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727044 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 16 07:33:48.730873 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727047 2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 16 07:33:48.730873 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727050 2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 16 07:33:48.731370 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727052 2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 16 07:33:48.731370 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727055 2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 16 07:33:48.731370 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727057 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 16 07:33:48.731370 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727060 2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 16 07:33:48.731370 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727064 2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 16 07:33:48.731370 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727067 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 16 07:33:48.731370 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727070 2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 16 07:33:48.731370 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727073 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 16 07:33:48.731370 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727076 2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 16 07:33:48.731370 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727078 2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 16 07:33:48.731370 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727081 2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 16 07:33:48.731370 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727084 2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 16 07:33:48.731370 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727087 2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 16 07:33:48.731370 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727090 2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 16 07:33:48.731370 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727093 2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 16 07:33:48.731370 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727109 2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 16 07:33:48.731370 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727114 2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 16 07:33:48.731370 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727117 2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 16 07:33:48.731370 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727120 2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 16 07:33:48.731821 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727123 2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 16 07:33:48.731821 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727126 2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 16 07:33:48.731821 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727129 2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 16 07:33:48.731821 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727131 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 16 07:33:48.731821 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727134 2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 16 07:33:48.731821 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727137 2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 16 07:33:48.731821 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727139 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 16 07:33:48.731821 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727142 2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 16 07:33:48.731821 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727144 2552 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 16 07:33:48.731821 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727147 2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 16 07:33:48.731821 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727150 2552 feature_gate.go:328] unrecognized feature gate: Example2 Sep 16 07:33:48.731821 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727153 2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 16 07:33:48.731821 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727155 2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 16 07:33:48.731821 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727158 2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 16 07:33:48.731821 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727161 2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 16 07:33:48.731821 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727163 2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 16 07:33:48.731821 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727166 2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 16 07:33:48.731821 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727169 2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 16 07:33:48.731821 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727171 2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 16 07:33:48.731821 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727174 2552 feature_gate.go:328] unrecognized feature gate: Example Sep 16 07:33:48.732317 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727176 2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 16 07:33:48.732317 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727179 2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 16 07:33:48.732317 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727181 2552 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 16 07:33:48.732317 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727184 2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 16 07:33:48.732317 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727186 2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 16 07:33:48.732317 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727190 2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 16 07:33:48.732317 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727192 2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 16 07:33:48.732317 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727195 2552 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 16 07:33:48.732317 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727197 2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 16 07:33:48.732317 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727200 2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 16 07:33:48.732317 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727203 2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 16 07:33:48.732317 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727205 2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 16 07:33:48.732317 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727208 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 16 07:33:48.732317 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727210 2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 16 07:33:48.732317 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727214 2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 16 07:33:48.732317 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727218 2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 16 07:33:48.732317 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727221 2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 16 07:33:48.732317 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727224 2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 16 07:33:48.732317 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727227 2552 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 16 07:33:48.732774 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727231 2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 16 07:33:48.732774 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727234 2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 16 07:33:48.732774 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727238 2552 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 16 07:33:48.732774 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727241 2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 16 07:33:48.732774 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727243 2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 16 07:33:48.732774 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727246 2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 16 07:33:48.732774 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:48.727248 2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 16 07:33:48.732774 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.727253 2552 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 16 07:33:48.732774 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.727351 2552 server.go:962] "Client rotation is on, will bootstrap in background" Sep 16 07:33:48.732774 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.729740 2552 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 16 07:33:48.732774 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.730830 2552 server.go:1019] "Starting client certificate rotation" Sep 16 07:33:48.732774 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.730928 2552 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 16 07:33:48.732774 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.731752 2552 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 16 07:33:48.733663 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.733650 2552 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 16 07:33:48.734725 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.734710 2552 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 16 07:33:48.745323 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.745305 2552 log.go:25] "Validated CRI v1 runtime API" Sep 16 07:33:48.750755 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.750739 2552 log.go:25] "Validated CRI v1 image API" Sep 16 07:33:48.751536 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.751523 2552 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 16 07:33:48.754837 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.754809 2552 fs.go:135] Filesystem UUIDs: map[38fb531f-66aa-41d3-8e24-a8ab2ca0a6d8:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 e7460ea3-1e00-4513-a58c-e103ec8e1424:/dev/nvme0n1p3] Sep 16 07:33:48.754917 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.754835 2552 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 16 07:33:48.760701 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.760590 2552 manager.go:216] Machine: {Timestamp:2026-09-16 07:33:48.759174448 +0000 UTC m=+0.342952777 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099750 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ec5be174acc4bd280257a32cef8b9 SystemUUID:ec2ec5be-174a-cc4b-d280-257a32cef8b9 BootID:fabc5646-7538-473d-ae2e-140fabdb8cef Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d7:0e:4c:8f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d7:0e:4c:8f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0a:76:2f:f8:9b:73 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 16 07:33:48.760701 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.760693 2552 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 16 07:33:48.760828 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.760775 2552 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 16 07:33:48.761842 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.761827 2552 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 16 07:33:48.764486 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.764460 2552 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 16 07:33:48.764612 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.764489 2552 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-170.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 16 07:33:48.765348 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.765337 2552 topology_manager.go:138] "Creating topology manager with none policy" Sep 16 07:33:48.765382 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.765350 2552 container_manager_linux.go:309] "Creating device plugin manager" Sep 16 07:33:48.765382 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.765358 2552 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 16 07:33:48.766146 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.766136 2552 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 16 07:33:48.767432 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.767423 2552 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 16 07:33:48.773037 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.773025 2552 state_mem.go:36] "Initialized new in-memory state store" Sep 16 07:33:48.773162 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.773152 2552 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 16 07:33:48.773197 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.773182 2552 kubelet.go:486] "Attempting to sync node with API server" Sep 16 07:33:48.773197 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.773192 2552 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 16 07:33:48.773287 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.773208 2552 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 16 07:33:48.773287 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.773217 2552 kubelet.go:398] "Adding apiserver pod source" Sep 16 07:33:48.773287 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.773226 2552 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 16 07:33:48.773522 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.773512 2552 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 16 07:33:48.773552 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.773530 2552 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 16 07:33:48.775756 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.775741 2552 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 16 07:33:48.778651 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.778636 2552 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 16 07:33:48.778701 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.778666 2552 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 16 07:33:48.780887 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.780875 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 16 07:33:48.780887 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.780890 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 16 07:33:48.781005 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.780897 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 16 07:33:48.781005 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.780902 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 16 07:33:48.781005 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.780910 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 16 07:33:48.781005 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.780916 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 16 07:33:48.781005 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.780929 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 16 07:33:48.781005 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.780935 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 16 07:33:48.781005 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.780942 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 16 07:33:48.781005 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.780947 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 16 07:33:48.781005 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.780955 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 16 07:33:48.781005 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.780960 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 16 07:33:48.782766 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.782755 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 16 07:33:48.782766 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.782766 2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 16 07:33:48.783124 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:48.783089 2552 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 16 07:33:48.783176 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:48.783134 2552 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-170.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 16 07:33:48.786379 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.786367 2552 server.go:1268] "Started kubelet" Sep 16 07:33:48.786494 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.786449 2552 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 16 07:33:48.786546 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.786515 2552 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 16 07:33:48.786546 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.786475 2552 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 16 07:33:48.787150 ip-10-0-142-170 systemd[1]: Started Kubernetes Kubelet. Sep 16 07:33:48.788244 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.788221 2552 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 16 07:33:48.788792 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.788707 2552 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dnzmx" Sep 16 07:33:48.788843 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.788816 2552 server.go:310] "Adding debug handlers to kubelet server" Sep 16 07:33:48.792441 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:48.792411 2552 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 16 07:33:48.794574 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.794559 2552 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-170.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 16 07:33:48.795702 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:48.794568 2552 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-170.ec2.internal.18d5bcd011bb36d5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-170.ec2.internal,UID:ip-10-0-142-170.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-170.ec2.internal,},FirstTimestamp:2026-09-16 07:33:48.786341589 +0000 UTC m=+0.370119918,LastTimestamp:2026-09-16 07:33:48.786341589 +0000 UTC m=+0.370119918,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-170.ec2.internal,}" Sep 16 07:33:48.797939 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.797921 2552 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-dnzmx" Sep 16 07:33:48.799683 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.799667 2552 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 16 07:33:48.800301 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.800286 2552 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 16 07:33:48.800593 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:48.800566 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-170.ec2.internal\" not found" Sep 16 07:33:48.801195 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.801177 2552 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 16 07:33:48.801195 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.801176 2552 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 16 07:33:48.801289 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.801206 2552 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 16 07:33:48.801289 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.801177 2552 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 16 07:33:48.801289 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.801262 2552 factory.go:55] Registering systemd factory Sep 16 07:33:48.801289 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.801273 2552 factory.go:223] Registration of the systemd container factory successfully Sep 16 07:33:48.801431 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.801300 2552 reconstruct.go:97] "Volume reconstruction finished" Sep 16 07:33:48.801431 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.801307 2552 reconciler.go:29] "Reconciler: start to sync state" Sep 16 07:33:48.801519 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.801508 2552 factory.go:153] Registering CRI-O factory Sep 16 07:33:48.801549 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.801521 2552 factory.go:223] Registration of the crio container factory successfully Sep 16 07:33:48.801549 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.801535 2552 factory.go:103] Registering Raw factory Sep 16 07:33:48.801549 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.801544 2552 manager.go:1196] Started watching for new ooms in manager Sep 16 07:33:48.801905 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.801891 2552 manager.go:349] Starting recovery of all containers Sep 16 07:33:48.803629 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.803507 2552 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 07:33:48.809557 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:48.809539 2552 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-142-170.ec2.internal\" not found" node="ip-10-0-142-170.ec2.internal" Sep 16 07:33:48.812983 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.812968 2552 manager.go:354] Recovery completed Sep 16 07:33:48.817977 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.817963 2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 07:33:48.820057 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.820038 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-170.ec2.internal" event="NodeHasSufficientMemory" Sep 16 07:33:48.820143 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.820070 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-170.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 07:33:48.820143 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.820091 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-170.ec2.internal" event="NodeHasSufficientPID" Sep 16 07:33:48.821077 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.821059 2552 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 16 07:33:48.821077 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.821075 2552 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 16 07:33:48.821207 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.821109 2552 state_mem.go:36] "Initialized new in-memory state store" Sep 16 07:33:48.821834 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.821455 2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 07:33:48.822495 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.822479 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-170.ec2.internal" event="NodeHasSufficientMemory" Sep 16 07:33:48.822572 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.822506 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-170.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 07:33:48.822572 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.822517 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-170.ec2.internal" event="NodeHasSufficientPID" Sep 16 07:33:48.823341 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.823327 2552 policy_none.go:49] "None policy: Start" Sep 16 07:33:48.823383 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.823354 2552 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 16 07:33:48.823383 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.823368 2552 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 16 07:33:48.825426 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.825414 2552 policy_none.go:47] "Start" Sep 16 07:33:48.874532 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.854718 2552 manager.go:337] "Starting Device Plugin manager" Sep 16 07:33:48.874532 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:48.854747 2552 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 16 07:33:48.874532 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.854756 2552 server.go:87] "Starting device plugin registration server" Sep 16 07:33:48.874532 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.855024 2552 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 16 07:33:48.874532 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.855035 2552 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 16 07:33:48.874532 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.855142 2552 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 16 07:33:48.874532 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.855222 2552 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 16 07:33:48.874532 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.855230 2552 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 16 07:33:48.874532 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:48.855747 2552 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 16 07:33:48.874532 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:48.855803 2552 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-170.ec2.internal\" not found" Sep 16 07:33:48.928963 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.928930 2552 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 16 07:33:48.930118 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.930094 2552 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 16 07:33:48.930211 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.930151 2552 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 16 07:33:48.930211 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.930176 2552 kubelet.go:2448] "Starting kubelet main sync loop" Sep 16 07:33:48.930296 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:48.930217 2552 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 16 07:33:48.932694 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.932678 2552 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 07:33:48.955706 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.955692 2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 07:33:48.956583 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.956529 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-170.ec2.internal" event="NodeHasSufficientMemory" Sep 16 07:33:48.956583 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.956556 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-170.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 07:33:48.956583 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.956566 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-170.ec2.internal" event="NodeHasSufficientPID" Sep 16 07:33:48.956713 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.956587 2552 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-170.ec2.internal" Sep 16 07:33:48.966075 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:48.966062 2552 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-170.ec2.internal" Sep 16 07:33:48.966128 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:48.966082 2552 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-170.ec2.internal\": node \"ip-10-0-142-170.ec2.internal\" not found" Sep 16 07:33:48.992993 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:48.992977 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-170.ec2.internal\" not found" Sep 16 07:33:49.030278 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:49.030255 2552 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-170.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-170.ec2.internal"] Sep 16 07:33:49.030343 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:49.030320 2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 07:33:49.031133 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:49.031113 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-170.ec2.internal" event="NodeHasSufficientMemory" Sep 16 07:33:49.031207 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:49.031144 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-170.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 07:33:49.031207 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:49.031161 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-170.ec2.internal" event="NodeHasSufficientPID" Sep 16 07:33:49.033447 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:49.033436 2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 07:33:49.033583 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:49.033570 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-170.ec2.internal" Sep 16 07:33:49.033619 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:49.033600 2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 07:33:49.034192 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:49.034172 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-170.ec2.internal" event="NodeHasSufficientMemory" Sep 16 07:33:49.034254 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:49.034201 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-170.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 07:33:49.034254 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:49.034210 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-170.ec2.internal" event="NodeHasSufficientPID" Sep 16 07:33:49.034696 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:49.034683 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-170.ec2.internal" event="NodeHasSufficientMemory" Sep 16 07:33:49.034755 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:49.034705 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-170.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 07:33:49.034755 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:49.034718 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-170.ec2.internal" event="NodeHasSufficientPID" Sep 16 07:33:49.036917 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:49.036905 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-170.ec2.internal" Sep 16 07:33:49.036959 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:49.036929 2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 16 07:33:49.037591 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:49.037576 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-170.ec2.internal" event="NodeHasSufficientMemory" Sep 16 07:33:49.037653 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:49.037605 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-170.ec2.internal" event="NodeHasNoDiskPressure" Sep 16 07:33:49.037653 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:49.037619 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-170.ec2.internal" event="NodeHasSufficientPID" Sep 16 07:33:49.069767 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:49.069752 2552 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-170.ec2.internal\" not found" node="ip-10-0-142-170.ec2.internal" Sep 16 07:33:49.073950 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:49.073936 2552 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-170.ec2.internal\" not found" node="ip-10-0-142-170.ec2.internal" Sep 16 07:33:49.093331 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:49.093317 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-170.ec2.internal\" not found" Sep 16 07:33:49.102465 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:49.102450 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9ae2c91e0de9f347633243d0f5bfd92f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-170.ec2.internal\" (UID: \"9ae2c91e0de9f347633243d0f5bfd92f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-170.ec2.internal" Sep 16 07:33:49.102536 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:49.102472 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ae2c91e0de9f347633243d0f5bfd92f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-170.ec2.internal\" (UID: \"9ae2c91e0de9f347633243d0f5bfd92f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-170.ec2.internal" Sep 16 07:33:49.102536 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:49.102488 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1fcac57a8698f66711e8e05f204f0824-config\") pod \"kube-apiserver-proxy-ip-10-0-142-170.ec2.internal\" (UID: \"1fcac57a8698f66711e8e05f204f0824\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-170.ec2.internal" Sep 16 07:33:49.194044 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:49.194017 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-170.ec2.internal\" not found" Sep 16 07:33:49.203418 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:49.203325 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ae2c91e0de9f347633243d0f5bfd92f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-170.ec2.internal\" (UID: \"9ae2c91e0de9f347633243d0f5bfd92f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-170.ec2.internal" Sep 16 07:33:49.203518 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:49.203436 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1fcac57a8698f66711e8e05f204f0824-config\") pod \"kube-apiserver-proxy-ip-10-0-142-170.ec2.internal\" (UID: \"1fcac57a8698f66711e8e05f204f0824\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-170.ec2.internal" Sep 16 07:33:49.203518 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:49.203456 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9ae2c91e0de9f347633243d0f5bfd92f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-170.ec2.internal\" (UID: \"9ae2c91e0de9f347633243d0f5bfd92f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-170.ec2.internal" Sep 16 07:33:49.203518 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:49.203387 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9ae2c91e0de9f347633243d0f5bfd92f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-170.ec2.internal\" (UID: \"9ae2c91e0de9f347633243d0f5bfd92f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-170.ec2.internal" Sep 16 07:33:49.203518 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:49.203489 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9ae2c91e0de9f347633243d0f5bfd92f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-170.ec2.internal\" (UID: \"9ae2c91e0de9f347633243d0f5bfd92f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-170.ec2.internal" Sep 16 07:33:49.203663 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:49.203521 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1fcac57a8698f66711e8e05f204f0824-config\") pod \"kube-apiserver-proxy-ip-10-0-142-170.ec2.internal\" (UID: \"1fcac57a8698f66711e8e05f204f0824\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-170.ec2.internal" Sep 16 07:33:49.294762 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:49.294708 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-170.ec2.internal\" not found" Sep 16 07:33:49.373887 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:49.373864 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-170.ec2.internal" Sep 16 07:33:49.377184 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:49.377166 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-170.ec2.internal" Sep 16 07:33:49.395345 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:49.395323 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-170.ec2.internal\" not found" Sep 16 07:33:49.498246 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:49.498202 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-170.ec2.internal\" not found" Sep 16 07:33:49.598996 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:49.598922 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-170.ec2.internal\" not found" Sep 16 07:33:49.703093 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:49.703073 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-170.ec2.internal\" not found" Sep 16 07:33:49.730555 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:49.730534 2552 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 16 07:33:49.730946 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:49.730661 2552 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 16 07:33:49.730946 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:49.730704 2552 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 16 07:33:49.801481 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:49.801439 2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 07:28:48 +0000 UTC" deadline="2028-04-22 07:56:57.703611585 +0000 UTC" Sep 16 07:33:49.801481 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:49.801472 2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14016h23m7.902142893s" Sep 16 07:33:49.803618 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:49.803599 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-170.ec2.internal\" not found" Sep 16 07:33:49.904262 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:49.904236 2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-170.ec2.internal\" not found" Sep 16 07:33:49.929552 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:49.929532 2552 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 07:33:50.001504 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.001469 2552 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-170.ec2.internal" Sep 16 07:33:50.012057 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.012035 2552 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 16 07:33:50.013591 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.013571 2552 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-170.ec2.internal" Sep 16 07:33:50.014357 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:50.014330 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ae2c91e0de9f347633243d0f5bfd92f.slice/crio-cec405866bc7aa0a92325f2f1761bf204538bc8e3930cc77277aa1bcf7d466c8 WatchSource:0}: Error finding container cec405866bc7aa0a92325f2f1761bf204538bc8e3930cc77277aa1bcf7d466c8: Status 404 returned error can't find the container with id cec405866bc7aa0a92325f2f1761bf204538bc8e3930cc77277aa1bcf7d466c8 Sep 16 07:33:50.014568 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:50.014547 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fcac57a8698f66711e8e05f204f0824.slice/crio-6df5d14bb0026118891632885e78d53a97a502170f56645aa344b9deb6cf5a4f WatchSource:0}: Error finding container 6df5d14bb0026118891632885e78d53a97a502170f56645aa344b9deb6cf5a4f: Status 404 returned error can't find the container with id 6df5d14bb0026118891632885e78d53a97a502170f56645aa344b9deb6cf5a4f Sep 16 07:33:50.019323 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.019309 2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 07:33:50.020247 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.020232 2552 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 16 07:33:50.369633 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.369607 2552 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 07:33:50.774821 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.774493 2552 apiserver.go:52] "Watching apiserver" Sep 16 07:33:50.782116 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.781942 2552 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 16 07:33:50.783116 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.783076 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-mdzsg","openshift-multus/multus-tf242","openshift-network-diagnostics/network-check-target-vx6vh","openshift-network-operator/iptables-alerter-t458f","kube-system/global-pull-secret-syncer-wbslq","kube-system/konnectivity-agent-7zl67","kube-system/kube-apiserver-proxy-ip-10-0-142-170.ec2.internal","openshift-cluster-node-tuning-operator/tuned-v5srt","openshift-multus/network-metrics-daemon-94n4h","openshift-ovn-kubernetes/ovnkube-node-xc9v8","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms2dc","openshift-dns/node-resolver-rxbfc","openshift-image-registry/node-ca-9xkmz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-170.ec2.internal"] Sep 16 07:33:50.786122 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.785820 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.788059 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.788036 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 16 07:33:50.788652 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.788635 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 16 07:33:50.788765 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.788747 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 16 07:33:50.788927 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.788914 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9xkmz" Sep 16 07:33:50.789379 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.789335 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 16 07:33:50.793235 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.793217 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 16 07:33:50.794682 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.793461 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xl8dw\"" Sep 16 07:33:50.794682 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.793641 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 16 07:33:50.794682 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.793812 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vx6vh" Sep 16 07:33:50.794682 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:50.793869 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vx6vh" podUID="3c6593a0-6ec1-4945-b929-f94446ee82b1" Sep 16 07:33:50.794682 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.794057 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-t458f" Sep 16 07:33:50.794682 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.794483 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 16 07:33:50.794682 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.794504 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 16 07:33:50.795054 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.794894 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wbslq" Sep 16 07:33:50.795054 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:50.794956 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wbslq" podUID="4152b2c2-a19c-44fa-96f5-dc3eb661aa9b" Sep 16 07:33:50.795386 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.795371 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4tdj5\"" Sep 16 07:33:50.795735 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.795540 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 16 07:33:50.796493 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.796081 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 16 07:33:50.796493 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.796271 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bmsbq\"" Sep 16 07:33:50.797341 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.797084 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 16 07:33:50.797341 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.797281 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 16 07:33:50.798811 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.798554 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7zl67" Sep 16 07:33:50.800735 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.800719 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7k7p6\"" Sep 16 07:33:50.802007 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.801278 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 16 07:33:50.802007 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.801415 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 16 07:33:50.802007 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.801486 2552 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 16 07:33:50.803243 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.803215 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-94n4h" Sep 16 07:33:50.803348 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:50.803328 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-94n4h" podUID="aca15303-6546-4428-bb32-539bb25d2486" Sep 16 07:33:50.814294 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.811869 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v5srt" Sep 16 07:33:50.814294 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.812687 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/37c8f2c4-2c9b-47f6-b709-55791b0741cb-ovn-node-metrics-cert\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.814294 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.812723 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ef5303b5-4dc6-4e4c-9716-bcf88a796cae-agent-certs\") pod \"konnectivity-agent-7zl67\" (UID: \"ef5303b5-4dc6-4e4c-9716-bcf88a796cae\") " pod="kube-system/konnectivity-agent-7zl67" Sep 16 07:33:50.814294 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.812755 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ef5303b5-4dc6-4e4c-9716-bcf88a796cae-konnectivity-ca\") pod \"konnectivity-agent-7zl67\" (UID: \"ef5303b5-4dc6-4e4c-9716-bcf88a796cae\") " pod="kube-system/konnectivity-agent-7zl67" Sep 16 07:33:50.814294 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.812791 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/37c8f2c4-2c9b-47f6-b709-55791b0741cb-host-cni-bin\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.814294 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.812825 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37c8f2c4-2c9b-47f6-b709-55791b0741cb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.814294 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.812858 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/29d2a484-46de-40a9-be4a-b9c6b266fbae-serviceca\") pod \"node-ca-9xkmz\" (UID: \"29d2a484-46de-40a9-be4a-b9c6b266fbae\") " pod="openshift-image-registry/node-ca-9xkmz" Sep 16 07:33:50.814294 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.812882 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzbkw\" (UniqueName: \"kubernetes.io/projected/29d2a484-46de-40a9-be4a-b9c6b266fbae-kube-api-access-xzbkw\") pod \"node-ca-9xkmz\" (UID: \"29d2a484-46de-40a9-be4a-b9c6b266fbae\") " pod="openshift-image-registry/node-ca-9xkmz" Sep 16 07:33:50.814294 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.812912 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4152b2c2-a19c-44fa-96f5-dc3eb661aa9b-kubelet-config\") pod \"global-pull-secret-syncer-wbslq\" (UID: \"4152b2c2-a19c-44fa-96f5-dc3eb661aa9b\") " pod="kube-system/global-pull-secret-syncer-wbslq" Sep 16 07:33:50.814294 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.812944 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/37c8f2c4-2c9b-47f6-b709-55791b0741cb-systemd-units\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.814294 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.812972 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/37c8f2c4-2c9b-47f6-b709-55791b0741cb-log-socket\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.814294 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.813005 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/37c8f2c4-2c9b-47f6-b709-55791b0741cb-ovnkube-script-lib\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.814294 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.813044 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnt8f\" (UniqueName: \"kubernetes.io/projected/3c6593a0-6ec1-4945-b929-f94446ee82b1-kube-api-access-hnt8f\") pod \"network-check-target-vx6vh\" (UID: \"3c6593a0-6ec1-4945-b929-f94446ee82b1\") " pod="openshift-network-diagnostics/network-check-target-vx6vh" Sep 16 07:33:50.814294 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.813075 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/37c8f2c4-2c9b-47f6-b709-55791b0741cb-node-log\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.814294 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.813120 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p8rr\" (UniqueName: \"kubernetes.io/projected/37c8f2c4-2c9b-47f6-b709-55791b0741cb-kube-api-access-7p8rr\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.814294 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.813151 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4152b2c2-a19c-44fa-96f5-dc3eb661aa9b-original-pull-secret\") pod \"global-pull-secret-syncer-wbslq\" (UID: \"4152b2c2-a19c-44fa-96f5-dc3eb661aa9b\") " pod="kube-system/global-pull-secret-syncer-wbslq" Sep 16 07:33:50.814294 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.813184 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/96bb36bc-d267-42e2-8509-0c72ab202bdc-iptables-alerter-script\") pod \"iptables-alerter-t458f\" (UID: \"96bb36bc-d267-42e2-8509-0c72ab202bdc\") " pod="openshift-network-operator/iptables-alerter-t458f" Sep 16 07:33:50.815127 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.813213 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/37c8f2c4-2c9b-47f6-b709-55791b0741cb-host-run-netns\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.815127 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.813244 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/37c8f2c4-2c9b-47f6-b709-55791b0741cb-env-overrides\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.815127 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.813284 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/37c8f2c4-2c9b-47f6-b709-55791b0741cb-host-kubelet\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.815127 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.813317 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/37c8f2c4-2c9b-47f6-b709-55791b0741cb-run-systemd\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.815127 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.813347 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/37c8f2c4-2c9b-47f6-b709-55791b0741cb-run-ovn\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.815127 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.813370 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/29d2a484-46de-40a9-be4a-b9c6b266fbae-host\") pod \"node-ca-9xkmz\" (UID: \"29d2a484-46de-40a9-be4a-b9c6b266fbae\") " pod="openshift-image-registry/node-ca-9xkmz" Sep 16 07:33:50.815127 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.813400 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/96bb36bc-d267-42e2-8509-0c72ab202bdc-host-slash\") pod \"iptables-alerter-t458f\" (UID: \"96bb36bc-d267-42e2-8509-0c72ab202bdc\") " pod="openshift-network-operator/iptables-alerter-t458f" Sep 16 07:33:50.815127 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.813431 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4152b2c2-a19c-44fa-96f5-dc3eb661aa9b-dbus\") pod \"global-pull-secret-syncer-wbslq\" (UID: \"4152b2c2-a19c-44fa-96f5-dc3eb661aa9b\") " pod="kube-system/global-pull-secret-syncer-wbslq" Sep 16 07:33:50.819084 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.815689 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7gnkh\"" Sep 16 07:33:50.819084 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.816043 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 16 07:33:50.819084 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.816454 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vgb8\" (UniqueName: \"kubernetes.io/projected/aca15303-6546-4428-bb32-539bb25d2486-kube-api-access-7vgb8\") pod \"network-metrics-daemon-94n4h\" (UID: \"aca15303-6546-4428-bb32-539bb25d2486\") " pod="openshift-multus/network-metrics-daemon-94n4h" Sep 16 07:33:50.819084 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.816520 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/37c8f2c4-2c9b-47f6-b709-55791b0741cb-host-slash\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.819084 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.816547 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37c8f2c4-2c9b-47f6-b709-55791b0741cb-etc-openvswitch\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.819084 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.816577 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37c8f2c4-2c9b-47f6-b709-55791b0741cb-run-openvswitch\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.819084 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.816622 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37c8f2c4-2c9b-47f6-b709-55791b0741cb-host-run-ovn-kubernetes\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.819084 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.816696 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/37c8f2c4-2c9b-47f6-b709-55791b0741cb-host-cni-netd\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.819084 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.816728 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/37c8f2c4-2c9b-47f6-b709-55791b0741cb-ovnkube-config\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.819084 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.816777 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aca15303-6546-4428-bb32-539bb25d2486-metrics-certs\") pod \"network-metrics-daemon-94n4h\" (UID: \"aca15303-6546-4428-bb32-539bb25d2486\") " pod="openshift-multus/network-metrics-daemon-94n4h" Sep 16 07:33:50.819084 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.816841 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 16 07:33:50.819084 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.816859 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fprrz\" (UniqueName: \"kubernetes.io/projected/96bb36bc-d267-42e2-8509-0c72ab202bdc-kube-api-access-fprrz\") pod \"iptables-alerter-t458f\" (UID: \"96bb36bc-d267-42e2-8509-0c72ab202bdc\") " pod="openshift-network-operator/iptables-alerter-t458f" Sep 16 07:33:50.819084 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.816905 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37c8f2c4-2c9b-47f6-b709-55791b0741cb-var-lib-openvswitch\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.819084 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.819068 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tf242" Sep 16 07:33:50.819773 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.819525 2552 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 16 07:33:50.822362 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.822338 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-sgvqd\"" Sep 16 07:33:50.822609 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.822594 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 16 07:33:50.822800 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.822784 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 16 07:33:50.822901 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.822883 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 16 07:33:50.823013 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.822998 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 16 07:33:50.823368 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.823338 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mdzsg" Sep 16 07:33:50.825676 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.825655 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dccdq\"" Sep 16 07:33:50.825930 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.825911 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 16 07:33:50.826297 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.826087 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 16 07:33:50.826668 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.826442 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms2dc" Sep 16 07:33:50.827871 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.827849 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rxbfc" Sep 16 07:33:50.828520 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.828501 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 16 07:33:50.828695 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.828681 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 16 07:33:50.828875 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.828863 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 16 07:33:50.829046 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.829029 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-lfzwn\"" Sep 16 07:33:50.830040 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.830018 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 16 07:33:50.830142 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.830061 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2pvw7\"" Sep 16 07:33:50.832081 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.832061 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 16 07:33:50.835189 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.834963 2552 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wpnlk" Sep 16 07:33:50.842895 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.842819 2552 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wpnlk" Sep 16 07:33:50.902302 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.902280 2552 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 16 07:33:50.906659 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.906623 2552 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 07:33:50.917071 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.917045 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/37c8f2c4-2c9b-47f6-b709-55791b0741cb-env-overrides\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.917195 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.917086 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8cdfedd6-88d2-459f-bf39-504bc65e2fed-host-var-lib-cni-multus\") pod \"multus-tf242\" (UID: \"8cdfedd6-88d2-459f-bf39-504bc65e2fed\") " pod="openshift-multus/multus-tf242" Sep 16 07:33:50.917195 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.917126 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgbbd\" (UniqueName: \"kubernetes.io/projected/8cdfedd6-88d2-459f-bf39-504bc65e2fed-kube-api-access-hgbbd\") pod \"multus-tf242\" (UID: \"8cdfedd6-88d2-459f-bf39-504bc65e2fed\") " pod="openshift-multus/multus-tf242" Sep 16 07:33:50.917195 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.917150 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k7kb\" (UniqueName: \"kubernetes.io/projected/024d7ae0-3a5a-4d5e-bedb-592058d44673-kube-api-access-8k7kb\") pod \"node-resolver-rxbfc\" (UID: \"024d7ae0-3a5a-4d5e-bedb-592058d44673\") " pod="openshift-dns/node-resolver-rxbfc" Sep 16 07:33:50.917195 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.917172 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9b58cf99-fce1-4445-a36c-5d624bf3d0a1-lib-modules\") pod \"tuned-v5srt\" (UID: \"9b58cf99-fce1-4445-a36c-5d624bf3d0a1\") " pod="openshift-cluster-node-tuning-operator/tuned-v5srt" Sep 16 07:33:50.917195 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.917193 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/16e6d2a0-df60-4490-a3f2-3c1d79d8cedd-socket-dir\") pod \"aws-ebs-csi-driver-node-ms2dc\" (UID: \"16e6d2a0-df60-4490-a3f2-3c1d79d8cedd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms2dc" Sep 16 07:33:50.917449 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.917229 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/37c8f2c4-2c9b-47f6-b709-55791b0741cb-host-kubelet\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.917449 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.917254 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/37c8f2c4-2c9b-47f6-b709-55791b0741cb-run-systemd\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.917449 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.917291 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/37c8f2c4-2c9b-47f6-b709-55791b0741cb-run-ovn\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.917449 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.917315 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/29d2a484-46de-40a9-be4a-b9c6b266fbae-host\") pod \"node-ca-9xkmz\" (UID: \"29d2a484-46de-40a9-be4a-b9c6b266fbae\") " pod="openshift-image-registry/node-ca-9xkmz" Sep 16 07:33:50.917449 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.917339 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9b58cf99-fce1-4445-a36c-5d624bf3d0a1-run\") pod \"tuned-v5srt\" (UID: \"9b58cf99-fce1-4445-a36c-5d624bf3d0a1\") " pod="openshift-cluster-node-tuning-operator/tuned-v5srt" Sep 16 07:33:50.917449 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.917359 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b58cf99-fce1-4445-a36c-5d624bf3d0a1-tmp\") pod \"tuned-v5srt\" (UID: \"9b58cf99-fce1-4445-a36c-5d624bf3d0a1\") " pod="openshift-cluster-node-tuning-operator/tuned-v5srt" Sep 16 07:33:50.917449 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.917397 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/16e6d2a0-df60-4490-a3f2-3c1d79d8cedd-registration-dir\") pod \"aws-ebs-csi-driver-node-ms2dc\" (UID: \"16e6d2a0-df60-4490-a3f2-3c1d79d8cedd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms2dc" Sep 16 07:33:50.917449 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.917423 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/16e6d2a0-df60-4490-a3f2-3c1d79d8cedd-etc-selinux\") pod \"aws-ebs-csi-driver-node-ms2dc\" (UID: \"16e6d2a0-df60-4490-a3f2-3c1d79d8cedd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms2dc" Sep 16 07:33:50.917449 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.917448 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mdzsg\" (UID: \"2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7\") " pod="openshift-multus/multus-additional-cni-plugins-mdzsg" Sep 16 07:33:50.917846 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.917470 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/024d7ae0-3a5a-4d5e-bedb-592058d44673-hosts-file\") pod \"node-resolver-rxbfc\" (UID: \"024d7ae0-3a5a-4d5e-bedb-592058d44673\") " pod="openshift-dns/node-resolver-rxbfc" Sep 16 07:33:50.917846 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.917494 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/16e6d2a0-df60-4490-a3f2-3c1d79d8cedd-sys-fs\") pod \"aws-ebs-csi-driver-node-ms2dc\" (UID: \"16e6d2a0-df60-4490-a3f2-3c1d79d8cedd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms2dc" Sep 16 07:33:50.917846 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.917501 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/37c8f2c4-2c9b-47f6-b709-55791b0741cb-env-overrides\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.917846 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.917525 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/37c8f2c4-2c9b-47f6-b709-55791b0741cb-ovnkube-config\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.917846 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.917551 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aca15303-6546-4428-bb32-539bb25d2486-metrics-certs\") pod \"network-metrics-daemon-94n4h\" (UID: \"aca15303-6546-4428-bb32-539bb25d2486\") " pod="openshift-multus/network-metrics-daemon-94n4h" Sep 16 07:33:50.917846 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.917581 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fprrz\" (UniqueName: \"kubernetes.io/projected/96bb36bc-d267-42e2-8509-0c72ab202bdc-kube-api-access-fprrz\") pod \"iptables-alerter-t458f\" (UID: \"96bb36bc-d267-42e2-8509-0c72ab202bdc\") " pod="openshift-network-operator/iptables-alerter-t458f" Sep 16 07:33:50.917846 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.917595 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/29d2a484-46de-40a9-be4a-b9c6b266fbae-host\") pod \"node-ca-9xkmz\" (UID: \"29d2a484-46de-40a9-be4a-b9c6b266fbae\") " pod="openshift-image-registry/node-ca-9xkmz" Sep 16 07:33:50.917846 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.917613 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8cdfedd6-88d2-459f-bf39-504bc65e2fed-multus-socket-dir-parent\") pod \"multus-tf242\" (UID: \"8cdfedd6-88d2-459f-bf39-504bc65e2fed\") " pod="openshift-multus/multus-tf242" Sep 16 07:33:50.917846 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.917639 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8cdfedd6-88d2-459f-bf39-504bc65e2fed-etc-kubernetes\") pod \"multus-tf242\" (UID: \"8cdfedd6-88d2-459f-bf39-504bc65e2fed\") " pod="openshift-multus/multus-tf242" Sep 16 07:33:50.917846 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.917710 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/37c8f2c4-2c9b-47f6-b709-55791b0741cb-run-ovn\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.917846 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.917738 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/37c8f2c4-2c9b-47f6-b709-55791b0741cb-host-kubelet\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.917846 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.917761 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/37c8f2c4-2c9b-47f6-b709-55791b0741cb-run-systemd\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.918627 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.917935 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9b58cf99-fce1-4445-a36c-5d624bf3d0a1-etc-sysconfig\") pod \"tuned-v5srt\" (UID: \"9b58cf99-fce1-4445-a36c-5d624bf3d0a1\") " pod="openshift-cluster-node-tuning-operator/tuned-v5srt" Sep 16 07:33:50.918627 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.917964 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/16e6d2a0-df60-4490-a3f2-3c1d79d8cedd-device-dir\") pod \"aws-ebs-csi-driver-node-ms2dc\" (UID: \"16e6d2a0-df60-4490-a3f2-3c1d79d8cedd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms2dc" Sep 16 07:33:50.918627 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.918003 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/37c8f2c4-2c9b-47f6-b709-55791b0741cb-ovn-node-metrics-cert\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.918627 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:50.918261 2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 07:33:50.918627 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:50.918438 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aca15303-6546-4428-bb32-539bb25d2486-metrics-certs podName:aca15303-6546-4428-bb32-539bb25d2486 nodeName:}" failed. No retries permitted until 2026-09-16 07:33:51.418393591 +0000 UTC m=+3.002171921 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aca15303-6546-4428-bb32-539bb25d2486-metrics-certs") pod "network-metrics-daemon-94n4h" (UID: "aca15303-6546-4428-bb32-539bb25d2486") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 07:33:50.918627 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.918068 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ef5303b5-4dc6-4e4c-9716-bcf88a796cae-agent-certs\") pod \"konnectivity-agent-7zl67\" (UID: \"ef5303b5-4dc6-4e4c-9716-bcf88a796cae\") " pod="kube-system/konnectivity-agent-7zl67" Sep 16 07:33:50.918627 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.918607 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8cdfedd6-88d2-459f-bf39-504bc65e2fed-system-cni-dir\") pod \"multus-tf242\" (UID: \"8cdfedd6-88d2-459f-bf39-504bc65e2fed\") " pod="openshift-multus/multus-tf242" Sep 16 07:33:50.918627 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.918626 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8cdfedd6-88d2-459f-bf39-504bc65e2fed-multus-conf-dir\") pod \"multus-tf242\" (UID: \"8cdfedd6-88d2-459f-bf39-504bc65e2fed\") " pod="openshift-multus/multus-tf242" Sep 16 07:33:50.918627 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.918626 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/37c8f2c4-2c9b-47f6-b709-55791b0741cb-ovnkube-config\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.919093 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.918642 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mdzsg\" (UID: \"2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7\") " pod="openshift-multus/multus-additional-cni-plugins-mdzsg" Sep 16 07:33:50.919093 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.918664 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b58cf99-fce1-4445-a36c-5d624bf3d0a1-host\") pod \"tuned-v5srt\" (UID: \"9b58cf99-fce1-4445-a36c-5d624bf3d0a1\") " pod="openshift-cluster-node-tuning-operator/tuned-v5srt" Sep 16 07:33:50.919093 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.918697 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/37c8f2c4-2c9b-47f6-b709-55791b0741cb-host-cni-bin\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.919093 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.918720 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzbkw\" (UniqueName: \"kubernetes.io/projected/29d2a484-46de-40a9-be4a-b9c6b266fbae-kube-api-access-xzbkw\") pod \"node-ca-9xkmz\" (UID: \"29d2a484-46de-40a9-be4a-b9c6b266fbae\") " pod="openshift-image-registry/node-ca-9xkmz" Sep 16 07:33:50.919093 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.918745 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8cdfedd6-88d2-459f-bf39-504bc65e2fed-host-run-k8s-cni-cncf-io\") pod \"multus-tf242\" (UID: \"8cdfedd6-88d2-459f-bf39-504bc65e2fed\") " pod="openshift-multus/multus-tf242" Sep 16 07:33:50.919093 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.918769 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8cdfedd6-88d2-459f-bf39-504bc65e2fed-host-run-netns\") pod \"multus-tf242\" (UID: \"8cdfedd6-88d2-459f-bf39-504bc65e2fed\") " pod="openshift-multus/multus-tf242" Sep 16 07:33:50.919093 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.918782 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/37c8f2c4-2c9b-47f6-b709-55791b0741cb-host-cni-bin\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.919093 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.918788 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/37c8f2c4-2c9b-47f6-b709-55791b0741cb-log-socket\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.919093 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.918820 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/37c8f2c4-2c9b-47f6-b709-55791b0741cb-log-socket\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.919093 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.918839 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/37c8f2c4-2c9b-47f6-b709-55791b0741cb-ovnkube-script-lib\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.919093 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.918870 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnt8f\" (UniqueName: \"kubernetes.io/projected/3c6593a0-6ec1-4945-b929-f94446ee82b1-kube-api-access-hnt8f\") pod \"network-check-target-vx6vh\" (UID: \"3c6593a0-6ec1-4945-b929-f94446ee82b1\") " pod="openshift-network-diagnostics/network-check-target-vx6vh" Sep 16 07:33:50.919093 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.918895 2552 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 16 07:33:50.919093 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.918913 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7-cni-binary-copy\") pod \"multus-additional-cni-plugins-mdzsg\" (UID: \"2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7\") " pod="openshift-multus/multus-additional-cni-plugins-mdzsg" Sep 16 07:33:50.919093 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.918940 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/37c8f2c4-2c9b-47f6-b709-55791b0741cb-node-log\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.919093 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.918965 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p8rr\" (UniqueName: \"kubernetes.io/projected/37c8f2c4-2c9b-47f6-b709-55791b0741cb-kube-api-access-7p8rr\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.919093 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.918989 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4152b2c2-a19c-44fa-96f5-dc3eb661aa9b-original-pull-secret\") pod \"global-pull-secret-syncer-wbslq\" (UID: \"4152b2c2-a19c-44fa-96f5-dc3eb661aa9b\") " pod="kube-system/global-pull-secret-syncer-wbslq" Sep 16 07:33:50.919093 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.919014 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/96bb36bc-d267-42e2-8509-0c72ab202bdc-iptables-alerter-script\") pod \"iptables-alerter-t458f\" (UID: \"96bb36bc-d267-42e2-8509-0c72ab202bdc\") " pod="openshift-network-operator/iptables-alerter-t458f" Sep 16 07:33:50.920004 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.919019 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/37c8f2c4-2c9b-47f6-b709-55791b0741cb-node-log\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.920004 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.919039 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7-system-cni-dir\") pod \"multus-additional-cni-plugins-mdzsg\" (UID: \"2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7\") " pod="openshift-multus/multus-additional-cni-plugins-mdzsg" Sep 16 07:33:50.920004 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.919065 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/37c8f2c4-2c9b-47f6-b709-55791b0741cb-host-run-netns\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.920004 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.919089 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8cdfedd6-88d2-459f-bf39-504bc65e2fed-cni-binary-copy\") pod \"multus-tf242\" (UID: \"8cdfedd6-88d2-459f-bf39-504bc65e2fed\") " pod="openshift-multus/multus-tf242" Sep 16 07:33:50.920004 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:50.919114 2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 07:33:50.920004 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.919132 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8cdfedd6-88d2-459f-bf39-504bc65e2fed-host-var-lib-cni-bin\") pod \"multus-tf242\" (UID: \"8cdfedd6-88d2-459f-bf39-504bc65e2fed\") " pod="openshift-multus/multus-tf242" Sep 16 07:33:50.920004 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.919159 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8cdfedd6-88d2-459f-bf39-504bc65e2fed-host-run-multus-certs\") pod \"multus-tf242\" (UID: \"8cdfedd6-88d2-459f-bf39-504bc65e2fed\") " pod="openshift-multus/multus-tf242" Sep 16 07:33:50.920004 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:50.919172 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4152b2c2-a19c-44fa-96f5-dc3eb661aa9b-original-pull-secret podName:4152b2c2-a19c-44fa-96f5-dc3eb661aa9b nodeName:}" failed. No retries permitted until 2026-09-16 07:33:51.419158221 +0000 UTC m=+3.002936539 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4152b2c2-a19c-44fa-96f5-dc3eb661aa9b-original-pull-secret") pod "global-pull-secret-syncer-wbslq" (UID: "4152b2c2-a19c-44fa-96f5-dc3eb661aa9b") : object "kube-system"/"original-pull-secret" not registered Sep 16 07:33:50.920004 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.919200 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b58cf99-fce1-4445-a36c-5d624bf3d0a1-etc-kubernetes\") pod \"tuned-v5srt\" (UID: \"9b58cf99-fce1-4445-a36c-5d624bf3d0a1\") " pod="openshift-cluster-node-tuning-operator/tuned-v5srt" Sep 16 07:33:50.920004 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.919243 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/37c8f2c4-2c9b-47f6-b709-55791b0741cb-host-run-netns\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.920004 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.919256 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9b58cf99-fce1-4445-a36c-5d624bf3d0a1-etc-tuned\") pod \"tuned-v5srt\" (UID: \"9b58cf99-fce1-4445-a36c-5d624bf3d0a1\") " pod="openshift-cluster-node-tuning-operator/tuned-v5srt" Sep 16 07:33:50.920004 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.919297 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/96bb36bc-d267-42e2-8509-0c72ab202bdc-host-slash\") pod \"iptables-alerter-t458f\" (UID: \"96bb36bc-d267-42e2-8509-0c72ab202bdc\") " pod="openshift-network-operator/iptables-alerter-t458f" Sep 16 07:33:50.920004 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.919324 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8cdfedd6-88d2-459f-bf39-504bc65e2fed-cnibin\") pod \"multus-tf242\" (UID: \"8cdfedd6-88d2-459f-bf39-504bc65e2fed\") " pod="openshift-multus/multus-tf242" Sep 16 07:33:50.920004 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.919350 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlfk9\" (UniqueName: \"kubernetes.io/projected/2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7-kube-api-access-qlfk9\") pod \"multus-additional-cni-plugins-mdzsg\" (UID: \"2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7\") " pod="openshift-multus/multus-additional-cni-plugins-mdzsg" Sep 16 07:33:50.920004 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.919378 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4152b2c2-a19c-44fa-96f5-dc3eb661aa9b-dbus\") pod \"global-pull-secret-syncer-wbslq\" (UID: \"4152b2c2-a19c-44fa-96f5-dc3eb661aa9b\") " pod="kube-system/global-pull-secret-syncer-wbslq" Sep 16 07:33:50.920004 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.919388 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/96bb36bc-d267-42e2-8509-0c72ab202bdc-host-slash\") pod \"iptables-alerter-t458f\" (UID: \"96bb36bc-d267-42e2-8509-0c72ab202bdc\") " pod="openshift-network-operator/iptables-alerter-t458f" Sep 16 07:33:50.920004 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.919396 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/37c8f2c4-2c9b-47f6-b709-55791b0741cb-ovnkube-script-lib\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.920826 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.919403 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vgb8\" (UniqueName: \"kubernetes.io/projected/aca15303-6546-4428-bb32-539bb25d2486-kube-api-access-7vgb8\") pod \"network-metrics-daemon-94n4h\" (UID: \"aca15303-6546-4428-bb32-539bb25d2486\") " pod="openshift-multus/network-metrics-daemon-94n4h" Sep 16 07:33:50.920826 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.919426 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8cdfedd6-88d2-459f-bf39-504bc65e2fed-host-var-lib-kubelet\") pod \"multus-tf242\" (UID: \"8cdfedd6-88d2-459f-bf39-504bc65e2fed\") " pod="openshift-multus/multus-tf242" Sep 16 07:33:50.920826 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.919475 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4152b2c2-a19c-44fa-96f5-dc3eb661aa9b-dbus\") pod \"global-pull-secret-syncer-wbslq\" (UID: \"4152b2c2-a19c-44fa-96f5-dc3eb661aa9b\") " pod="kube-system/global-pull-secret-syncer-wbslq" Sep 16 07:33:50.920826 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.919540 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9b58cf99-fce1-4445-a36c-5d624bf3d0a1-etc-modprobe-d\") pod \"tuned-v5srt\" (UID: \"9b58cf99-fce1-4445-a36c-5d624bf3d0a1\") " pod="openshift-cluster-node-tuning-operator/tuned-v5srt" Sep 16 07:33:50.920826 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.919580 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9b58cf99-fce1-4445-a36c-5d624bf3d0a1-etc-systemd\") pod \"tuned-v5srt\" (UID: \"9b58cf99-fce1-4445-a36c-5d624bf3d0a1\") " pod="openshift-cluster-node-tuning-operator/tuned-v5srt" Sep 16 07:33:50.920826 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.919599 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/96bb36bc-d267-42e2-8509-0c72ab202bdc-iptables-alerter-script\") pod \"iptables-alerter-t458f\" (UID: \"96bb36bc-d267-42e2-8509-0c72ab202bdc\") " pod="openshift-network-operator/iptables-alerter-t458f" Sep 16 07:33:50.920826 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.919608 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sq2f\" (UniqueName: \"kubernetes.io/projected/9b58cf99-fce1-4445-a36c-5d624bf3d0a1-kube-api-access-7sq2f\") pod \"tuned-v5srt\" (UID: \"9b58cf99-fce1-4445-a36c-5d624bf3d0a1\") " pod="openshift-cluster-node-tuning-operator/tuned-v5srt" Sep 16 07:33:50.920826 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.919645 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/37c8f2c4-2c9b-47f6-b709-55791b0741cb-host-slash\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.920826 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.919699 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37c8f2c4-2c9b-47f6-b709-55791b0741cb-etc-openvswitch\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.920826 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.919702 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/37c8f2c4-2c9b-47f6-b709-55791b0741cb-host-slash\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.920826 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.919735 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37c8f2c4-2c9b-47f6-b709-55791b0741cb-run-openvswitch\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.920826 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.919754 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37c8f2c4-2c9b-47f6-b709-55791b0741cb-etc-openvswitch\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.920826 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.919799 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37c8f2c4-2c9b-47f6-b709-55791b0741cb-host-run-ovn-kubernetes\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.920826 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.919821 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37c8f2c4-2c9b-47f6-b709-55791b0741cb-run-openvswitch\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.920826 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.919857 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/37c8f2c4-2c9b-47f6-b709-55791b0741cb-host-cni-netd\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.920826 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.919878 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ef5303b5-4dc6-4e4c-9716-bcf88a796cae-konnectivity-ca\") pod \"konnectivity-agent-7zl67\" (UID: \"ef5303b5-4dc6-4e4c-9716-bcf88a796cae\") " pod="kube-system/konnectivity-agent-7zl67" Sep 16 07:33:50.920826 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.919903 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/024d7ae0-3a5a-4d5e-bedb-592058d44673-tmp-dir\") pod \"node-resolver-rxbfc\" (UID: \"024d7ae0-3a5a-4d5e-bedb-592058d44673\") " pod="openshift-dns/node-resolver-rxbfc" Sep 16 07:33:50.921544 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.919917 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/37c8f2c4-2c9b-47f6-b709-55791b0741cb-host-cni-netd\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.921544 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.919937 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37c8f2c4-2c9b-47f6-b709-55791b0741cb-host-run-ovn-kubernetes\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.921544 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.919960 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9b58cf99-fce1-4445-a36c-5d624bf3d0a1-etc-sysctl-d\") pod \"tuned-v5srt\" (UID: \"9b58cf99-fce1-4445-a36c-5d624bf3d0a1\") " pod="openshift-cluster-node-tuning-operator/tuned-v5srt" Sep 16 07:33:50.921544 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.919997 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37c8f2c4-2c9b-47f6-b709-55791b0741cb-var-lib-openvswitch\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.921544 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.920016 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b58cf99-fce1-4445-a36c-5d624bf3d0a1-sys\") pod \"tuned-v5srt\" (UID: \"9b58cf99-fce1-4445-a36c-5d624bf3d0a1\") " pod="openshift-cluster-node-tuning-operator/tuned-v5srt" Sep 16 07:33:50.921544 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.920041 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b58cf99-fce1-4445-a36c-5d624bf3d0a1-var-lib-kubelet\") pod \"tuned-v5srt\" (UID: \"9b58cf99-fce1-4445-a36c-5d624bf3d0a1\") " pod="openshift-cluster-node-tuning-operator/tuned-v5srt" Sep 16 07:33:50.921544 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.920073 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/16e6d2a0-df60-4490-a3f2-3c1d79d8cedd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ms2dc\" (UID: \"16e6d2a0-df60-4490-a3f2-3c1d79d8cedd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms2dc" Sep 16 07:33:50.921544 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.920094 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84z6f\" (UniqueName: \"kubernetes.io/projected/16e6d2a0-df60-4490-a3f2-3c1d79d8cedd-kube-api-access-84z6f\") pod \"aws-ebs-csi-driver-node-ms2dc\" (UID: \"16e6d2a0-df60-4490-a3f2-3c1d79d8cedd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms2dc" Sep 16 07:33:50.921544 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.920133 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37c8f2c4-2c9b-47f6-b709-55791b0741cb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.921544 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.920153 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/29d2a484-46de-40a9-be4a-b9c6b266fbae-serviceca\") pod \"node-ca-9xkmz\" (UID: \"29d2a484-46de-40a9-be4a-b9c6b266fbae\") " pod="openshift-image-registry/node-ca-9xkmz" Sep 16 07:33:50.921544 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.920190 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4152b2c2-a19c-44fa-96f5-dc3eb661aa9b-kubelet-config\") pod \"global-pull-secret-syncer-wbslq\" (UID: \"4152b2c2-a19c-44fa-96f5-dc3eb661aa9b\") " pod="kube-system/global-pull-secret-syncer-wbslq" Sep 16 07:33:50.921544 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.920210 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8cdfedd6-88d2-459f-bf39-504bc65e2fed-multus-cni-dir\") pod \"multus-tf242\" (UID: \"8cdfedd6-88d2-459f-bf39-504bc65e2fed\") " pod="openshift-multus/multus-tf242" Sep 16 07:33:50.921544 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.920271 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4152b2c2-a19c-44fa-96f5-dc3eb661aa9b-kubelet-config\") pod \"global-pull-secret-syncer-wbslq\" (UID: \"4152b2c2-a19c-44fa-96f5-dc3eb661aa9b\") " pod="kube-system/global-pull-secret-syncer-wbslq" Sep 16 07:33:50.921544 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.920295 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/37c8f2c4-2c9b-47f6-b709-55791b0741cb-var-lib-openvswitch\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.921544 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.920310 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/37c8f2c4-2c9b-47f6-b709-55791b0741cb-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.921544 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.920333 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8cdfedd6-88d2-459f-bf39-504bc65e2fed-multus-daemon-config\") pod \"multus-tf242\" (UID: \"8cdfedd6-88d2-459f-bf39-504bc65e2fed\") " pod="openshift-multus/multus-tf242" Sep 16 07:33:50.921544 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.920378 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7-cnibin\") pod \"multus-additional-cni-plugins-mdzsg\" (UID: \"2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7\") " pod="openshift-multus/multus-additional-cni-plugins-mdzsg" Sep 16 07:33:50.922344 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.920380 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/ef5303b5-4dc6-4e4c-9716-bcf88a796cae-konnectivity-ca\") pod \"konnectivity-agent-7zl67\" (UID: \"ef5303b5-4dc6-4e4c-9716-bcf88a796cae\") " pod="kube-system/konnectivity-agent-7zl67" Sep 16 07:33:50.922344 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.920414 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/37c8f2c4-2c9b-47f6-b709-55791b0741cb-systemd-units\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.922344 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.920457 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/37c8f2c4-2c9b-47f6-b709-55791b0741cb-systemd-units\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.922344 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.920468 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mdzsg\" (UID: \"2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7\") " pod="openshift-multus/multus-additional-cni-plugins-mdzsg" Sep 16 07:33:50.922344 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.920508 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9b58cf99-fce1-4445-a36c-5d624bf3d0a1-etc-sysctl-conf\") pod \"tuned-v5srt\" (UID: \"9b58cf99-fce1-4445-a36c-5d624bf3d0a1\") " pod="openshift-cluster-node-tuning-operator/tuned-v5srt" Sep 16 07:33:50.922344 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.920536 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8cdfedd6-88d2-459f-bf39-504bc65e2fed-hostroot\") pod \"multus-tf242\" (UID: \"8cdfedd6-88d2-459f-bf39-504bc65e2fed\") " pod="openshift-multus/multus-tf242" Sep 16 07:33:50.922344 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.920659 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/29d2a484-46de-40a9-be4a-b9c6b266fbae-serviceca\") pod \"node-ca-9xkmz\" (UID: \"29d2a484-46de-40a9-be4a-b9c6b266fbae\") " pod="openshift-image-registry/node-ca-9xkmz" Sep 16 07:33:50.923609 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.923582 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/37c8f2c4-2c9b-47f6-b709-55791b0741cb-ovn-node-metrics-cert\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.923910 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.923892 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/ef5303b5-4dc6-4e4c-9716-bcf88a796cae-agent-certs\") pod \"konnectivity-agent-7zl67\" (UID: \"ef5303b5-4dc6-4e4c-9716-bcf88a796cae\") " pod="kube-system/konnectivity-agent-7zl67" Sep 16 07:33:50.926123 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:50.926020 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 07:33:50.926123 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:50.926042 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 07:33:50.926123 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:50.926055 2552 projected.go:196] Error preparing data for projected volume kube-api-access-hnt8f for pod openshift-network-diagnostics/network-check-target-vx6vh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 07:33:50.926123 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:50.926116 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c6593a0-6ec1-4945-b929-f94446ee82b1-kube-api-access-hnt8f podName:3c6593a0-6ec1-4945-b929-f94446ee82b1 nodeName:}" failed. No retries permitted until 2026-09-16 07:33:51.426089297 +0000 UTC m=+3.009867644 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hnt8f" (UniqueName: "kubernetes.io/projected/3c6593a0-6ec1-4945-b929-f94446ee82b1-kube-api-access-hnt8f") pod "network-check-target-vx6vh" (UID: "3c6593a0-6ec1-4945-b929-f94446ee82b1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 07:33:50.926894 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.926869 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fprrz\" (UniqueName: \"kubernetes.io/projected/96bb36bc-d267-42e2-8509-0c72ab202bdc-kube-api-access-fprrz\") pod \"iptables-alerter-t458f\" (UID: \"96bb36bc-d267-42e2-8509-0c72ab202bdc\") " pod="openshift-network-operator/iptables-alerter-t458f" Sep 16 07:33:50.929033 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.927619 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vgb8\" (UniqueName: \"kubernetes.io/projected/aca15303-6546-4428-bb32-539bb25d2486-kube-api-access-7vgb8\") pod \"network-metrics-daemon-94n4h\" (UID: \"aca15303-6546-4428-bb32-539bb25d2486\") " pod="openshift-multus/network-metrics-daemon-94n4h" Sep 16 07:33:50.929033 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.928787 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzbkw\" (UniqueName: \"kubernetes.io/projected/29d2a484-46de-40a9-be4a-b9c6b266fbae-kube-api-access-xzbkw\") pod \"node-ca-9xkmz\" (UID: \"29d2a484-46de-40a9-be4a-b9c6b266fbae\") " pod="openshift-image-registry/node-ca-9xkmz" Sep 16 07:33:50.929033 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.928974 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p8rr\" (UniqueName: \"kubernetes.io/projected/37c8f2c4-2c9b-47f6-b709-55791b0741cb-kube-api-access-7p8rr\") pod \"ovnkube-node-xc9v8\" (UID: \"37c8f2c4-2c9b-47f6-b709-55791b0741cb\") " pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:50.937497 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.936091 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-170.ec2.internal" event={"ID":"1fcac57a8698f66711e8e05f204f0824","Type":"ContainerStarted","Data":"6df5d14bb0026118891632885e78d53a97a502170f56645aa344b9deb6cf5a4f"} Sep 16 07:33:50.937497 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.937284 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-170.ec2.internal" event={"ID":"9ae2c91e0de9f347633243d0f5bfd92f","Type":"ContainerStarted","Data":"cec405866bc7aa0a92325f2f1761bf204538bc8e3930cc77277aa1bcf7d466c8"} Sep 16 07:33:50.970576 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:50.970548 2552 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 16 07:33:51.020983 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.020943 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8cdfedd6-88d2-459f-bf39-504bc65e2fed-cnibin\") pod \"multus-tf242\" (UID: \"8cdfedd6-88d2-459f-bf39-504bc65e2fed\") " pod="openshift-multus/multus-tf242" Sep 16 07:33:51.020983 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.020984 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlfk9\" (UniqueName: \"kubernetes.io/projected/2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7-kube-api-access-qlfk9\") pod \"multus-additional-cni-plugins-mdzsg\" (UID: \"2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7\") " pod="openshift-multus/multus-additional-cni-plugins-mdzsg" Sep 16 07:33:51.021203 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.021005 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8cdfedd6-88d2-459f-bf39-504bc65e2fed-host-var-lib-kubelet\") pod \"multus-tf242\" (UID: \"8cdfedd6-88d2-459f-bf39-504bc65e2fed\") " pod="openshift-multus/multus-tf242" Sep 16 07:33:51.021203 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.021026 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9b58cf99-fce1-4445-a36c-5d624bf3d0a1-etc-modprobe-d\") pod \"tuned-v5srt\" (UID: \"9b58cf99-fce1-4445-a36c-5d624bf3d0a1\") " pod="openshift-cluster-node-tuning-operator/tuned-v5srt" Sep 16 07:33:51.021203 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.021051 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9b58cf99-fce1-4445-a36c-5d624bf3d0a1-etc-systemd\") pod \"tuned-v5srt\" (UID: \"9b58cf99-fce1-4445-a36c-5d624bf3d0a1\") " pod="openshift-cluster-node-tuning-operator/tuned-v5srt" Sep 16 07:33:51.021203 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.021077 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sq2f\" (UniqueName: \"kubernetes.io/projected/9b58cf99-fce1-4445-a36c-5d624bf3d0a1-kube-api-access-7sq2f\") pod \"tuned-v5srt\" (UID: \"9b58cf99-fce1-4445-a36c-5d624bf3d0a1\") " pod="openshift-cluster-node-tuning-operator/tuned-v5srt" Sep 16 07:33:51.021203 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.021079 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8cdfedd6-88d2-459f-bf39-504bc65e2fed-cnibin\") pod \"multus-tf242\" (UID: \"8cdfedd6-88d2-459f-bf39-504bc65e2fed\") " pod="openshift-multus/multus-tf242" Sep 16 07:33:51.021203 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.021121 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/024d7ae0-3a5a-4d5e-bedb-592058d44673-tmp-dir\") pod \"node-resolver-rxbfc\" (UID: \"024d7ae0-3a5a-4d5e-bedb-592058d44673\") " pod="openshift-dns/node-resolver-rxbfc" Sep 16 07:33:51.021203 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.021148 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9b58cf99-fce1-4445-a36c-5d624bf3d0a1-etc-sysctl-d\") pod \"tuned-v5srt\" (UID: \"9b58cf99-fce1-4445-a36c-5d624bf3d0a1\") " pod="openshift-cluster-node-tuning-operator/tuned-v5srt" Sep 16 07:33:51.021203 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.021164 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b58cf99-fce1-4445-a36c-5d624bf3d0a1-sys\") pod \"tuned-v5srt\" (UID: \"9b58cf99-fce1-4445-a36c-5d624bf3d0a1\") " pod="openshift-cluster-node-tuning-operator/tuned-v5srt" Sep 16 07:33:51.021203 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.021177 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b58cf99-fce1-4445-a36c-5d624bf3d0a1-var-lib-kubelet\") pod \"tuned-v5srt\" (UID: \"9b58cf99-fce1-4445-a36c-5d624bf3d0a1\") " pod="openshift-cluster-node-tuning-operator/tuned-v5srt" Sep 16 07:33:51.021203 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.021199 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/16e6d2a0-df60-4490-a3f2-3c1d79d8cedd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ms2dc\" (UID: \"16e6d2a0-df60-4490-a3f2-3c1d79d8cedd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms2dc" Sep 16 07:33:51.021731 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.021221 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84z6f\" (UniqueName: \"kubernetes.io/projected/16e6d2a0-df60-4490-a3f2-3c1d79d8cedd-kube-api-access-84z6f\") pod \"aws-ebs-csi-driver-node-ms2dc\" (UID: \"16e6d2a0-df60-4490-a3f2-3c1d79d8cedd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms2dc" Sep 16 07:33:51.021731 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.021232 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9b58cf99-fce1-4445-a36c-5d624bf3d0a1-etc-modprobe-d\") pod \"tuned-v5srt\" (UID: \"9b58cf99-fce1-4445-a36c-5d624bf3d0a1\") " pod="openshift-cluster-node-tuning-operator/tuned-v5srt" Sep 16 07:33:51.021731 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.021248 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8cdfedd6-88d2-459f-bf39-504bc65e2fed-multus-cni-dir\") pod \"multus-tf242\" (UID: \"8cdfedd6-88d2-459f-bf39-504bc65e2fed\") " pod="openshift-multus/multus-tf242" Sep 16 07:33:51.021731 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.021271 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8cdfedd6-88d2-459f-bf39-504bc65e2fed-multus-daemon-config\") pod \"multus-tf242\" (UID: \"8cdfedd6-88d2-459f-bf39-504bc65e2fed\") " pod="openshift-multus/multus-tf242" Sep 16 07:33:51.021731 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.021299 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7-cnibin\") pod \"multus-additional-cni-plugins-mdzsg\" (UID: \"2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7\") " pod="openshift-multus/multus-additional-cni-plugins-mdzsg" Sep 16 07:33:51.021731 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.021326 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mdzsg\" (UID: \"2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7\") " pod="openshift-multus/multus-additional-cni-plugins-mdzsg" Sep 16 07:33:51.021731 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.021349 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9b58cf99-fce1-4445-a36c-5d624bf3d0a1-etc-sysctl-conf\") pod \"tuned-v5srt\" (UID: \"9b58cf99-fce1-4445-a36c-5d624bf3d0a1\") " pod="openshift-cluster-node-tuning-operator/tuned-v5srt" Sep 16 07:33:51.021731 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.021364 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8cdfedd6-88d2-459f-bf39-504bc65e2fed-hostroot\") pod \"multus-tf242\" (UID: \"8cdfedd6-88d2-459f-bf39-504bc65e2fed\") " pod="openshift-multus/multus-tf242" Sep 16 07:33:51.021731 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.021378 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8cdfedd6-88d2-459f-bf39-504bc65e2fed-host-var-lib-cni-multus\") pod \"multus-tf242\" (UID: \"8cdfedd6-88d2-459f-bf39-504bc65e2fed\") " pod="openshift-multus/multus-tf242" Sep 16 07:33:51.021731 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.021392 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgbbd\" (UniqueName: \"kubernetes.io/projected/8cdfedd6-88d2-459f-bf39-504bc65e2fed-kube-api-access-hgbbd\") pod \"multus-tf242\" (UID: \"8cdfedd6-88d2-459f-bf39-504bc65e2fed\") " pod="openshift-multus/multus-tf242" Sep 16 07:33:51.021731 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.021411 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k7kb\" (UniqueName: \"kubernetes.io/projected/024d7ae0-3a5a-4d5e-bedb-592058d44673-kube-api-access-8k7kb\") pod \"node-resolver-rxbfc\" (UID: \"024d7ae0-3a5a-4d5e-bedb-592058d44673\") " pod="openshift-dns/node-resolver-rxbfc" Sep 16 07:33:51.021731 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.021433 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9b58cf99-fce1-4445-a36c-5d624bf3d0a1-lib-modules\") pod \"tuned-v5srt\" (UID: \"9b58cf99-fce1-4445-a36c-5d624bf3d0a1\") " pod="openshift-cluster-node-tuning-operator/tuned-v5srt" Sep 16 07:33:51.021731 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.021451 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/16e6d2a0-df60-4490-a3f2-3c1d79d8cedd-socket-dir\") pod \"aws-ebs-csi-driver-node-ms2dc\" (UID: \"16e6d2a0-df60-4490-a3f2-3c1d79d8cedd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms2dc" Sep 16 07:33:51.021731 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.021468 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9b58cf99-fce1-4445-a36c-5d624bf3d0a1-run\") pod \"tuned-v5srt\" (UID: \"9b58cf99-fce1-4445-a36c-5d624bf3d0a1\") " pod="openshift-cluster-node-tuning-operator/tuned-v5srt" Sep 16 07:33:51.021731 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.021482 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b58cf99-fce1-4445-a36c-5d624bf3d0a1-tmp\") pod \"tuned-v5srt\" (UID: \"9b58cf99-fce1-4445-a36c-5d624bf3d0a1\") " pod="openshift-cluster-node-tuning-operator/tuned-v5srt" Sep 16 07:33:51.021731 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.021501 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/16e6d2a0-df60-4490-a3f2-3c1d79d8cedd-registration-dir\") pod \"aws-ebs-csi-driver-node-ms2dc\" (UID: \"16e6d2a0-df60-4490-a3f2-3c1d79d8cedd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms2dc" Sep 16 07:33:51.021731 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.021528 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/16e6d2a0-df60-4490-a3f2-3c1d79d8cedd-etc-selinux\") pod \"aws-ebs-csi-driver-node-ms2dc\" (UID: \"16e6d2a0-df60-4490-a3f2-3c1d79d8cedd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms2dc" Sep 16 07:33:51.022516 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.021548 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mdzsg\" (UID: \"2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7\") " pod="openshift-multus/multus-additional-cni-plugins-mdzsg" Sep 16 07:33:51.022516 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.021563 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/024d7ae0-3a5a-4d5e-bedb-592058d44673-hosts-file\") pod \"node-resolver-rxbfc\" (UID: \"024d7ae0-3a5a-4d5e-bedb-592058d44673\") " pod="openshift-dns/node-resolver-rxbfc" Sep 16 07:33:51.022516 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.021576 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/16e6d2a0-df60-4490-a3f2-3c1d79d8cedd-sys-fs\") pod \"aws-ebs-csi-driver-node-ms2dc\" (UID: \"16e6d2a0-df60-4490-a3f2-3c1d79d8cedd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms2dc" Sep 16 07:33:51.022516 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.021606 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8cdfedd6-88d2-459f-bf39-504bc65e2fed-multus-socket-dir-parent\") pod \"multus-tf242\" (UID: \"8cdfedd6-88d2-459f-bf39-504bc65e2fed\") " pod="openshift-multus/multus-tf242" Sep 16 07:33:51.022516 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.021621 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8cdfedd6-88d2-459f-bf39-504bc65e2fed-etc-kubernetes\") pod \"multus-tf242\" (UID: \"8cdfedd6-88d2-459f-bf39-504bc65e2fed\") " pod="openshift-multus/multus-tf242" Sep 16 07:33:51.022516 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.021639 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9b58cf99-fce1-4445-a36c-5d624bf3d0a1-etc-sysconfig\") pod \"tuned-v5srt\" (UID: \"9b58cf99-fce1-4445-a36c-5d624bf3d0a1\") " pod="openshift-cluster-node-tuning-operator/tuned-v5srt" Sep 16 07:33:51.022516 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.021670 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/16e6d2a0-df60-4490-a3f2-3c1d79d8cedd-device-dir\") pod \"aws-ebs-csi-driver-node-ms2dc\" (UID: \"16e6d2a0-df60-4490-a3f2-3c1d79d8cedd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms2dc" Sep 16 07:33:51.022516 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.021696 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8cdfedd6-88d2-459f-bf39-504bc65e2fed-system-cni-dir\") pod \"multus-tf242\" (UID: \"8cdfedd6-88d2-459f-bf39-504bc65e2fed\") " pod="openshift-multus/multus-tf242" Sep 16 07:33:51.022516 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.021720 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8cdfedd6-88d2-459f-bf39-504bc65e2fed-multus-conf-dir\") pod \"multus-tf242\" (UID: \"8cdfedd6-88d2-459f-bf39-504bc65e2fed\") " pod="openshift-multus/multus-tf242" Sep 16 07:33:51.022516 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.021745 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mdzsg\" (UID: \"2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7\") " pod="openshift-multus/multus-additional-cni-plugins-mdzsg" Sep 16 07:33:51.022516 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.021770 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b58cf99-fce1-4445-a36c-5d624bf3d0a1-host\") pod \"tuned-v5srt\" (UID: \"9b58cf99-fce1-4445-a36c-5d624bf3d0a1\") " pod="openshift-cluster-node-tuning-operator/tuned-v5srt" Sep 16 07:33:51.022516 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.021798 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8cdfedd6-88d2-459f-bf39-504bc65e2fed-host-run-k8s-cni-cncf-io\") pod \"multus-tf242\" (UID: \"8cdfedd6-88d2-459f-bf39-504bc65e2fed\") " pod="openshift-multus/multus-tf242" Sep 16 07:33:51.022516 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.021822 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8cdfedd6-88d2-459f-bf39-504bc65e2fed-host-run-netns\") pod \"multus-tf242\" (UID: \"8cdfedd6-88d2-459f-bf39-504bc65e2fed\") " pod="openshift-multus/multus-tf242" Sep 16 07:33:51.022516 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.021862 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7-cni-binary-copy\") pod \"multus-additional-cni-plugins-mdzsg\" (UID: \"2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7\") " pod="openshift-multus/multus-additional-cni-plugins-mdzsg" Sep 16 07:33:51.022516 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.021903 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7-system-cni-dir\") pod \"multus-additional-cni-plugins-mdzsg\" (UID: \"2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7\") " pod="openshift-multus/multus-additional-cni-plugins-mdzsg" Sep 16 07:33:51.022516 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.021929 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8cdfedd6-88d2-459f-bf39-504bc65e2fed-cni-binary-copy\") pod \"multus-tf242\" (UID: \"8cdfedd6-88d2-459f-bf39-504bc65e2fed\") " pod="openshift-multus/multus-tf242" Sep 16 07:33:51.022516 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.021953 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8cdfedd6-88d2-459f-bf39-504bc65e2fed-host-var-lib-cni-bin\") pod \"multus-tf242\" (UID: \"8cdfedd6-88d2-459f-bf39-504bc65e2fed\") " pod="openshift-multus/multus-tf242" Sep 16 07:33:51.023421 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.021978 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8cdfedd6-88d2-459f-bf39-504bc65e2fed-host-run-multus-certs\") pod \"multus-tf242\" (UID: \"8cdfedd6-88d2-459f-bf39-504bc65e2fed\") " pod="openshift-multus/multus-tf242" Sep 16 07:33:51.023421 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.022002 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b58cf99-fce1-4445-a36c-5d624bf3d0a1-etc-kubernetes\") pod \"tuned-v5srt\" (UID: \"9b58cf99-fce1-4445-a36c-5d624bf3d0a1\") " pod="openshift-cluster-node-tuning-operator/tuned-v5srt" Sep 16 07:33:51.023421 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.022024 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9b58cf99-fce1-4445-a36c-5d624bf3d0a1-etc-tuned\") pod \"tuned-v5srt\" (UID: \"9b58cf99-fce1-4445-a36c-5d624bf3d0a1\") " pod="openshift-cluster-node-tuning-operator/tuned-v5srt" Sep 16 07:33:51.023421 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.022036 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/024d7ae0-3a5a-4d5e-bedb-592058d44673-tmp-dir\") pod \"node-resolver-rxbfc\" (UID: \"024d7ae0-3a5a-4d5e-bedb-592058d44673\") " pod="openshift-dns/node-resolver-rxbfc" Sep 16 07:33:51.023421 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.022214 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9b58cf99-fce1-4445-a36c-5d624bf3d0a1-etc-sysctl-d\") pod \"tuned-v5srt\" (UID: \"9b58cf99-fce1-4445-a36c-5d624bf3d0a1\") " pod="openshift-cluster-node-tuning-operator/tuned-v5srt" Sep 16 07:33:51.023421 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.022589 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/16e6d2a0-df60-4490-a3f2-3c1d79d8cedd-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ms2dc\" (UID: \"16e6d2a0-df60-4490-a3f2-3c1d79d8cedd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms2dc" Sep 16 07:33:51.023421 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.022614 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mdzsg\" (UID: \"2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7\") " pod="openshift-multus/multus-additional-cni-plugins-mdzsg" Sep 16 07:33:51.023421 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.022651 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9b58cf99-fce1-4445-a36c-5d624bf3d0a1-sys\") pod \"tuned-v5srt\" (UID: \"9b58cf99-fce1-4445-a36c-5d624bf3d0a1\") " pod="openshift-cluster-node-tuning-operator/tuned-v5srt" Sep 16 07:33:51.023421 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.022703 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9b58cf99-fce1-4445-a36c-5d624bf3d0a1-var-lib-kubelet\") pod \"tuned-v5srt\" (UID: \"9b58cf99-fce1-4445-a36c-5d624bf3d0a1\") " pod="openshift-cluster-node-tuning-operator/tuned-v5srt" Sep 16 07:33:51.023421 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.022753 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/16e6d2a0-df60-4490-a3f2-3c1d79d8cedd-sys-fs\") pod \"aws-ebs-csi-driver-node-ms2dc\" (UID: \"16e6d2a0-df60-4490-a3f2-3c1d79d8cedd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms2dc" Sep 16 07:33:51.023421 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.022758 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9b58cf99-fce1-4445-a36c-5d624bf3d0a1-etc-sysctl-conf\") pod \"tuned-v5srt\" (UID: \"9b58cf99-fce1-4445-a36c-5d624bf3d0a1\") " pod="openshift-cluster-node-tuning-operator/tuned-v5srt" Sep 16 07:33:51.023421 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.022798 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8cdfedd6-88d2-459f-bf39-504bc65e2fed-hostroot\") pod \"multus-tf242\" (UID: \"8cdfedd6-88d2-459f-bf39-504bc65e2fed\") " pod="openshift-multus/multus-tf242" Sep 16 07:33:51.023421 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.022836 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8cdfedd6-88d2-459f-bf39-504bc65e2fed-multus-socket-dir-parent\") pod \"multus-tf242\" (UID: \"8cdfedd6-88d2-459f-bf39-504bc65e2fed\") " pod="openshift-multus/multus-tf242" Sep 16 07:33:51.023421 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.022840 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8cdfedd6-88d2-459f-bf39-504bc65e2fed-host-var-lib-cni-multus\") pod \"multus-tf242\" (UID: \"8cdfedd6-88d2-459f-bf39-504bc65e2fed\") " pod="openshift-multus/multus-tf242" Sep 16 07:33:51.023421 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.022870 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8cdfedd6-88d2-459f-bf39-504bc65e2fed-etc-kubernetes\") pod \"multus-tf242\" (UID: \"8cdfedd6-88d2-459f-bf39-504bc65e2fed\") " pod="openshift-multus/multus-tf242" Sep 16 07:33:51.023421 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.022919 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9b58cf99-fce1-4445-a36c-5d624bf3d0a1-etc-sysconfig\") pod \"tuned-v5srt\" (UID: \"9b58cf99-fce1-4445-a36c-5d624bf3d0a1\") " pod="openshift-cluster-node-tuning-operator/tuned-v5srt" Sep 16 07:33:51.023421 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.022966 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/16e6d2a0-df60-4490-a3f2-3c1d79d8cedd-device-dir\") pod \"aws-ebs-csi-driver-node-ms2dc\" (UID: \"16e6d2a0-df60-4490-a3f2-3c1d79d8cedd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms2dc" Sep 16 07:33:51.023421 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.023046 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8cdfedd6-88d2-459f-bf39-504bc65e2fed-system-cni-dir\") pod \"multus-tf242\" (UID: \"8cdfedd6-88d2-459f-bf39-504bc65e2fed\") " pod="openshift-multus/multus-tf242" Sep 16 07:33:51.024333 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.023083 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8cdfedd6-88d2-459f-bf39-504bc65e2fed-multus-conf-dir\") pod \"multus-tf242\" (UID: \"8cdfedd6-88d2-459f-bf39-504bc65e2fed\") " pod="openshift-multus/multus-tf242" Sep 16 07:33:51.024333 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.023237 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mdzsg\" (UID: \"2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7\") " pod="openshift-multus/multus-additional-cni-plugins-mdzsg" Sep 16 07:33:51.024333 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.023281 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9b58cf99-fce1-4445-a36c-5d624bf3d0a1-lib-modules\") pod \"tuned-v5srt\" (UID: \"9b58cf99-fce1-4445-a36c-5d624bf3d0a1\") " pod="openshift-cluster-node-tuning-operator/tuned-v5srt" Sep 16 07:33:51.024333 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.023289 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9b58cf99-fce1-4445-a36c-5d624bf3d0a1-host\") pod \"tuned-v5srt\" (UID: \"9b58cf99-fce1-4445-a36c-5d624bf3d0a1\") " pod="openshift-cluster-node-tuning-operator/tuned-v5srt" Sep 16 07:33:51.024333 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.023335 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8cdfedd6-88d2-459f-bf39-504bc65e2fed-host-run-k8s-cni-cncf-io\") pod \"multus-tf242\" (UID: \"8cdfedd6-88d2-459f-bf39-504bc65e2fed\") " pod="openshift-multus/multus-tf242" Sep 16 07:33:51.024333 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.023374 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8cdfedd6-88d2-459f-bf39-504bc65e2fed-host-run-netns\") pod \"multus-tf242\" (UID: \"8cdfedd6-88d2-459f-bf39-504bc65e2fed\") " pod="openshift-multus/multus-tf242" Sep 16 07:33:51.024333 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.023390 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/16e6d2a0-df60-4490-a3f2-3c1d79d8cedd-socket-dir\") pod \"aws-ebs-csi-driver-node-ms2dc\" (UID: \"16e6d2a0-df60-4490-a3f2-3c1d79d8cedd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms2dc" Sep 16 07:33:51.024333 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.023446 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9b58cf99-fce1-4445-a36c-5d624bf3d0a1-run\") pod \"tuned-v5srt\" (UID: \"9b58cf99-fce1-4445-a36c-5d624bf3d0a1\") " pod="openshift-cluster-node-tuning-operator/tuned-v5srt" Sep 16 07:33:51.024333 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.023752 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8cdfedd6-88d2-459f-bf39-504bc65e2fed-host-var-lib-cni-bin\") pod \"multus-tf242\" (UID: \"8cdfedd6-88d2-459f-bf39-504bc65e2fed\") " pod="openshift-multus/multus-tf242" Sep 16 07:33:51.024333 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.023800 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7-cni-binary-copy\") pod \"multus-additional-cni-plugins-mdzsg\" (UID: \"2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7\") " pod="openshift-multus/multus-additional-cni-plugins-mdzsg" Sep 16 07:33:51.024333 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.023809 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7-system-cni-dir\") pod \"multus-additional-cni-plugins-mdzsg\" (UID: \"2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7\") " pod="openshift-multus/multus-additional-cni-plugins-mdzsg" Sep 16 07:33:51.024333 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.023864 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8cdfedd6-88d2-459f-bf39-504bc65e2fed-host-run-multus-certs\") pod \"multus-tf242\" (UID: \"8cdfedd6-88d2-459f-bf39-504bc65e2fed\") " pod="openshift-multus/multus-tf242" Sep 16 07:33:51.024333 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.023919 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9b58cf99-fce1-4445-a36c-5d624bf3d0a1-etc-kubernetes\") pod \"tuned-v5srt\" (UID: \"9b58cf99-fce1-4445-a36c-5d624bf3d0a1\") " pod="openshift-cluster-node-tuning-operator/tuned-v5srt" Sep 16 07:33:51.024333 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.024017 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/16e6d2a0-df60-4490-a3f2-3c1d79d8cedd-etc-selinux\") pod \"aws-ebs-csi-driver-node-ms2dc\" (UID: \"16e6d2a0-df60-4490-a3f2-3c1d79d8cedd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms2dc" Sep 16 07:33:51.024333 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.024069 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/16e6d2a0-df60-4490-a3f2-3c1d79d8cedd-registration-dir\") pod \"aws-ebs-csi-driver-node-ms2dc\" (UID: \"16e6d2a0-df60-4490-a3f2-3c1d79d8cedd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms2dc" Sep 16 07:33:51.024333 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.021502 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8cdfedd6-88d2-459f-bf39-504bc65e2fed-host-var-lib-kubelet\") pod \"multus-tf242\" (UID: \"8cdfedd6-88d2-459f-bf39-504bc65e2fed\") " pod="openshift-multus/multus-tf242" Sep 16 07:33:51.024333 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.024130 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7-cnibin\") pod \"multus-additional-cni-plugins-mdzsg\" (UID: \"2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7\") " pod="openshift-multus/multus-additional-cni-plugins-mdzsg" Sep 16 07:33:51.024333 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.024188 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8cdfedd6-88d2-459f-bf39-504bc65e2fed-multus-cni-dir\") pod \"multus-tf242\" (UID: \"8cdfedd6-88d2-459f-bf39-504bc65e2fed\") " pod="openshift-multus/multus-tf242" Sep 16 07:33:51.025232 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.024336 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8cdfedd6-88d2-459f-bf39-504bc65e2fed-cni-binary-copy\") pod \"multus-tf242\" (UID: \"8cdfedd6-88d2-459f-bf39-504bc65e2fed\") " pod="openshift-multus/multus-tf242" Sep 16 07:33:51.025232 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.021561 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9b58cf99-fce1-4445-a36c-5d624bf3d0a1-etc-systemd\") pod \"tuned-v5srt\" (UID: \"9b58cf99-fce1-4445-a36c-5d624bf3d0a1\") " pod="openshift-cluster-node-tuning-operator/tuned-v5srt" Sep 16 07:33:51.025232 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.024408 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/024d7ae0-3a5a-4d5e-bedb-592058d44673-hosts-file\") pod \"node-resolver-rxbfc\" (UID: \"024d7ae0-3a5a-4d5e-bedb-592058d44673\") " pod="openshift-dns/node-resolver-rxbfc" Sep 16 07:33:51.025232 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.024672 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mdzsg\" (UID: \"2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7\") " pod="openshift-multus/multus-additional-cni-plugins-mdzsg" Sep 16 07:33:51.025232 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.024756 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8cdfedd6-88d2-459f-bf39-504bc65e2fed-multus-daemon-config\") pod \"multus-tf242\" (UID: \"8cdfedd6-88d2-459f-bf39-504bc65e2fed\") " pod="openshift-multus/multus-tf242" Sep 16 07:33:51.025726 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.025701 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b58cf99-fce1-4445-a36c-5d624bf3d0a1-tmp\") pod \"tuned-v5srt\" (UID: \"9b58cf99-fce1-4445-a36c-5d624bf3d0a1\") " pod="openshift-cluster-node-tuning-operator/tuned-v5srt" Sep 16 07:33:51.026453 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.026434 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9b58cf99-fce1-4445-a36c-5d624bf3d0a1-etc-tuned\") pod \"tuned-v5srt\" (UID: \"9b58cf99-fce1-4445-a36c-5d624bf3d0a1\") " pod="openshift-cluster-node-tuning-operator/tuned-v5srt" Sep 16 07:33:51.030836 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.030798 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlfk9\" (UniqueName: \"kubernetes.io/projected/2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7-kube-api-access-qlfk9\") pod \"multus-additional-cni-plugins-mdzsg\" (UID: \"2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7\") " pod="openshift-multus/multus-additional-cni-plugins-mdzsg" Sep 16 07:33:51.031477 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.031454 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sq2f\" (UniqueName: \"kubernetes.io/projected/9b58cf99-fce1-4445-a36c-5d624bf3d0a1-kube-api-access-7sq2f\") pod \"tuned-v5srt\" (UID: \"9b58cf99-fce1-4445-a36c-5d624bf3d0a1\") " pod="openshift-cluster-node-tuning-operator/tuned-v5srt" Sep 16 07:33:51.032332 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.032310 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgbbd\" (UniqueName: \"kubernetes.io/projected/8cdfedd6-88d2-459f-bf39-504bc65e2fed-kube-api-access-hgbbd\") pod \"multus-tf242\" (UID: \"8cdfedd6-88d2-459f-bf39-504bc65e2fed\") " pod="openshift-multus/multus-tf242" Sep 16 07:33:51.032502 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.032486 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k7kb\" (UniqueName: \"kubernetes.io/projected/024d7ae0-3a5a-4d5e-bedb-592058d44673-kube-api-access-8k7kb\") pod \"node-resolver-rxbfc\" (UID: \"024d7ae0-3a5a-4d5e-bedb-592058d44673\") " pod="openshift-dns/node-resolver-rxbfc" Sep 16 07:33:51.033081 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.033060 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84z6f\" (UniqueName: \"kubernetes.io/projected/16e6d2a0-df60-4490-a3f2-3c1d79d8cedd-kube-api-access-84z6f\") pod \"aws-ebs-csi-driver-node-ms2dc\" (UID: \"16e6d2a0-df60-4490-a3f2-3c1d79d8cedd\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms2dc" Sep 16 07:33:51.104253 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.104223 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:33:51.124125 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.123935 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9xkmz" Sep 16 07:33:51.137645 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.137627 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-t458f" Sep 16 07:33:51.145344 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.145322 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-7zl67" Sep 16 07:33:51.152869 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.152852 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-v5srt" Sep 16 07:33:51.159453 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.159435 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tf242" Sep 16 07:33:51.167072 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.167055 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mdzsg" Sep 16 07:33:51.174685 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.174667 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms2dc" Sep 16 07:33:51.180678 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.180662 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rxbfc" Sep 16 07:33:51.425408 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.425331 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4152b2c2-a19c-44fa-96f5-dc3eb661aa9b-original-pull-secret\") pod \"global-pull-secret-syncer-wbslq\" (UID: \"4152b2c2-a19c-44fa-96f5-dc3eb661aa9b\") " pod="kube-system/global-pull-secret-syncer-wbslq" Sep 16 07:33:51.425408 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.425401 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aca15303-6546-4428-bb32-539bb25d2486-metrics-certs\") pod \"network-metrics-daemon-94n4h\" (UID: \"aca15303-6546-4428-bb32-539bb25d2486\") " pod="openshift-multus/network-metrics-daemon-94n4h" Sep 16 07:33:51.425626 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:51.425494 2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 07:33:51.425626 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:51.425507 2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 07:33:51.425626 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:51.425569 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aca15303-6546-4428-bb32-539bb25d2486-metrics-certs podName:aca15303-6546-4428-bb32-539bb25d2486 nodeName:}" failed. No retries permitted until 2026-09-16 07:33:52.425550769 +0000 UTC m=+4.009329101 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aca15303-6546-4428-bb32-539bb25d2486-metrics-certs") pod "network-metrics-daemon-94n4h" (UID: "aca15303-6546-4428-bb32-539bb25d2486") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 07:33:51.425626 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:51.425589 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4152b2c2-a19c-44fa-96f5-dc3eb661aa9b-original-pull-secret podName:4152b2c2-a19c-44fa-96f5-dc3eb661aa9b nodeName:}" failed. No retries permitted until 2026-09-16 07:33:52.425580278 +0000 UTC m=+4.009358605 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4152b2c2-a19c-44fa-96f5-dc3eb661aa9b-original-pull-secret") pod "global-pull-secret-syncer-wbslq" (UID: "4152b2c2-a19c-44fa-96f5-dc3eb661aa9b") : object "kube-system"/"original-pull-secret" not registered Sep 16 07:33:51.526475 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.526430 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnt8f\" (UniqueName: \"kubernetes.io/projected/3c6593a0-6ec1-4945-b929-f94446ee82b1-kube-api-access-hnt8f\") pod \"network-check-target-vx6vh\" (UID: \"3c6593a0-6ec1-4945-b929-f94446ee82b1\") " pod="openshift-network-diagnostics/network-check-target-vx6vh" Sep 16 07:33:51.526662 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:51.526640 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 07:33:51.526712 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:51.526672 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 07:33:51.526712 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:51.526687 2552 projected.go:196] Error preparing data for projected volume kube-api-access-hnt8f for pod openshift-network-diagnostics/network-check-target-vx6vh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 07:33:51.526787 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:51.526758 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c6593a0-6ec1-4945-b929-f94446ee82b1-kube-api-access-hnt8f podName:3c6593a0-6ec1-4945-b929-f94446ee82b1 nodeName:}" failed. No retries permitted until 2026-09-16 07:33:52.526735145 +0000 UTC m=+4.110513481 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hnt8f" (UniqueName: "kubernetes.io/projected/3c6593a0-6ec1-4945-b929-f94446ee82b1-kube-api-access-hnt8f") pod "network-check-target-vx6vh" (UID: "3c6593a0-6ec1-4945-b929-f94446ee82b1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 07:33:51.845583 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.845550 2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 07:28:50 +0000 UTC" deadline="2028-04-24 08:20:14.376613821 +0000 UTC" Sep 16 07:33:51.845583 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.845580 2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14064h46m22.531037986s" Sep 16 07:33:51.849059 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:51.849040 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29d2a484_46de_40a9_be4a_b9c6b266fbae.slice/crio-3f2b74d57da15c2c8182f7e3735aa6abd0932377e57a76f53edf6d4c2c8b01e0 WatchSource:0}: Error finding container 3f2b74d57da15c2c8182f7e3735aa6abd0932377e57a76f53edf6d4c2c8b01e0: Status 404 returned error can't find the container with id 3f2b74d57da15c2c8182f7e3735aa6abd0932377e57a76f53edf6d4c2c8b01e0 Sep 16 07:33:51.850005 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:51.849980 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b58cf99_fce1_4445_a36c_5d624bf3d0a1.slice/crio-629637ed2250e84212dc55836254a4e6c65a8857709ae7dda9504922653186d4 WatchSource:0}: Error finding container 629637ed2250e84212dc55836254a4e6c65a8857709ae7dda9504922653186d4: Status 404 returned error can't find the container with id 629637ed2250e84212dc55836254a4e6c65a8857709ae7dda9504922653186d4 Sep 16 07:33:51.852040 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:51.852011 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cdfedd6_88d2_459f_bf39_504bc65e2fed.slice/crio-a3264b9768bfe56b97d3ed7b7480b5681e1bdbf4c4fcb5f69a019be0a310f74f WatchSource:0}: Error finding container a3264b9768bfe56b97d3ed7b7480b5681e1bdbf4c4fcb5f69a019be0a310f74f: Status 404 returned error can't find the container with id a3264b9768bfe56b97d3ed7b7480b5681e1bdbf4c4fcb5f69a019be0a310f74f Sep 16 07:33:51.853709 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:51.853689 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37c8f2c4_2c9b_47f6_b709_55791b0741cb.slice/crio-46522d91cb3066d9fb1649b0c529cf7f68266d0faddc4cc5a8f0bde954374ad8 WatchSource:0}: Error finding container 46522d91cb3066d9fb1649b0c529cf7f68266d0faddc4cc5a8f0bde954374ad8: Status 404 returned error can't find the container with id 46522d91cb3066d9fb1649b0c529cf7f68266d0faddc4cc5a8f0bde954374ad8 Sep 16 07:33:51.855303 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:51.855271 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16e6d2a0_df60_4490_a3f2_3c1d79d8cedd.slice/crio-00596960d8fe69bda3c4d6ffc77af1b70147cc0e2ec63022ff8a27403dbee5f2 WatchSource:0}: Error finding container 00596960d8fe69bda3c4d6ffc77af1b70147cc0e2ec63022ff8a27403dbee5f2: Status 404 returned error can't find the container with id 00596960d8fe69bda3c4d6ffc77af1b70147cc0e2ec63022ff8a27403dbee5f2 Sep 16 07:33:51.860702 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:51.860547 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod024d7ae0_3a5a_4d5e_bedb_592058d44673.slice/crio-73525dad4c780afa48061790035ce3bc43a69e10890cd46e555a8520519e6b71 WatchSource:0}: Error finding container 73525dad4c780afa48061790035ce3bc43a69e10890cd46e555a8520519e6b71: Status 404 returned error can't find the container with id 73525dad4c780afa48061790035ce3bc43a69e10890cd46e555a8520519e6b71 Sep 16 07:33:51.861366 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:33:51.861334 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef5303b5_4dc6_4e4c_9716_bcf88a796cae.slice/crio-75b0ba119f78b27c484d370989344394b67c3791488d59769c1cb94c3ed0ffbc WatchSource:0}: Error finding container 75b0ba119f78b27c484d370989344394b67c3791488d59769c1cb94c3ed0ffbc: Status 404 returned error can't find the container with id 75b0ba119f78b27c484d370989344394b67c3791488d59769c1cb94c3ed0ffbc Sep 16 07:33:51.946213 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.946183 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-170.ec2.internal" event={"ID":"1fcac57a8698f66711e8e05f204f0824","Type":"ContainerStarted","Data":"b5317788ff514f46b805edfb1dc87410dc7990c3d025e1cc3456b2e9fe7f203c"} Sep 16 07:33:51.947827 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.947804 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-t458f" event={"ID":"96bb36bc-d267-42e2-8509-0c72ab202bdc","Type":"ContainerStarted","Data":"0514363e58e01e3295e2261f2b24843d150555454a78f14cac7c6f6fe388e17a"} Sep 16 07:33:51.950360 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.950339 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mdzsg" event={"ID":"2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7","Type":"ContainerStarted","Data":"fbc182fee563db6ebc441851dc1aa52920c1d868512ce6a57c2e628bd2735925"} Sep 16 07:33:51.952750 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.952729 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" event={"ID":"37c8f2c4-2c9b-47f6-b709-55791b0741cb","Type":"ContainerStarted","Data":"46522d91cb3066d9fb1649b0c529cf7f68266d0faddc4cc5a8f0bde954374ad8"} Sep 16 07:33:51.953795 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.953770 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v5srt" event={"ID":"9b58cf99-fce1-4445-a36c-5d624bf3d0a1","Type":"ContainerStarted","Data":"629637ed2250e84212dc55836254a4e6c65a8857709ae7dda9504922653186d4"} Sep 16 07:33:51.954799 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.954779 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7zl67" event={"ID":"ef5303b5-4dc6-4e4c-9716-bcf88a796cae","Type":"ContainerStarted","Data":"75b0ba119f78b27c484d370989344394b67c3791488d59769c1cb94c3ed0ffbc"} Sep 16 07:33:51.955794 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.955754 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rxbfc" event={"ID":"024d7ae0-3a5a-4d5e-bedb-592058d44673","Type":"ContainerStarted","Data":"73525dad4c780afa48061790035ce3bc43a69e10890cd46e555a8520519e6b71"} Sep 16 07:33:51.958637 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.958620 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms2dc" event={"ID":"16e6d2a0-df60-4490-a3f2-3c1d79d8cedd","Type":"ContainerStarted","Data":"00596960d8fe69bda3c4d6ffc77af1b70147cc0e2ec63022ff8a27403dbee5f2"} Sep 16 07:33:51.961159 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.961132 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tf242" event={"ID":"8cdfedd6-88d2-459f-bf39-504bc65e2fed","Type":"ContainerStarted","Data":"a3264b9768bfe56b97d3ed7b7480b5681e1bdbf4c4fcb5f69a019be0a310f74f"} Sep 16 07:33:51.962792 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:51.962774 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9xkmz" event={"ID":"29d2a484-46de-40a9-be4a-b9c6b266fbae","Type":"ContainerStarted","Data":"3f2b74d57da15c2c8182f7e3735aa6abd0932377e57a76f53edf6d4c2c8b01e0"} Sep 16 07:33:52.434132 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:52.433335 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aca15303-6546-4428-bb32-539bb25d2486-metrics-certs\") pod \"network-metrics-daemon-94n4h\" (UID: \"aca15303-6546-4428-bb32-539bb25d2486\") " pod="openshift-multus/network-metrics-daemon-94n4h" Sep 16 07:33:52.434132 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:52.433408 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4152b2c2-a19c-44fa-96f5-dc3eb661aa9b-original-pull-secret\") pod \"global-pull-secret-syncer-wbslq\" (UID: \"4152b2c2-a19c-44fa-96f5-dc3eb661aa9b\") " pod="kube-system/global-pull-secret-syncer-wbslq" Sep 16 07:33:52.434132 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:52.433529 2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 07:33:52.434132 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:52.433591 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4152b2c2-a19c-44fa-96f5-dc3eb661aa9b-original-pull-secret podName:4152b2c2-a19c-44fa-96f5-dc3eb661aa9b nodeName:}" failed. No retries permitted until 2026-09-16 07:33:54.433571877 +0000 UTC m=+6.017350208 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4152b2c2-a19c-44fa-96f5-dc3eb661aa9b-original-pull-secret") pod "global-pull-secret-syncer-wbslq" (UID: "4152b2c2-a19c-44fa-96f5-dc3eb661aa9b") : object "kube-system"/"original-pull-secret" not registered Sep 16 07:33:52.434132 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:52.433990 2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 07:33:52.434132 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:52.434041 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aca15303-6546-4428-bb32-539bb25d2486-metrics-certs podName:aca15303-6546-4428-bb32-539bb25d2486 nodeName:}" failed. No retries permitted until 2026-09-16 07:33:54.4340252 +0000 UTC m=+6.017803532 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aca15303-6546-4428-bb32-539bb25d2486-metrics-certs") pod "network-metrics-daemon-94n4h" (UID: "aca15303-6546-4428-bb32-539bb25d2486") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 07:33:52.535131 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:52.534511 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnt8f\" (UniqueName: \"kubernetes.io/projected/3c6593a0-6ec1-4945-b929-f94446ee82b1-kube-api-access-hnt8f\") pod \"network-check-target-vx6vh\" (UID: \"3c6593a0-6ec1-4945-b929-f94446ee82b1\") " pod="openshift-network-diagnostics/network-check-target-vx6vh" Sep 16 07:33:52.535131 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:52.534692 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 07:33:52.535131 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:52.534713 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 07:33:52.535131 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:52.534725 2552 projected.go:196] Error preparing data for projected volume kube-api-access-hnt8f for pod openshift-network-diagnostics/network-check-target-vx6vh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 07:33:52.535131 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:52.534785 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c6593a0-6ec1-4945-b929-f94446ee82b1-kube-api-access-hnt8f podName:3c6593a0-6ec1-4945-b929-f94446ee82b1 nodeName:}" failed. No retries permitted until 2026-09-16 07:33:54.534766725 +0000 UTC m=+6.118545064 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hnt8f" (UniqueName: "kubernetes.io/projected/3c6593a0-6ec1-4945-b929-f94446ee82b1-kube-api-access-hnt8f") pod "network-check-target-vx6vh" (UID: "3c6593a0-6ec1-4945-b929-f94446ee82b1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 07:33:52.845964 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:52.845838 2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 07:28:50 +0000 UTC" deadline="2028-04-13 15:00:09.944339168 +0000 UTC" Sep 16 07:33:52.845964 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:52.845893 2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13807h26m17.098450288s" Sep 16 07:33:52.933076 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:52.933013 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vx6vh" Sep 16 07:33:52.933247 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:52.933153 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vx6vh" podUID="3c6593a0-6ec1-4945-b929-f94446ee82b1" Sep 16 07:33:52.934049 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:52.933786 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wbslq" Sep 16 07:33:52.934049 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:52.933888 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wbslq" podUID="4152b2c2-a19c-44fa-96f5-dc3eb661aa9b" Sep 16 07:33:52.934049 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:52.933954 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-94n4h" Sep 16 07:33:52.934049 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:52.934034 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-94n4h" podUID="aca15303-6546-4428-bb32-539bb25d2486" Sep 16 07:33:52.972186 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:52.971477 2552 generic.go:356] "Generic (PLEG): container finished" podID="9ae2c91e0de9f347633243d0f5bfd92f" containerID="9ddb2db8a654c099a4b104935fd774dc052fa1db0d9f07c1f4ad656d4f5b0e97" exitCode=0 Sep 16 07:33:52.972186 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:52.971966 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-170.ec2.internal" event={"ID":"9ae2c91e0de9f347633243d0f5bfd92f","Type":"ContainerDied","Data":"9ddb2db8a654c099a4b104935fd774dc052fa1db0d9f07c1f4ad656d4f5b0e97"} Sep 16 07:33:52.986592 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:52.986518 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-170.ec2.internal" podStartSLOduration=2.986498819 podStartE2EDuration="2.986498819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 07:33:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:33:51.96055034 +0000 UTC m=+3.544328678" watchObservedRunningTime="2026-09-16 07:33:52.986498819 +0000 UTC m=+4.570277159" Sep 16 07:33:53.995970 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:53.995792 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-170.ec2.internal" event={"ID":"9ae2c91e0de9f347633243d0f5bfd92f","Type":"ContainerStarted","Data":"0321f556069c21cafec30790daccf417a9528f7949eb9762b1debfebf01dca25"} Sep 16 07:33:54.457116 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:54.455186 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aca15303-6546-4428-bb32-539bb25d2486-metrics-certs\") pod \"network-metrics-daemon-94n4h\" (UID: \"aca15303-6546-4428-bb32-539bb25d2486\") " pod="openshift-multus/network-metrics-daemon-94n4h" Sep 16 07:33:54.457116 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:54.455260 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4152b2c2-a19c-44fa-96f5-dc3eb661aa9b-original-pull-secret\") pod \"global-pull-secret-syncer-wbslq\" (UID: \"4152b2c2-a19c-44fa-96f5-dc3eb661aa9b\") " pod="kube-system/global-pull-secret-syncer-wbslq" Sep 16 07:33:54.464313 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:54.457444 2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 07:33:54.464313 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:54.457699 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4152b2c2-a19c-44fa-96f5-dc3eb661aa9b-original-pull-secret podName:4152b2c2-a19c-44fa-96f5-dc3eb661aa9b nodeName:}" failed. No retries permitted until 2026-09-16 07:33:58.457666735 +0000 UTC m=+10.041445056 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4152b2c2-a19c-44fa-96f5-dc3eb661aa9b-original-pull-secret") pod "global-pull-secret-syncer-wbslq" (UID: "4152b2c2-a19c-44fa-96f5-dc3eb661aa9b") : object "kube-system"/"original-pull-secret" not registered Sep 16 07:33:54.464313 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:54.461229 2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 07:33:54.464313 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:54.461291 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aca15303-6546-4428-bb32-539bb25d2486-metrics-certs podName:aca15303-6546-4428-bb32-539bb25d2486 nodeName:}" failed. No retries permitted until 2026-09-16 07:33:58.461275948 +0000 UTC m=+10.045054269 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aca15303-6546-4428-bb32-539bb25d2486-metrics-certs") pod "network-metrics-daemon-94n4h" (UID: "aca15303-6546-4428-bb32-539bb25d2486") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 07:33:54.556714 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:54.556066 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnt8f\" (UniqueName: \"kubernetes.io/projected/3c6593a0-6ec1-4945-b929-f94446ee82b1-kube-api-access-hnt8f\") pod \"network-check-target-vx6vh\" (UID: \"3c6593a0-6ec1-4945-b929-f94446ee82b1\") " pod="openshift-network-diagnostics/network-check-target-vx6vh" Sep 16 07:33:54.556714 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:54.556266 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 07:33:54.556714 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:54.556287 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 07:33:54.556714 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:54.556301 2552 projected.go:196] Error preparing data for projected volume kube-api-access-hnt8f for pod openshift-network-diagnostics/network-check-target-vx6vh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 07:33:54.556714 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:54.556361 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c6593a0-6ec1-4945-b929-f94446ee82b1-kube-api-access-hnt8f podName:3c6593a0-6ec1-4945-b929-f94446ee82b1 nodeName:}" failed. No retries permitted until 2026-09-16 07:33:58.55634212 +0000 UTC m=+10.140120442 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hnt8f" (UniqueName: "kubernetes.io/projected/3c6593a0-6ec1-4945-b929-f94446ee82b1-kube-api-access-hnt8f") pod "network-check-target-vx6vh" (UID: "3c6593a0-6ec1-4945-b929-f94446ee82b1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 07:33:54.932477 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:54.932402 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wbslq" Sep 16 07:33:54.932631 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:54.932527 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wbslq" podUID="4152b2c2-a19c-44fa-96f5-dc3eb661aa9b" Sep 16 07:33:54.932930 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:54.932897 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vx6vh" Sep 16 07:33:54.933030 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:54.933001 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vx6vh" podUID="3c6593a0-6ec1-4945-b929-f94446ee82b1" Sep 16 07:33:54.933087 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:54.933043 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-94n4h" Sep 16 07:33:54.933207 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:54.933162 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-94n4h" podUID="aca15303-6546-4428-bb32-539bb25d2486" Sep 16 07:33:56.931012 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:56.930972 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vx6vh" Sep 16 07:33:56.931375 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:56.931122 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vx6vh" podUID="3c6593a0-6ec1-4945-b929-f94446ee82b1" Sep 16 07:33:56.931512 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:56.931491 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wbslq" Sep 16 07:33:56.931613 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:56.931593 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wbslq" podUID="4152b2c2-a19c-44fa-96f5-dc3eb661aa9b" Sep 16 07:33:56.931753 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:56.931737 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-94n4h" Sep 16 07:33:56.931861 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:56.931841 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-94n4h" podUID="aca15303-6546-4428-bb32-539bb25d2486" Sep 16 07:33:58.489319 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:58.489288 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aca15303-6546-4428-bb32-539bb25d2486-metrics-certs\") pod \"network-metrics-daemon-94n4h\" (UID: \"aca15303-6546-4428-bb32-539bb25d2486\") " pod="openshift-multus/network-metrics-daemon-94n4h" Sep 16 07:33:58.489769 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:58.489350 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4152b2c2-a19c-44fa-96f5-dc3eb661aa9b-original-pull-secret\") pod \"global-pull-secret-syncer-wbslq\" (UID: \"4152b2c2-a19c-44fa-96f5-dc3eb661aa9b\") " pod="kube-system/global-pull-secret-syncer-wbslq" Sep 16 07:33:58.489769 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:58.489453 2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 07:33:58.489769 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:58.489481 2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 07:33:58.489769 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:58.489522 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aca15303-6546-4428-bb32-539bb25d2486-metrics-certs podName:aca15303-6546-4428-bb32-539bb25d2486 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:06.489503166 +0000 UTC m=+18.073281485 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aca15303-6546-4428-bb32-539bb25d2486-metrics-certs") pod "network-metrics-daemon-94n4h" (UID: "aca15303-6546-4428-bb32-539bb25d2486") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 07:33:58.489769 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:58.489538 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4152b2c2-a19c-44fa-96f5-dc3eb661aa9b-original-pull-secret podName:4152b2c2-a19c-44fa-96f5-dc3eb661aa9b nodeName:}" failed. No retries permitted until 2026-09-16 07:34:06.489531537 +0000 UTC m=+18.073309853 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4152b2c2-a19c-44fa-96f5-dc3eb661aa9b-original-pull-secret") pod "global-pull-secret-syncer-wbslq" (UID: "4152b2c2-a19c-44fa-96f5-dc3eb661aa9b") : object "kube-system"/"original-pull-secret" not registered Sep 16 07:33:58.590567 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:58.590531 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnt8f\" (UniqueName: \"kubernetes.io/projected/3c6593a0-6ec1-4945-b929-f94446ee82b1-kube-api-access-hnt8f\") pod \"network-check-target-vx6vh\" (UID: \"3c6593a0-6ec1-4945-b929-f94446ee82b1\") " pod="openshift-network-diagnostics/network-check-target-vx6vh" Sep 16 07:33:58.590740 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:58.590683 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 07:33:58.590740 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:58.590706 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 07:33:58.590740 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:58.590719 2552 projected.go:196] Error preparing data for projected volume kube-api-access-hnt8f for pod openshift-network-diagnostics/network-check-target-vx6vh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 07:33:58.590916 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:58.590775 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c6593a0-6ec1-4945-b929-f94446ee82b1-kube-api-access-hnt8f podName:3c6593a0-6ec1-4945-b929-f94446ee82b1 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:06.590757593 +0000 UTC m=+18.174535914 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hnt8f" (UniqueName: "kubernetes.io/projected/3c6593a0-6ec1-4945-b929-f94446ee82b1-kube-api-access-hnt8f") pod "network-check-target-vx6vh" (UID: "3c6593a0-6ec1-4945-b929-f94446ee82b1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 07:33:58.931650 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:58.931345 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vx6vh" Sep 16 07:33:58.931802 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:58.931659 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vx6vh" podUID="3c6593a0-6ec1-4945-b929-f94446ee82b1" Sep 16 07:33:58.931890 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:58.931866 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wbslq" Sep 16 07:33:58.932037 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:58.931998 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wbslq" podUID="4152b2c2-a19c-44fa-96f5-dc3eb661aa9b" Sep 16 07:33:58.932192 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:33:58.932168 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-94n4h" Sep 16 07:33:58.932314 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:33:58.932278 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-94n4h" podUID="aca15303-6546-4428-bb32-539bb25d2486" Sep 16 07:34:00.930920 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:00.930890 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-94n4h" Sep 16 07:34:00.931368 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:00.930890 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wbslq" Sep 16 07:34:00.931368 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:00.931028 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-94n4h" podUID="aca15303-6546-4428-bb32-539bb25d2486" Sep 16 07:34:00.931368 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:00.930895 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vx6vh" Sep 16 07:34:00.931368 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:00.931114 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wbslq" podUID="4152b2c2-a19c-44fa-96f5-dc3eb661aa9b" Sep 16 07:34:00.931368 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:00.931269 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vx6vh" podUID="3c6593a0-6ec1-4945-b929-f94446ee82b1" Sep 16 07:34:02.930696 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:02.930603 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vx6vh" Sep 16 07:34:02.931141 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:02.930620 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wbslq" Sep 16 07:34:02.931141 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:02.930728 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vx6vh" podUID="3c6593a0-6ec1-4945-b929-f94446ee82b1" Sep 16 07:34:02.931141 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:02.930624 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-94n4h" Sep 16 07:34:02.931141 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:02.930812 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wbslq" podUID="4152b2c2-a19c-44fa-96f5-dc3eb661aa9b" Sep 16 07:34:02.931141 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:02.930899 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-94n4h" podUID="aca15303-6546-4428-bb32-539bb25d2486" Sep 16 07:34:04.931620 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:04.931177 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wbslq" Sep 16 07:34:04.931620 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:04.931214 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vx6vh" Sep 16 07:34:04.931620 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:04.931184 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-94n4h" Sep 16 07:34:04.931620 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:04.931313 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wbslq" podUID="4152b2c2-a19c-44fa-96f5-dc3eb661aa9b" Sep 16 07:34:04.931620 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:04.931374 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-94n4h" podUID="aca15303-6546-4428-bb32-539bb25d2486" Sep 16 07:34:04.931620 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:04.931438 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vx6vh" podUID="3c6593a0-6ec1-4945-b929-f94446ee82b1" Sep 16 07:34:06.548179 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:06.548138 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aca15303-6546-4428-bb32-539bb25d2486-metrics-certs\") pod \"network-metrics-daemon-94n4h\" (UID: \"aca15303-6546-4428-bb32-539bb25d2486\") " pod="openshift-multus/network-metrics-daemon-94n4h" Sep 16 07:34:06.548668 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:06.548202 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4152b2c2-a19c-44fa-96f5-dc3eb661aa9b-original-pull-secret\") pod \"global-pull-secret-syncer-wbslq\" (UID: \"4152b2c2-a19c-44fa-96f5-dc3eb661aa9b\") " pod="kube-system/global-pull-secret-syncer-wbslq" Sep 16 07:34:06.548668 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:06.548302 2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 07:34:06.548668 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:06.548356 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4152b2c2-a19c-44fa-96f5-dc3eb661aa9b-original-pull-secret podName:4152b2c2-a19c-44fa-96f5-dc3eb661aa9b nodeName:}" failed. No retries permitted until 2026-09-16 07:34:22.548342022 +0000 UTC m=+34.132120338 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4152b2c2-a19c-44fa-96f5-dc3eb661aa9b-original-pull-secret") pod "global-pull-secret-syncer-wbslq" (UID: "4152b2c2-a19c-44fa-96f5-dc3eb661aa9b") : object "kube-system"/"original-pull-secret" not registered Sep 16 07:34:06.548668 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:06.548300 2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 07:34:06.548668 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:06.548418 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aca15303-6546-4428-bb32-539bb25d2486-metrics-certs podName:aca15303-6546-4428-bb32-539bb25d2486 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:22.548406738 +0000 UTC m=+34.132185072 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aca15303-6546-4428-bb32-539bb25d2486-metrics-certs") pod "network-metrics-daemon-94n4h" (UID: "aca15303-6546-4428-bb32-539bb25d2486") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 07:34:06.649442 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:06.649392 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnt8f\" (UniqueName: \"kubernetes.io/projected/3c6593a0-6ec1-4945-b929-f94446ee82b1-kube-api-access-hnt8f\") pod \"network-check-target-vx6vh\" (UID: \"3c6593a0-6ec1-4945-b929-f94446ee82b1\") " pod="openshift-network-diagnostics/network-check-target-vx6vh" Sep 16 07:34:06.649644 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:06.649556 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 07:34:06.649644 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:06.649578 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 07:34:06.649644 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:06.649590 2552 projected.go:196] Error preparing data for projected volume kube-api-access-hnt8f for pod openshift-network-diagnostics/network-check-target-vx6vh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 07:34:06.649807 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:06.649659 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c6593a0-6ec1-4945-b929-f94446ee82b1-kube-api-access-hnt8f podName:3c6593a0-6ec1-4945-b929-f94446ee82b1 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:22.64963861 +0000 UTC m=+34.233416932 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hnt8f" (UniqueName: "kubernetes.io/projected/3c6593a0-6ec1-4945-b929-f94446ee82b1-kube-api-access-hnt8f") pod "network-check-target-vx6vh" (UID: "3c6593a0-6ec1-4945-b929-f94446ee82b1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 07:34:06.931546 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:06.931463 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wbslq" Sep 16 07:34:06.931721 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:06.931463 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vx6vh" Sep 16 07:34:06.931721 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:06.931602 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wbslq" podUID="4152b2c2-a19c-44fa-96f5-dc3eb661aa9b" Sep 16 07:34:06.931721 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:06.931635 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-94n4h" Sep 16 07:34:06.931878 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:06.931739 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vx6vh" podUID="3c6593a0-6ec1-4945-b929-f94446ee82b1" Sep 16 07:34:06.931878 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:06.931810 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-94n4h" podUID="aca15303-6546-4428-bb32-539bb25d2486" Sep 16 07:34:08.931865 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:08.931829 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vx6vh" Sep 16 07:34:08.932376 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:08.931938 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vx6vh" podUID="3c6593a0-6ec1-4945-b929-f94446ee82b1" Sep 16 07:34:08.932376 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:08.932034 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wbslq" Sep 16 07:34:08.932376 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:08.932166 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wbslq" podUID="4152b2c2-a19c-44fa-96f5-dc3eb661aa9b" Sep 16 07:34:08.932376 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:08.932216 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-94n4h" Sep 16 07:34:08.932376 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:08.932295 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-94n4h" podUID="aca15303-6546-4428-bb32-539bb25d2486" Sep 16 07:34:10.029328 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:10.029237 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-v5srt" event={"ID":"9b58cf99-fce1-4445-a36c-5d624bf3d0a1","Type":"ContainerStarted","Data":"e4ba1aebcd87a3126dc2720ed2db156db6b4634568e14a4c33cdc5832bf07815"} Sep 16 07:34:10.030537 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:10.030513 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-7zl67" event={"ID":"ef5303b5-4dc6-4e4c-9716-bcf88a796cae","Type":"ContainerStarted","Data":"478cfead1107e9c1fb8a74ada4eee5864b17798636783ff4db3885184a6e4f42"} Sep 16 07:34:10.031964 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:10.031932 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rxbfc" event={"ID":"024d7ae0-3a5a-4d5e-bedb-592058d44673","Type":"ContainerStarted","Data":"a14d1cb933c98b4e10177cb400b2d46ff46cb276d582e5f9d58cd4b8fbc1ff3e"} Sep 16 07:34:10.033626 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:10.033601 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms2dc" event={"ID":"16e6d2a0-df60-4490-a3f2-3c1d79d8cedd","Type":"ContainerStarted","Data":"871fb4445de25f20d1c151358dbfebe1d9bf886b380ecc87334cd73e7906efae"} Sep 16 07:34:10.035312 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:10.035288 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tf242" event={"ID":"8cdfedd6-88d2-459f-bf39-504bc65e2fed","Type":"ContainerStarted","Data":"efbdc7368693703af128d27a239116e5565bd2d66d27a902bb6cf20708a314b2"} Sep 16 07:34:10.036911 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:10.036876 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9xkmz" event={"ID":"29d2a484-46de-40a9-be4a-b9c6b266fbae","Type":"ContainerStarted","Data":"51fc328c2749add031ee0551c7beb58e2aa5dca95dfc8e5feeb943be7afb4595"} Sep 16 07:34:10.038623 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:10.038603 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mdzsg" event={"ID":"2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7","Type":"ContainerStarted","Data":"d2e0bcbd76e5734513047ad53236791a749c6ad57a8ed5a4db932b35d72e5f21"} Sep 16 07:34:10.045988 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:10.045950 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-170.ec2.internal" podStartSLOduration=20.045937074 podStartE2EDuration="20.045937074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 07:33:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:33:54.01203707 +0000 UTC m=+5.595815409" watchObservedRunningTime="2026-09-16 07:34:10.045937074 +0000 UTC m=+21.629715412" Sep 16 07:34:10.046397 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:10.046368 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-v5srt" podStartSLOduration=3.252416715 podStartE2EDuration="21.046358386s" totalImagesPullingTime="17.793941671s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 07:33:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:10.045672631 +0000 UTC m=+21.629450981" watchObservedRunningTime="2026-09-16 07:34:10.046358386 +0000 UTC m=+21.630136721" Sep 16 07:34:10.079440 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:10.079389 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-7zl67" podStartSLOduration=3.298153756 podStartE2EDuration="21.079367631s" totalImagesPullingTime="17.781213875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:33:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:10.079306371 +0000 UTC m=+21.663084710" watchObservedRunningTime="2026-09-16 07:34:10.079367631 +0000 UTC m=+21.663145970" Sep 16 07:34:10.096314 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:10.096258 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-tf242" podStartSLOduration=3.26979759 podStartE2EDuration="21.096236882s" totalImagesPullingTime="17.826439292s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 07:33:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:10.096167575 +0000 UTC m=+21.679945913" watchObservedRunningTime="2026-09-16 07:34:10.096236882 +0000 UTC m=+21.680015220" Sep 16 07:34:10.112405 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:10.112352 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-rxbfc" podStartSLOduration=3.329819558 podStartE2EDuration="21.112331469s" totalImagesPullingTime="17.782511911s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 07:33:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:10.11204559 +0000 UTC m=+21.695823922" watchObservedRunningTime="2026-09-16 07:34:10.112331469 +0000 UTC m=+21.696109810" Sep 16 07:34:10.126940 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:10.126807 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9xkmz" podStartSLOduration=7.255583371 podStartE2EDuration="21.126792117s" totalImagesPullingTime="13.871208746s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 07:33:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:10.126366292 +0000 UTC m=+21.710144629" watchObservedRunningTime="2026-09-16 07:34:10.126792117 +0000 UTC m=+21.710570454" Sep 16 07:34:10.930990 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:10.930953 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wbslq" Sep 16 07:34:10.931203 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:10.931009 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vx6vh" Sep 16 07:34:10.931203 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:10.931087 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-94n4h" Sep 16 07:34:10.931331 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:10.931210 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wbslq" podUID="4152b2c2-a19c-44fa-96f5-dc3eb661aa9b" Sep 16 07:34:10.931512 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:10.931479 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vx6vh" podUID="3c6593a0-6ec1-4945-b929-f94446ee82b1" Sep 16 07:34:10.931624 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:10.931593 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-94n4h" podUID="aca15303-6546-4428-bb32-539bb25d2486" Sep 16 07:34:11.043845 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:11.043803 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-t458f" event={"ID":"96bb36bc-d267-42e2-8509-0c72ab202bdc","Type":"ContainerStarted","Data":"57a0f3202d4a0e1013dc15617a0ddaaca5475c0e1d586396a7643ac57dcdca13"} Sep 16 07:34:11.049275 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:11.049088 2552 generic.go:356] "Generic (PLEG): container finished" podID="2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7" containerID="d2e0bcbd76e5734513047ad53236791a749c6ad57a8ed5a4db932b35d72e5f21" exitCode=0 Sep 16 07:34:11.049275 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:11.049208 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mdzsg" event={"ID":"2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7","Type":"ContainerDied","Data":"d2e0bcbd76e5734513047ad53236791a749c6ad57a8ed5a4db932b35d72e5f21"} Sep 16 07:34:11.053084 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:11.053050 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" event={"ID":"37c8f2c4-2c9b-47f6-b709-55791b0741cb","Type":"ContainerStarted","Data":"23fb3859f5d6ad7eb085c3a0cafd9c5e93c5dac33a7f103133c826fea9df1e53"} Sep 16 07:34:11.053225 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:11.053151 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" event={"ID":"37c8f2c4-2c9b-47f6-b709-55791b0741cb","Type":"ContainerStarted","Data":"6a315f124cba72b358ad5eaa239f5d56e3d9e9fec443c939c7ec469d7720a3bc"} Sep 16 07:34:11.053225 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:11.053205 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" event={"ID":"37c8f2c4-2c9b-47f6-b709-55791b0741cb","Type":"ContainerStarted","Data":"8d1121b5fdbe23243359adf950747a4e3f990e0d4130b815603cc21daf673089"} Sep 16 07:34:11.053330 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:11.053255 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" event={"ID":"37c8f2c4-2c9b-47f6-b709-55791b0741cb","Type":"ContainerStarted","Data":"5bba2339a1d8b78bdbedf485817a10ae3f127b0dba4524165429d3db34b50d98"} Sep 16 07:34:11.053330 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:11.053304 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" event={"ID":"37c8f2c4-2c9b-47f6-b709-55791b0741cb","Type":"ContainerStarted","Data":"53c070bb85626883efe7098a801cc4932fb3ab80a4d1d675ed161d6f3914e31d"} Sep 16 07:34:11.053436 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:11.053349 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" event={"ID":"37c8f2c4-2c9b-47f6-b709-55791b0741cb","Type":"ContainerStarted","Data":"ad1d48c52b13cf56d00f565ee25a5d68a7997a184ea96f18be3527319da7b2db"} Sep 16 07:34:11.061309 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:11.061246 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-t458f" podStartSLOduration=4.277125105 podStartE2EDuration="22.061230628s" totalImagesPullingTime="17.784105523s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 07:33:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:11.06098322 +0000 UTC m=+22.644761561" watchObservedRunningTime="2026-09-16 07:34:11.061230628 +0000 UTC m=+22.645008965" Sep 16 07:34:11.104154 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:11.103916 2552 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 16 07:34:11.873988 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:11.873857 2552 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T07:34:11.104150252Z","UUID":"9573ea21-3a18-4a0d-8b28-c86f35cab81e","Handler":null,"Name":"","Endpoint":""} Sep 16 07:34:11.875752 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:11.875730 2552 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 16 07:34:11.875910 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:11.875763 2552 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 16 07:34:12.057626 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:12.057587 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms2dc" event={"ID":"16e6d2a0-df60-4490-a3f2-3c1d79d8cedd","Type":"ContainerStarted","Data":"b14fb6bbf3b0ed4db93ef0babe69aaa600aee9f3d26e4cd1621834d464277458"} Sep 16 07:34:12.930823 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:12.930784 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wbslq" Sep 16 07:34:12.931006 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:12.930784 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-94n4h" Sep 16 07:34:12.931006 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:12.930924 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vx6vh" Sep 16 07:34:12.931006 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:12.930919 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wbslq" podUID="4152b2c2-a19c-44fa-96f5-dc3eb661aa9b" Sep 16 07:34:12.931161 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:12.931056 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-94n4h" podUID="aca15303-6546-4428-bb32-539bb25d2486" Sep 16 07:34:12.931220 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:12.931158 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vx6vh" podUID="3c6593a0-6ec1-4945-b929-f94446ee82b1" Sep 16 07:34:13.062237 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:13.062201 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms2dc" event={"ID":"16e6d2a0-df60-4490-a3f2-3c1d79d8cedd","Type":"ContainerStarted","Data":"5105d52c96208959f9a49dd3e31ba9d97019ea22092bd7651002a628399e3a5f"} Sep 16 07:34:13.081160 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:13.081094 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ms2dc" podStartSLOduration=4.070566134 podStartE2EDuration="24.081073569s" totalImagesPullingTime="20.010507435s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 07:33:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:13.080535316 +0000 UTC m=+24.664313656" watchObservedRunningTime="2026-09-16 07:34:13.081073569 +0000 UTC m=+24.664851920" Sep 16 07:34:13.763365 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:13.763329 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-7zl67" Sep 16 07:34:13.764053 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:13.764031 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-7zl67" Sep 16 07:34:14.065725 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:14.065624 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-7zl67" Sep 16 07:34:14.066114 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:14.065984 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-7zl67" Sep 16 07:34:14.930471 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:14.930390 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vx6vh" Sep 16 07:34:14.930621 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:14.930390 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-94n4h" Sep 16 07:34:14.930621 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:14.930528 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vx6vh" podUID="3c6593a0-6ec1-4945-b929-f94446ee82b1" Sep 16 07:34:14.930621 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:14.930392 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wbslq" Sep 16 07:34:14.930621 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:14.930595 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-94n4h" podUID="aca15303-6546-4428-bb32-539bb25d2486" Sep 16 07:34:14.930779 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:14.930656 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wbslq" podUID="4152b2c2-a19c-44fa-96f5-dc3eb661aa9b" Sep 16 07:34:15.068924 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:15.068892 2552 generic.go:356] "Generic (PLEG): container finished" podID="2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7" containerID="1ae73d89675e6661db0b6e6ee2a3099a2baa45ab02cdab7a8b518285bb032136" exitCode=0 Sep 16 07:34:15.069383 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:15.068993 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mdzsg" event={"ID":"2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7","Type":"ContainerDied","Data":"1ae73d89675e6661db0b6e6ee2a3099a2baa45ab02cdab7a8b518285bb032136"} Sep 16 07:34:15.073027 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:15.073001 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" event={"ID":"37c8f2c4-2c9b-47f6-b709-55791b0741cb","Type":"ContainerStarted","Data":"bed8965bbedddf912f22b68e508fc87efb4972c71eb427771d128bd87a9ab87d"} Sep 16 07:34:16.076979 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:16.076932 2552 generic.go:356] "Generic (PLEG): container finished" podID="2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7" containerID="f7a966e5c03aa119cbf035b5348cb7d70288dcb3ee57513c165d34ec4bdb66e4" exitCode=0 Sep 16 07:34:16.077368 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:16.077015 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mdzsg" event={"ID":"2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7","Type":"ContainerDied","Data":"f7a966e5c03aa119cbf035b5348cb7d70288dcb3ee57513c165d34ec4bdb66e4"} Sep 16 07:34:16.931476 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:16.931220 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-94n4h" Sep 16 07:34:16.931624 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:16.931220 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wbslq" Sep 16 07:34:16.931624 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:16.931597 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-94n4h" podUID="aca15303-6546-4428-bb32-539bb25d2486" Sep 16 07:34:16.931750 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:16.931331 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vx6vh" Sep 16 07:34:16.931750 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:16.931682 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wbslq" podUID="4152b2c2-a19c-44fa-96f5-dc3eb661aa9b" Sep 16 07:34:16.931843 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:16.931745 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vx6vh" podUID="3c6593a0-6ec1-4945-b929-f94446ee82b1" Sep 16 07:34:17.081328 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:17.081294 2552 generic.go:356] "Generic (PLEG): container finished" podID="2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7" containerID="f62d1b1c5eed601c4a8d50dfbbfbf359b4695c53d44f341d6ee9d29d5c7dd69c" exitCode=0 Sep 16 07:34:17.082152 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:17.081373 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mdzsg" event={"ID":"2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7","Type":"ContainerDied","Data":"f62d1b1c5eed601c4a8d50dfbbfbf359b4695c53d44f341d6ee9d29d5c7dd69c"} Sep 16 07:34:17.085276 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:17.085253 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" event={"ID":"37c8f2c4-2c9b-47f6-b709-55791b0741cb","Type":"ContainerStarted","Data":"3e3064154727c6635c3e862063e89f62f447136be853597ded6d3330b78020a4"} Sep 16 07:34:17.085566 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:17.085545 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:34:17.085640 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:17.085591 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:34:17.085640 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:17.085630 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:34:17.101211 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:17.101174 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:34:17.101344 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:17.101286 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:34:17.132854 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:17.131091 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" podStartSLOduration=10.021204474 podStartE2EDuration="28.1310701s" totalImagesPullingTime="18.109865626s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 07:33:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:17.129043856 +0000 UTC m=+28.712822193" watchObservedRunningTime="2026-09-16 07:34:17.1310701 +0000 UTC m=+28.714848439" Sep 16 07:34:18.503595 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:18.503316 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wbslq"] Sep 16 07:34:18.504162 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:18.503747 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wbslq" Sep 16 07:34:18.504162 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:18.503844 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wbslq" podUID="4152b2c2-a19c-44fa-96f5-dc3eb661aa9b" Sep 16 07:34:18.506482 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:18.506453 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vx6vh"] Sep 16 07:34:18.506623 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:18.506571 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vx6vh" Sep 16 07:34:18.506694 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:18.506666 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vx6vh" podUID="3c6593a0-6ec1-4945-b929-f94446ee82b1" Sep 16 07:34:18.508024 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:18.508001 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-94n4h"] Sep 16 07:34:18.508158 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:18.508125 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-94n4h" Sep 16 07:34:18.508256 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:18.508230 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-94n4h" podUID="aca15303-6546-4428-bb32-539bb25d2486" Sep 16 07:34:19.930506 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:19.930465 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-94n4h" Sep 16 07:34:19.930963 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:19.930513 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wbslq" Sep 16 07:34:19.930963 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:19.930513 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vx6vh" Sep 16 07:34:19.930963 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:19.930625 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-94n4h" podUID="aca15303-6546-4428-bb32-539bb25d2486" Sep 16 07:34:19.930963 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:19.930692 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wbslq" podUID="4152b2c2-a19c-44fa-96f5-dc3eb661aa9b" Sep 16 07:34:19.930963 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:19.930761 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vx6vh" podUID="3c6593a0-6ec1-4945-b929-f94446ee82b1" Sep 16 07:34:21.930808 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:21.930765 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-94n4h" Sep 16 07:34:21.931333 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:21.930807 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vx6vh" Sep 16 07:34:21.931333 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:21.930814 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wbslq" Sep 16 07:34:21.931333 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:21.930912 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-94n4h" podUID="aca15303-6546-4428-bb32-539bb25d2486" Sep 16 07:34:21.931333 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:21.930993 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vx6vh" podUID="3c6593a0-6ec1-4945-b929-f94446ee82b1" Sep 16 07:34:21.931333 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:21.931055 2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wbslq" podUID="4152b2c2-a19c-44fa-96f5-dc3eb661aa9b" Sep 16 07:34:22.561347 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.561261 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aca15303-6546-4428-bb32-539bb25d2486-metrics-certs\") pod \"network-metrics-daemon-94n4h\" (UID: \"aca15303-6546-4428-bb32-539bb25d2486\") " pod="openshift-multus/network-metrics-daemon-94n4h" Sep 16 07:34:22.561347 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.561325 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4152b2c2-a19c-44fa-96f5-dc3eb661aa9b-original-pull-secret\") pod \"global-pull-secret-syncer-wbslq\" (UID: \"4152b2c2-a19c-44fa-96f5-dc3eb661aa9b\") " pod="kube-system/global-pull-secret-syncer-wbslq" Sep 16 07:34:22.561549 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:22.561413 2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 07:34:22.561549 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:22.561424 2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 16 07:34:22.561549 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:22.561483 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4152b2c2-a19c-44fa-96f5-dc3eb661aa9b-original-pull-secret podName:4152b2c2-a19c-44fa-96f5-dc3eb661aa9b nodeName:}" failed. No retries permitted until 2026-09-16 07:34:54.561469075 +0000 UTC m=+66.145247404 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4152b2c2-a19c-44fa-96f5-dc3eb661aa9b-original-pull-secret") pod "global-pull-secret-syncer-wbslq" (UID: "4152b2c2-a19c-44fa-96f5-dc3eb661aa9b") : object "kube-system"/"original-pull-secret" not registered Sep 16 07:34:22.561549 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:22.561500 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aca15303-6546-4428-bb32-539bb25d2486-metrics-certs podName:aca15303-6546-4428-bb32-539bb25d2486 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:54.561493334 +0000 UTC m=+66.145271650 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/aca15303-6546-4428-bb32-539bb25d2486-metrics-certs") pod "network-metrics-daemon-94n4h" (UID: "aca15303-6546-4428-bb32-539bb25d2486") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 16 07:34:22.652318 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.652288 2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-170.ec2.internal" event="NodeReady" Sep 16 07:34:22.652481 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.652467 2552 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 16 07:34:22.662635 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.662600 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnt8f\" (UniqueName: \"kubernetes.io/projected/3c6593a0-6ec1-4945-b929-f94446ee82b1-kube-api-access-hnt8f\") pod \"network-check-target-vx6vh\" (UID: \"3c6593a0-6ec1-4945-b929-f94446ee82b1\") " pod="openshift-network-diagnostics/network-check-target-vx6vh" Sep 16 07:34:22.662768 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:22.662731 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 16 07:34:22.662768 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:22.662745 2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 16 07:34:22.662768 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:22.662754 2552 projected.go:196] Error preparing data for projected volume kube-api-access-hnt8f for pod openshift-network-diagnostics/network-check-target-vx6vh: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 07:34:22.662865 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:22.662807 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3c6593a0-6ec1-4945-b929-f94446ee82b1-kube-api-access-hnt8f podName:3c6593a0-6ec1-4945-b929-f94446ee82b1 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:54.662794791 +0000 UTC m=+66.246573107 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hnt8f" (UniqueName: "kubernetes.io/projected/3c6593a0-6ec1-4945-b929-f94446ee82b1-kube-api-access-hnt8f") pod "network-check-target-vx6vh" (UID: "3c6593a0-6ec1-4945-b929-f94446ee82b1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 16 07:34:22.713675 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.713647 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2gfrg"] Sep 16 07:34:22.728181 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.728143 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qks9b"] Sep 16 07:34:22.728352 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.728250 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2gfrg" Sep 16 07:34:22.730800 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.730777 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mh2l7\"" Sep 16 07:34:22.730800 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.730772 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 16 07:34:22.731023 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.731003 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 16 07:34:22.731248 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.731225 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 16 07:34:22.740956 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.740926 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-9ff4cbfdc-kvcdr"] Sep 16 07:34:22.741127 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.741043 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qks9b" Sep 16 07:34:22.744545 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.743976 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2msf4\"" Sep 16 07:34:22.744545 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.744315 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 16 07:34:22.744545 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.744437 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 16 07:34:22.759359 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.759320 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2gfrg"] Sep 16 07:34:22.759526 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.759380 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-9ff4cbfdc-kvcdr"] Sep 16 07:34:22.759526 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.759403 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qks9b"] Sep 16 07:34:22.759526 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.759440 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9ff4cbfdc-kvcdr" Sep 16 07:34:22.763281 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.763257 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 16 07:34:22.763448 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.763265 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6bpwz\"" Sep 16 07:34:22.763568 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.763550 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 16 07:34:22.763663 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.763647 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 16 07:34:22.768715 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.768691 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 16 07:34:22.863356 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.863318 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhsjp\" (UniqueName: \"kubernetes.io/projected/4aad7734-8782-4fac-b802-89803b94da62-kube-api-access-hhsjp\") pod \"ingress-canary-2gfrg\" (UID: \"4aad7734-8782-4fac-b802-89803b94da62\") " pod="openshift-ingress-canary/ingress-canary-2gfrg" Sep 16 07:34:22.863356 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.863359 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4aad7734-8782-4fac-b802-89803b94da62-cert\") pod \"ingress-canary-2gfrg\" (UID: \"4aad7734-8782-4fac-b802-89803b94da62\") " pod="openshift-ingress-canary/ingress-canary-2gfrg" Sep 16 07:34:22.863556 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.863382 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/50be9bc4-8115-41e1-afb4-51a305538a5a-image-registry-private-configuration\") pod \"image-registry-9ff4cbfdc-kvcdr\" (UID: \"50be9bc4-8115-41e1-afb4-51a305538a5a\") " pod="openshift-image-registry/image-registry-9ff4cbfdc-kvcdr" Sep 16 07:34:22.863556 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.863447 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qcxn\" (UniqueName: \"kubernetes.io/projected/9852905b-319e-4378-8c40-266974260880-kube-api-access-5qcxn\") pod \"dns-default-qks9b\" (UID: \"9852905b-319e-4378-8c40-266974260880\") " pod="openshift-dns/dns-default-qks9b" Sep 16 07:34:22.863556 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.863499 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9852905b-319e-4378-8c40-266974260880-metrics-tls\") pod \"dns-default-qks9b\" (UID: \"9852905b-319e-4378-8c40-266974260880\") " pod="openshift-dns/dns-default-qks9b" Sep 16 07:34:22.863556 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.863517 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/50be9bc4-8115-41e1-afb4-51a305538a5a-registry-certificates\") pod \"image-registry-9ff4cbfdc-kvcdr\" (UID: \"50be9bc4-8115-41e1-afb4-51a305538a5a\") " pod="openshift-image-registry/image-registry-9ff4cbfdc-kvcdr" Sep 16 07:34:22.863556 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.863534 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/50be9bc4-8115-41e1-afb4-51a305538a5a-bound-sa-token\") pod \"image-registry-9ff4cbfdc-kvcdr\" (UID: \"50be9bc4-8115-41e1-afb4-51a305538a5a\") " pod="openshift-image-registry/image-registry-9ff4cbfdc-kvcdr" Sep 16 07:34:22.863556 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.863552 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdztr\" (UniqueName: \"kubernetes.io/projected/50be9bc4-8115-41e1-afb4-51a305538a5a-kube-api-access-qdztr\") pod \"image-registry-9ff4cbfdc-kvcdr\" (UID: \"50be9bc4-8115-41e1-afb4-51a305538a5a\") " pod="openshift-image-registry/image-registry-9ff4cbfdc-kvcdr" Sep 16 07:34:22.863838 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.863568 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9852905b-319e-4378-8c40-266974260880-tmp-dir\") pod \"dns-default-qks9b\" (UID: \"9852905b-319e-4378-8c40-266974260880\") " pod="openshift-dns/dns-default-qks9b" Sep 16 07:34:22.863838 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.863587 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/50be9bc4-8115-41e1-afb4-51a305538a5a-installation-pull-secrets\") pod \"image-registry-9ff4cbfdc-kvcdr\" (UID: \"50be9bc4-8115-41e1-afb4-51a305538a5a\") " pod="openshift-image-registry/image-registry-9ff4cbfdc-kvcdr" Sep 16 07:34:22.863838 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.863659 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/50be9bc4-8115-41e1-afb4-51a305538a5a-registry-tls\") pod \"image-registry-9ff4cbfdc-kvcdr\" (UID: \"50be9bc4-8115-41e1-afb4-51a305538a5a\") " pod="openshift-image-registry/image-registry-9ff4cbfdc-kvcdr" Sep 16 07:34:22.863838 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.863704 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9852905b-319e-4378-8c40-266974260880-config-volume\") pod \"dns-default-qks9b\" (UID: \"9852905b-319e-4378-8c40-266974260880\") " pod="openshift-dns/dns-default-qks9b" Sep 16 07:34:22.863838 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.863732 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/50be9bc4-8115-41e1-afb4-51a305538a5a-ca-trust-extracted\") pod \"image-registry-9ff4cbfdc-kvcdr\" (UID: \"50be9bc4-8115-41e1-afb4-51a305538a5a\") " pod="openshift-image-registry/image-registry-9ff4cbfdc-kvcdr" Sep 16 07:34:22.863838 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.863758 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/50be9bc4-8115-41e1-afb4-51a305538a5a-trusted-ca\") pod \"image-registry-9ff4cbfdc-kvcdr\" (UID: \"50be9bc4-8115-41e1-afb4-51a305538a5a\") " pod="openshift-image-registry/image-registry-9ff4cbfdc-kvcdr" Sep 16 07:34:22.964875 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.964833 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/50be9bc4-8115-41e1-afb4-51a305538a5a-installation-pull-secrets\") pod \"image-registry-9ff4cbfdc-kvcdr\" (UID: \"50be9bc4-8115-41e1-afb4-51a305538a5a\") " pod="openshift-image-registry/image-registry-9ff4cbfdc-kvcdr" Sep 16 07:34:22.964875 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.964879 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/50be9bc4-8115-41e1-afb4-51a305538a5a-registry-tls\") pod \"image-registry-9ff4cbfdc-kvcdr\" (UID: \"50be9bc4-8115-41e1-afb4-51a305538a5a\") " pod="openshift-image-registry/image-registry-9ff4cbfdc-kvcdr" Sep 16 07:34:22.965431 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.964909 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9852905b-319e-4378-8c40-266974260880-config-volume\") pod \"dns-default-qks9b\" (UID: \"9852905b-319e-4378-8c40-266974260880\") " pod="openshift-dns/dns-default-qks9b" Sep 16 07:34:22.965431 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.964934 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/50be9bc4-8115-41e1-afb4-51a305538a5a-ca-trust-extracted\") pod \"image-registry-9ff4cbfdc-kvcdr\" (UID: \"50be9bc4-8115-41e1-afb4-51a305538a5a\") " pod="openshift-image-registry/image-registry-9ff4cbfdc-kvcdr" Sep 16 07:34:22.965431 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.964956 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/50be9bc4-8115-41e1-afb4-51a305538a5a-trusted-ca\") pod \"image-registry-9ff4cbfdc-kvcdr\" (UID: \"50be9bc4-8115-41e1-afb4-51a305538a5a\") " pod="openshift-image-registry/image-registry-9ff4cbfdc-kvcdr" Sep 16 07:34:22.965431 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.964998 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhsjp\" (UniqueName: \"kubernetes.io/projected/4aad7734-8782-4fac-b802-89803b94da62-kube-api-access-hhsjp\") pod \"ingress-canary-2gfrg\" (UID: \"4aad7734-8782-4fac-b802-89803b94da62\") " pod="openshift-ingress-canary/ingress-canary-2gfrg" Sep 16 07:34:22.965431 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:22.965032 2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 07:34:22.965431 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:22.965058 2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9ff4cbfdc-kvcdr: secret "image-registry-tls" not found Sep 16 07:34:22.965431 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.965143 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4aad7734-8782-4fac-b802-89803b94da62-cert\") pod \"ingress-canary-2gfrg\" (UID: \"4aad7734-8782-4fac-b802-89803b94da62\") " pod="openshift-ingress-canary/ingress-canary-2gfrg" Sep 16 07:34:22.965431 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.965192 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/50be9bc4-8115-41e1-afb4-51a305538a5a-image-registry-private-configuration\") pod \"image-registry-9ff4cbfdc-kvcdr\" (UID: \"50be9bc4-8115-41e1-afb4-51a305538a5a\") " pod="openshift-image-registry/image-registry-9ff4cbfdc-kvcdr" Sep 16 07:34:22.965431 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.965245 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qcxn\" (UniqueName: \"kubernetes.io/projected/9852905b-319e-4378-8c40-266974260880-kube-api-access-5qcxn\") pod \"dns-default-qks9b\" (UID: \"9852905b-319e-4378-8c40-266974260880\") " pod="openshift-dns/dns-default-qks9b" Sep 16 07:34:22.965431 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.965311 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9852905b-319e-4378-8c40-266974260880-metrics-tls\") pod \"dns-default-qks9b\" (UID: \"9852905b-319e-4378-8c40-266974260880\") " pod="openshift-dns/dns-default-qks9b" Sep 16 07:34:22.965431 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.965337 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/50be9bc4-8115-41e1-afb4-51a305538a5a-registry-certificates\") pod \"image-registry-9ff4cbfdc-kvcdr\" (UID: \"50be9bc4-8115-41e1-afb4-51a305538a5a\") " pod="openshift-image-registry/image-registry-9ff4cbfdc-kvcdr" Sep 16 07:34:22.965431 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.965361 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/50be9bc4-8115-41e1-afb4-51a305538a5a-bound-sa-token\") pod \"image-registry-9ff4cbfdc-kvcdr\" (UID: \"50be9bc4-8115-41e1-afb4-51a305538a5a\") " pod="openshift-image-registry/image-registry-9ff4cbfdc-kvcdr" Sep 16 07:34:22.965431 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.965391 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdztr\" (UniqueName: \"kubernetes.io/projected/50be9bc4-8115-41e1-afb4-51a305538a5a-kube-api-access-qdztr\") pod \"image-registry-9ff4cbfdc-kvcdr\" (UID: \"50be9bc4-8115-41e1-afb4-51a305538a5a\") " pod="openshift-image-registry/image-registry-9ff4cbfdc-kvcdr" Sep 16 07:34:22.965431 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.965418 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9852905b-319e-4378-8c40-266974260880-tmp-dir\") pod \"dns-default-qks9b\" (UID: \"9852905b-319e-4378-8c40-266974260880\") " pod="openshift-dns/dns-default-qks9b" Sep 16 07:34:22.966045 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.965497 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/50be9bc4-8115-41e1-afb4-51a305538a5a-ca-trust-extracted\") pod \"image-registry-9ff4cbfdc-kvcdr\" (UID: \"50be9bc4-8115-41e1-afb4-51a305538a5a\") " pod="openshift-image-registry/image-registry-9ff4cbfdc-kvcdr" Sep 16 07:34:22.966045 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:22.965594 2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 07:34:22.966045 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:22.965659 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9852905b-319e-4378-8c40-266974260880-metrics-tls podName:9852905b-319e-4378-8c40-266974260880 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:23.465637294 +0000 UTC m=+35.049415638 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9852905b-319e-4378-8c40-266974260880-metrics-tls") pod "dns-default-qks9b" (UID: "9852905b-319e-4378-8c40-266974260880") : secret "dns-default-metrics-tls" not found Sep 16 07:34:22.966045 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.965675 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9852905b-319e-4378-8c40-266974260880-tmp-dir\") pod \"dns-default-qks9b\" (UID: \"9852905b-319e-4378-8c40-266974260880\") " pod="openshift-dns/dns-default-qks9b" Sep 16 07:34:22.966045 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:22.965686 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50be9bc4-8115-41e1-afb4-51a305538a5a-registry-tls podName:50be9bc4-8115-41e1-afb4-51a305538a5a nodeName:}" failed. No retries permitted until 2026-09-16 07:34:23.465675553 +0000 UTC m=+35.049453875 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/50be9bc4-8115-41e1-afb4-51a305538a5a-registry-tls") pod "image-registry-9ff4cbfdc-kvcdr" (UID: "50be9bc4-8115-41e1-afb4-51a305538a5a") : secret "image-registry-tls" not found Sep 16 07:34:22.966045 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.965723 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9852905b-319e-4378-8c40-266974260880-config-volume\") pod \"dns-default-qks9b\" (UID: \"9852905b-319e-4378-8c40-266974260880\") " pod="openshift-dns/dns-default-qks9b" Sep 16 07:34:22.966045 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.965975 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/50be9bc4-8115-41e1-afb4-51a305538a5a-trusted-ca\") pod \"image-registry-9ff4cbfdc-kvcdr\" (UID: \"50be9bc4-8115-41e1-afb4-51a305538a5a\") " pod="openshift-image-registry/image-registry-9ff4cbfdc-kvcdr" Sep 16 07:34:22.966374 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.966142 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/50be9bc4-8115-41e1-afb4-51a305538a5a-registry-certificates\") pod \"image-registry-9ff4cbfdc-kvcdr\" (UID: \"50be9bc4-8115-41e1-afb4-51a305538a5a\") " pod="openshift-image-registry/image-registry-9ff4cbfdc-kvcdr" Sep 16 07:34:22.969048 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.969018 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/50be9bc4-8115-41e1-afb4-51a305538a5a-image-registry-private-configuration\") pod \"image-registry-9ff4cbfdc-kvcdr\" (UID: \"50be9bc4-8115-41e1-afb4-51a305538a5a\") " pod="openshift-image-registry/image-registry-9ff4cbfdc-kvcdr" Sep 16 07:34:22.969197 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.969158 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/50be9bc4-8115-41e1-afb4-51a305538a5a-installation-pull-secrets\") pod \"image-registry-9ff4cbfdc-kvcdr\" (UID: \"50be9bc4-8115-41e1-afb4-51a305538a5a\") " pod="openshift-image-registry/image-registry-9ff4cbfdc-kvcdr" Sep 16 07:34:22.969197 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.969173 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4aad7734-8782-4fac-b802-89803b94da62-cert\") pod \"ingress-canary-2gfrg\" (UID: \"4aad7734-8782-4fac-b802-89803b94da62\") " pod="openshift-ingress-canary/ingress-canary-2gfrg" Sep 16 07:34:22.976009 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.975985 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/50be9bc4-8115-41e1-afb4-51a305538a5a-bound-sa-token\") pod \"image-registry-9ff4cbfdc-kvcdr\" (UID: \"50be9bc4-8115-41e1-afb4-51a305538a5a\") " pod="openshift-image-registry/image-registry-9ff4cbfdc-kvcdr" Sep 16 07:34:22.976145 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.976010 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhsjp\" (UniqueName: \"kubernetes.io/projected/4aad7734-8782-4fac-b802-89803b94da62-kube-api-access-hhsjp\") pod \"ingress-canary-2gfrg\" (UID: \"4aad7734-8782-4fac-b802-89803b94da62\") " pod="openshift-ingress-canary/ingress-canary-2gfrg" Sep 16 07:34:22.976145 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.976059 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qcxn\" (UniqueName: \"kubernetes.io/projected/9852905b-319e-4378-8c40-266974260880-kube-api-access-5qcxn\") pod \"dns-default-qks9b\" (UID: \"9852905b-319e-4378-8c40-266974260880\") " pod="openshift-dns/dns-default-qks9b" Sep 16 07:34:22.976223 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:22.976142 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdztr\" (UniqueName: \"kubernetes.io/projected/50be9bc4-8115-41e1-afb4-51a305538a5a-kube-api-access-qdztr\") pod \"image-registry-9ff4cbfdc-kvcdr\" (UID: \"50be9bc4-8115-41e1-afb4-51a305538a5a\") " pod="openshift-image-registry/image-registry-9ff4cbfdc-kvcdr" Sep 16 07:34:23.043231 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:23.043194 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2gfrg" Sep 16 07:34:23.075634 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:23.075405 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:23.105061 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:23.105026 2552 generic.go:356] "Generic (PLEG): container finished" podID="2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7" containerID="e4a9537c6a55625f1a03bfbc9a7b57cd917cf555dae3c9d3d04a09a9c462ef5c" exitCode=0 Sep 16 07:34:23.105250 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:23.105084 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mdzsg" event={"ID":"2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7","Type":"ContainerDied","Data":"e4a9537c6a55625f1a03bfbc9a7b57cd917cf555dae3c9d3d04a09a9c462ef5c"} Sep 16 07:34:23.190576 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:23.190537 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:23.348191 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:23.348093 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2gfrg"] Sep 16 07:34:23.351275 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:34:23.351248 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aad7734_8782_4fac_b802_89803b94da62.slice/crio-c9bed3d04086984ff12b69b72b84563415e8c04f60f32504e72137bd3ec2134c WatchSource:0}: Error finding container c9bed3d04086984ff12b69b72b84563415e8c04f60f32504e72137bd3ec2134c: Status 404 returned error can't find the container with id c9bed3d04086984ff12b69b72b84563415e8c04f60f32504e72137bd3ec2134c Sep 16 07:34:23.468060 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:23.468023 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9852905b-319e-4378-8c40-266974260880-metrics-tls\") pod \"dns-default-qks9b\" (UID: \"9852905b-319e-4378-8c40-266974260880\") " pod="openshift-dns/dns-default-qks9b" Sep 16 07:34:23.468220 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:23.468073 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/50be9bc4-8115-41e1-afb4-51a305538a5a-registry-tls\") pod \"image-registry-9ff4cbfdc-kvcdr\" (UID: \"50be9bc4-8115-41e1-afb4-51a305538a5a\") " pod="openshift-image-registry/image-registry-9ff4cbfdc-kvcdr" Sep 16 07:34:23.468220 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:23.468205 2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 07:34:23.468317 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:23.468276 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9852905b-319e-4378-8c40-266974260880-metrics-tls podName:9852905b-319e-4378-8c40-266974260880 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:24.468261308 +0000 UTC m=+36.052039628 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9852905b-319e-4378-8c40-266974260880-metrics-tls") pod "dns-default-qks9b" (UID: "9852905b-319e-4378-8c40-266974260880") : secret "dns-default-metrics-tls" not found Sep 16 07:34:23.468317 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:23.468211 2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 07:34:23.468400 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:23.468317 2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9ff4cbfdc-kvcdr: secret "image-registry-tls" not found Sep 16 07:34:23.468400 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:23.468359 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50be9bc4-8115-41e1-afb4-51a305538a5a-registry-tls podName:50be9bc4-8115-41e1-afb4-51a305538a5a nodeName:}" failed. No retries permitted until 2026-09-16 07:34:24.468345934 +0000 UTC m=+36.052124265 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/50be9bc4-8115-41e1-afb4-51a305538a5a-registry-tls") pod "image-registry-9ff4cbfdc-kvcdr" (UID: "50be9bc4-8115-41e1-afb4-51a305538a5a") : secret "image-registry-tls" not found Sep 16 07:34:23.930723 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:23.930690 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vx6vh" Sep 16 07:34:23.930924 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:23.930691 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-94n4h" Sep 16 07:34:23.931135 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:23.931040 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wbslq" Sep 16 07:34:23.933020 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:23.933001 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 16 07:34:23.933020 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:23.933016 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lz8hv\"" Sep 16 07:34:23.933906 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:23.933874 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 16 07:34:23.934018 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:23.933874 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 16 07:34:23.934018 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:23.933944 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zf7hx\"" Sep 16 07:34:23.934018 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:23.933995 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 16 07:34:24.109586 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:24.109547 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2gfrg" event={"ID":"4aad7734-8782-4fac-b802-89803b94da62","Type":"ContainerStarted","Data":"c9bed3d04086984ff12b69b72b84563415e8c04f60f32504e72137bd3ec2134c"} Sep 16 07:34:24.113392 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:24.113361 2552 generic.go:356] "Generic (PLEG): container finished" podID="2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7" containerID="68708e45c9c4bc65408880403c409b7b80d0dbb9180b04026ee351553a5f8fda" exitCode=0 Sep 16 07:34:24.113530 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:24.113418 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mdzsg" event={"ID":"2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7","Type":"ContainerDied","Data":"68708e45c9c4bc65408880403c409b7b80d0dbb9180b04026ee351553a5f8fda"} Sep 16 07:34:24.478119 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:24.478012 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9852905b-319e-4378-8c40-266974260880-metrics-tls\") pod \"dns-default-qks9b\" (UID: \"9852905b-319e-4378-8c40-266974260880\") " pod="openshift-dns/dns-default-qks9b" Sep 16 07:34:24.478119 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:24.478063 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/50be9bc4-8115-41e1-afb4-51a305538a5a-registry-tls\") pod \"image-registry-9ff4cbfdc-kvcdr\" (UID: \"50be9bc4-8115-41e1-afb4-51a305538a5a\") " pod="openshift-image-registry/image-registry-9ff4cbfdc-kvcdr" Sep 16 07:34:24.478360 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:24.478191 2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 07:34:24.478360 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:24.478205 2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 07:34:24.478360 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:24.478223 2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9ff4cbfdc-kvcdr: secret "image-registry-tls" not found Sep 16 07:34:24.478360 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:24.478272 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9852905b-319e-4378-8c40-266974260880-metrics-tls podName:9852905b-319e-4378-8c40-266974260880 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:26.478250491 +0000 UTC m=+38.062028838 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9852905b-319e-4378-8c40-266974260880-metrics-tls") pod "dns-default-qks9b" (UID: "9852905b-319e-4378-8c40-266974260880") : secret "dns-default-metrics-tls" not found Sep 16 07:34:24.478360 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:24.478295 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50be9bc4-8115-41e1-afb4-51a305538a5a-registry-tls podName:50be9bc4-8115-41e1-afb4-51a305538a5a nodeName:}" failed. No retries permitted until 2026-09-16 07:34:26.478284277 +0000 UTC m=+38.062062598 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/50be9bc4-8115-41e1-afb4-51a305538a5a-registry-tls") pod "image-registry-9ff4cbfdc-kvcdr" (UID: "50be9bc4-8115-41e1-afb4-51a305538a5a") : secret "image-registry-tls" not found Sep 16 07:34:25.119791 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:25.119753 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mdzsg" event={"ID":"2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7","Type":"ContainerStarted","Data":"6c5e3305b4f26b75475b7df0a1eed5774398eae37450b4df5be695f102fa8054"} Sep 16 07:34:25.146760 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:25.146589 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mdzsg" podStartSLOduration=6.725316021 podStartE2EDuration="36.146565811s" totalImagesPullingTime="29.42124979s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 07:33:49 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:25.145476737 +0000 UTC m=+36.729255076" watchObservedRunningTime="2026-09-16 07:34:25.146565811 +0000 UTC m=+36.730344148" Sep 16 07:34:25.273509 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:25.273477 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:25.338066 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:25.338029 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:25.922838 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:25.922558 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:25.990983 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:25.990948 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:26.050803 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:26.050763 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:26.124023 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:26.123887 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2gfrg" event={"ID":"4aad7734-8782-4fac-b802-89803b94da62","Type":"ContainerStarted","Data":"8f432e60c4f83d90d63c5b295105cdf7d5077776fa8bc901d192120ec8607de5"} Sep 16 07:34:26.141490 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:26.141435 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2gfrg" podStartSLOduration=1.6190473829999998 podStartE2EDuration="4.141417697s" totalImagesPullingTime="2.522370314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:34:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:26.140139987 +0000 UTC m=+37.723918327" watchObservedRunningTime="2026-09-16 07:34:26.141417697 +0000 UTC m=+37.725196032" Sep 16 07:34:26.423615 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:26.423580 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:26.486448 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:26.486407 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:26.497474 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:26.497439 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9852905b-319e-4378-8c40-266974260880-metrics-tls\") pod \"dns-default-qks9b\" (UID: \"9852905b-319e-4378-8c40-266974260880\") " pod="openshift-dns/dns-default-qks9b" Sep 16 07:34:26.497608 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:26.497488 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/50be9bc4-8115-41e1-afb4-51a305538a5a-registry-tls\") pod \"image-registry-9ff4cbfdc-kvcdr\" (UID: \"50be9bc4-8115-41e1-afb4-51a305538a5a\") " pod="openshift-image-registry/image-registry-9ff4cbfdc-kvcdr" Sep 16 07:34:26.497608 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:26.497600 2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 07:34:26.497687 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:26.497632 2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 07:34:26.497687 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:26.497644 2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9ff4cbfdc-kvcdr: secret "image-registry-tls" not found Sep 16 07:34:26.497687 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:26.497669 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9852905b-319e-4378-8c40-266974260880-metrics-tls podName:9852905b-319e-4378-8c40-266974260880 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:30.497650063 +0000 UTC m=+42.081428404 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9852905b-319e-4378-8c40-266974260880-metrics-tls") pod "dns-default-qks9b" (UID: "9852905b-319e-4378-8c40-266974260880") : secret "dns-default-metrics-tls" not found Sep 16 07:34:26.497687 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:26.497685 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50be9bc4-8115-41e1-afb4-51a305538a5a-registry-tls podName:50be9bc4-8115-41e1-afb4-51a305538a5a nodeName:}" failed. No retries permitted until 2026-09-16 07:34:30.497676857 +0000 UTC m=+42.081455178 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/50be9bc4-8115-41e1-afb4-51a305538a5a-registry-tls") pod "image-registry-9ff4cbfdc-kvcdr" (UID: "50be9bc4-8115-41e1-afb4-51a305538a5a") : secret "image-registry-tls" not found Sep 16 07:34:27.527881 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:27.527850 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:27.628459 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:27.628368 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:30.528213 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:30.528166 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9852905b-319e-4378-8c40-266974260880-metrics-tls\") pod \"dns-default-qks9b\" (UID: \"9852905b-319e-4378-8c40-266974260880\") " pod="openshift-dns/dns-default-qks9b" Sep 16 07:34:30.528660 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:30.528223 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/50be9bc4-8115-41e1-afb4-51a305538a5a-registry-tls\") pod \"image-registry-9ff4cbfdc-kvcdr\" (UID: \"50be9bc4-8115-41e1-afb4-51a305538a5a\") " pod="openshift-image-registry/image-registry-9ff4cbfdc-kvcdr" Sep 16 07:34:30.528660 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:30.528339 2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 07:34:30.528660 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:30.528353 2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 07:34:30.528660 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:30.528372 2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9ff4cbfdc-kvcdr: secret "image-registry-tls" not found Sep 16 07:34:30.528660 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:30.528409 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9852905b-319e-4378-8c40-266974260880-metrics-tls podName:9852905b-319e-4378-8c40-266974260880 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:38.528392069 +0000 UTC m=+50.112170414 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9852905b-319e-4378-8c40-266974260880-metrics-tls") pod "dns-default-qks9b" (UID: "9852905b-319e-4378-8c40-266974260880") : secret "dns-default-metrics-tls" not found Sep 16 07:34:30.528660 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:30.528424 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50be9bc4-8115-41e1-afb4-51a305538a5a-registry-tls podName:50be9bc4-8115-41e1-afb4-51a305538a5a nodeName:}" failed. No retries permitted until 2026-09-16 07:34:38.528417859 +0000 UTC m=+50.112196175 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/50be9bc4-8115-41e1-afb4-51a305538a5a-registry-tls") pod "image-registry-9ff4cbfdc-kvcdr" (UID: "50be9bc4-8115-41e1-afb4-51a305538a5a") : secret "image-registry-tls" not found Sep 16 07:34:32.253398 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:32.253358 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:32.258783 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:32.258750 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rw4xp"] Sep 16 07:34:32.283763 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:32.283716 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rw4xp"] Sep 16 07:34:32.283909 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:32.283855 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rw4xp" Sep 16 07:34:32.286555 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:32.286533 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 16 07:34:32.286555 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:32.286547 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 16 07:34:32.286770 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:32.286532 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 16 07:34:32.287082 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:32.287066 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 16 07:34:32.287132 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:32.287069 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-57x9m\"" Sep 16 07:34:32.321064 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:32.321016 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:32.442336 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:32.442296 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3cc58375-e99c-4fd2-8d0c-1a652869ac2d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rw4xp\" (UID: \"3cc58375-e99c-4fd2-8d0c-1a652869ac2d\") " pod="openshift-insights/insights-runtime-extractor-rw4xp" Sep 16 07:34:32.442477 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:32.442357 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3cc58375-e99c-4fd2-8d0c-1a652869ac2d-data-volume\") pod \"insights-runtime-extractor-rw4xp\" (UID: \"3cc58375-e99c-4fd2-8d0c-1a652869ac2d\") " pod="openshift-insights/insights-runtime-extractor-rw4xp" Sep 16 07:34:32.442477 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:32.442428 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3cc58375-e99c-4fd2-8d0c-1a652869ac2d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rw4xp\" (UID: \"3cc58375-e99c-4fd2-8d0c-1a652869ac2d\") " pod="openshift-insights/insights-runtime-extractor-rw4xp" Sep 16 07:34:32.442544 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:32.442479 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3cc58375-e99c-4fd2-8d0c-1a652869ac2d-crio-socket\") pod \"insights-runtime-extractor-rw4xp\" (UID: \"3cc58375-e99c-4fd2-8d0c-1a652869ac2d\") " pod="openshift-insights/insights-runtime-extractor-rw4xp" Sep 16 07:34:32.442544 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:32.442508 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh98p\" (UniqueName: \"kubernetes.io/projected/3cc58375-e99c-4fd2-8d0c-1a652869ac2d-kube-api-access-mh98p\") pod \"insights-runtime-extractor-rw4xp\" (UID: \"3cc58375-e99c-4fd2-8d0c-1a652869ac2d\") " pod="openshift-insights/insights-runtime-extractor-rw4xp" Sep 16 07:34:32.543253 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:32.543182 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3cc58375-e99c-4fd2-8d0c-1a652869ac2d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rw4xp\" (UID: \"3cc58375-e99c-4fd2-8d0c-1a652869ac2d\") " pod="openshift-insights/insights-runtime-extractor-rw4xp" Sep 16 07:34:32.543253 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:32.543237 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3cc58375-e99c-4fd2-8d0c-1a652869ac2d-crio-socket\") pod \"insights-runtime-extractor-rw4xp\" (UID: \"3cc58375-e99c-4fd2-8d0c-1a652869ac2d\") " pod="openshift-insights/insights-runtime-extractor-rw4xp" Sep 16 07:34:32.543399 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:32.543267 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh98p\" (UniqueName: \"kubernetes.io/projected/3cc58375-e99c-4fd2-8d0c-1a652869ac2d-kube-api-access-mh98p\") pod \"insights-runtime-extractor-rw4xp\" (UID: \"3cc58375-e99c-4fd2-8d0c-1a652869ac2d\") " pod="openshift-insights/insights-runtime-extractor-rw4xp" Sep 16 07:34:32.543399 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:32.543302 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3cc58375-e99c-4fd2-8d0c-1a652869ac2d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rw4xp\" (UID: \"3cc58375-e99c-4fd2-8d0c-1a652869ac2d\") " pod="openshift-insights/insights-runtime-extractor-rw4xp" Sep 16 07:34:32.543399 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:32.543324 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3cc58375-e99c-4fd2-8d0c-1a652869ac2d-data-volume\") pod \"insights-runtime-extractor-rw4xp\" (UID: \"3cc58375-e99c-4fd2-8d0c-1a652869ac2d\") " pod="openshift-insights/insights-runtime-extractor-rw4xp" Sep 16 07:34:32.543399 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:32.543338 2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 07:34:32.543537 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:32.543414 2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 07:34:32.543537 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:32.543430 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cc58375-e99c-4fd2-8d0c-1a652869ac2d-insights-runtime-extractor-tls podName:3cc58375-e99c-4fd2-8d0c-1a652869ac2d nodeName:}" failed. No retries permitted until 2026-09-16 07:34:33.043408508 +0000 UTC m=+44.627186826 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3cc58375-e99c-4fd2-8d0c-1a652869ac2d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rw4xp" (UID: "3cc58375-e99c-4fd2-8d0c-1a652869ac2d") : secret "insights-runtime-extractor-tls" not found Sep 16 07:34:32.543537 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:32.543492 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3cc58375-e99c-4fd2-8d0c-1a652869ac2d-kube-rbac-proxy-cm podName:3cc58375-e99c-4fd2-8d0c-1a652869ac2d nodeName:}" failed. No retries permitted until 2026-09-16 07:34:33.043474437 +0000 UTC m=+44.627252755 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3cc58375-e99c-4fd2-8d0c-1a652869ac2d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rw4xp" (UID: "3cc58375-e99c-4fd2-8d0c-1a652869ac2d") : configmap "kube-rbac-proxy" not found Sep 16 07:34:32.543537 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:32.543511 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3cc58375-e99c-4fd2-8d0c-1a652869ac2d-crio-socket\") pod \"insights-runtime-extractor-rw4xp\" (UID: \"3cc58375-e99c-4fd2-8d0c-1a652869ac2d\") " pod="openshift-insights/insights-runtime-extractor-rw4xp" Sep 16 07:34:32.543670 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:32.543588 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3cc58375-e99c-4fd2-8d0c-1a652869ac2d-data-volume\") pod \"insights-runtime-extractor-rw4xp\" (UID: \"3cc58375-e99c-4fd2-8d0c-1a652869ac2d\") " pod="openshift-insights/insights-runtime-extractor-rw4xp" Sep 16 07:34:32.554345 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:32.554320 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh98p\" (UniqueName: \"kubernetes.io/projected/3cc58375-e99c-4fd2-8d0c-1a652869ac2d-kube-api-access-mh98p\") pod \"insights-runtime-extractor-rw4xp\" (UID: \"3cc58375-e99c-4fd2-8d0c-1a652869ac2d\") " pod="openshift-insights/insights-runtime-extractor-rw4xp" Sep 16 07:34:33.047159 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:33.047091 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3cc58375-e99c-4fd2-8d0c-1a652869ac2d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rw4xp\" (UID: \"3cc58375-e99c-4fd2-8d0c-1a652869ac2d\") " pod="openshift-insights/insights-runtime-extractor-rw4xp" Sep 16 07:34:33.047373 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:33.047217 2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 07:34:33.047373 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:33.047229 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3cc58375-e99c-4fd2-8d0c-1a652869ac2d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rw4xp\" (UID: \"3cc58375-e99c-4fd2-8d0c-1a652869ac2d\") " pod="openshift-insights/insights-runtime-extractor-rw4xp" Sep 16 07:34:33.047373 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:33.047306 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cc58375-e99c-4fd2-8d0c-1a652869ac2d-insights-runtime-extractor-tls podName:3cc58375-e99c-4fd2-8d0c-1a652869ac2d nodeName:}" failed. No retries permitted until 2026-09-16 07:34:34.047263233 +0000 UTC m=+45.631041562 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3cc58375-e99c-4fd2-8d0c-1a652869ac2d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rw4xp" (UID: "3cc58375-e99c-4fd2-8d0c-1a652869ac2d") : secret "insights-runtime-extractor-tls" not found Sep 16 07:34:33.047373 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:33.047314 2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 07:34:33.047373 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:33.047376 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3cc58375-e99c-4fd2-8d0c-1a652869ac2d-kube-rbac-proxy-cm podName:3cc58375-e99c-4fd2-8d0c-1a652869ac2d nodeName:}" failed. No retries permitted until 2026-09-16 07:34:34.047359337 +0000 UTC m=+45.631137669 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3cc58375-e99c-4fd2-8d0c-1a652869ac2d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rw4xp" (UID: "3cc58375-e99c-4fd2-8d0c-1a652869ac2d") : configmap "kube-rbac-proxy" not found Sep 16 07:34:33.058391 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:33.058361 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-7r4rg"] Sep 16 07:34:33.061732 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:33.061701 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:33.074311 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:33.074238 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-7r4rg" Sep 16 07:34:33.075440 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:33.075416 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-7r4rg"] Sep 16 07:34:33.077038 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:33.077020 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Sep 16 07:34:33.077858 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:33.077832 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-6pjlx\"" Sep 16 07:34:33.077963 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:33.077836 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Sep 16 07:34:33.150469 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:33.150436 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:33.248812 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:33.248780 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jdsh\" (UniqueName: \"kubernetes.io/projected/fd239b36-8f44-43b4-b6d5-cd450071846a-kube-api-access-7jdsh\") pod \"migrator-8b4d697c5-7r4rg\" (UID: \"fd239b36-8f44-43b4-b6d5-cd450071846a\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-7r4rg" Sep 16 07:34:33.254848 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:33.254806 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:33.350226 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:33.350138 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jdsh\" (UniqueName: \"kubernetes.io/projected/fd239b36-8f44-43b4-b6d5-cd450071846a-kube-api-access-7jdsh\") pod \"migrator-8b4d697c5-7r4rg\" (UID: \"fd239b36-8f44-43b4-b6d5-cd450071846a\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-7r4rg" Sep 16 07:34:33.370703 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:33.370670 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jdsh\" (UniqueName: \"kubernetes.io/projected/fd239b36-8f44-43b4-b6d5-cd450071846a-kube-api-access-7jdsh\") pod \"migrator-8b4d697c5-7r4rg\" (UID: \"fd239b36-8f44-43b4-b6d5-cd450071846a\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-7r4rg" Sep 16 07:34:33.383657 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:33.383629 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-7r4rg" Sep 16 07:34:33.491876 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:33.491842 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:33.558831 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:33.558796 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:33.675721 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:33.675692 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-7r4rg"] Sep 16 07:34:33.678591 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:34:33.678561 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd239b36_8f44_43b4_b6d5_cd450071846a.slice/crio-8f447ae3658bfa1dc2f0319e5ca473a9164ddcfa25c5b9d504b58d8d2276e319 WatchSource:0}: Error finding container 8f447ae3658bfa1dc2f0319e5ca473a9164ddcfa25c5b9d504b58d8d2276e319: Status 404 returned error can't find the container with id 8f447ae3658bfa1dc2f0319e5ca473a9164ddcfa25c5b9d504b58d8d2276e319 Sep 16 07:34:33.937742 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:33.937655 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:34.008181 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:34.008151 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rxbfc_024d7ae0-3a5a-4d5e-bedb-592058d44673/dns-node-resolver/0.log" Sep 16 07:34:34.056963 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:34.056934 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3cc58375-e99c-4fd2-8d0c-1a652869ac2d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rw4xp\" (UID: \"3cc58375-e99c-4fd2-8d0c-1a652869ac2d\") " pod="openshift-insights/insights-runtime-extractor-rw4xp" Sep 16 07:34:34.057138 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:34.057118 2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 07:34:34.057183 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:34.057133 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3cc58375-e99c-4fd2-8d0c-1a652869ac2d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rw4xp\" (UID: \"3cc58375-e99c-4fd2-8d0c-1a652869ac2d\") " pod="openshift-insights/insights-runtime-extractor-rw4xp" Sep 16 07:34:34.057270 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:34.057182 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cc58375-e99c-4fd2-8d0c-1a652869ac2d-insights-runtime-extractor-tls podName:3cc58375-e99c-4fd2-8d0c-1a652869ac2d nodeName:}" failed. No retries permitted until 2026-09-16 07:34:36.057165313 +0000 UTC m=+47.640943629 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3cc58375-e99c-4fd2-8d0c-1a652869ac2d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rw4xp" (UID: "3cc58375-e99c-4fd2-8d0c-1a652869ac2d") : secret "insights-runtime-extractor-tls" not found Sep 16 07:34:34.057270 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:34.057219 2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 07:34:34.057335 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:34.057286 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3cc58375-e99c-4fd2-8d0c-1a652869ac2d-kube-rbac-proxy-cm podName:3cc58375-e99c-4fd2-8d0c-1a652869ac2d nodeName:}" failed. No retries permitted until 2026-09-16 07:34:36.057270662 +0000 UTC m=+47.641048993 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3cc58375-e99c-4fd2-8d0c-1a652869ac2d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rw4xp" (UID: "3cc58375-e99c-4fd2-8d0c-1a652869ac2d") : configmap "kube-rbac-proxy" not found Sep 16 07:34:34.143918 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:34.143882 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-7r4rg" event={"ID":"fd239b36-8f44-43b4-b6d5-cd450071846a","Type":"ContainerStarted","Data":"8f447ae3658bfa1dc2f0319e5ca473a9164ddcfa25c5b9d504b58d8d2276e319"} Sep 16 07:34:34.196897 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:34.196799 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:34.296820 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:34.296774 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:34.716108 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:34.716069 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:34.788194 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:34.788152 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:34.804913 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:34.804883 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9xkmz_29d2a484-46de-40a9-be4a-b9c6b266fbae/node-ca/0.log" Sep 16 07:34:35.404797 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:35.404766 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2gfrg_4aad7734-8782-4fac-b802-89803b94da62/serve-healthcheck-canary/0.log" Sep 16 07:34:35.841275 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:35.841238 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:35.941742 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:35.941703 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:36.076023 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:36.075923 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3cc58375-e99c-4fd2-8d0c-1a652869ac2d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rw4xp\" (UID: \"3cc58375-e99c-4fd2-8d0c-1a652869ac2d\") " pod="openshift-insights/insights-runtime-extractor-rw4xp" Sep 16 07:34:36.076192 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:36.076035 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3cc58375-e99c-4fd2-8d0c-1a652869ac2d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rw4xp\" (UID: \"3cc58375-e99c-4fd2-8d0c-1a652869ac2d\") " pod="openshift-insights/insights-runtime-extractor-rw4xp" Sep 16 07:34:36.076192 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:36.076078 2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 07:34:36.076192 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:36.076130 2552 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 16 07:34:36.076192 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:36.076170 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cc58375-e99c-4fd2-8d0c-1a652869ac2d-insights-runtime-extractor-tls podName:3cc58375-e99c-4fd2-8d0c-1a652869ac2d nodeName:}" failed. No retries permitted until 2026-09-16 07:34:40.076153753 +0000 UTC m=+51.659932070 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3cc58375-e99c-4fd2-8d0c-1a652869ac2d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rw4xp" (UID: "3cc58375-e99c-4fd2-8d0c-1a652869ac2d") : secret "insights-runtime-extractor-tls" not found Sep 16 07:34:36.076192 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:36.076186 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3cc58375-e99c-4fd2-8d0c-1a652869ac2d-kube-rbac-proxy-cm podName:3cc58375-e99c-4fd2-8d0c-1a652869ac2d nodeName:}" failed. No retries permitted until 2026-09-16 07:34:40.076179698 +0000 UTC m=+51.659958013 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3cc58375-e99c-4fd2-8d0c-1a652869ac2d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rw4xp" (UID: "3cc58375-e99c-4fd2-8d0c-1a652869ac2d") : configmap "kube-rbac-proxy" not found Sep 16 07:34:36.150310 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:36.150269 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-7r4rg" event={"ID":"fd239b36-8f44-43b4-b6d5-cd450071846a","Type":"ContainerStarted","Data":"8853f1285e55cab56aa815d4549baa1cc7569aea051ea8b491735c8ec6c3c854"} Sep 16 07:34:36.150471 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:36.150357 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-7r4rg" event={"ID":"fd239b36-8f44-43b4-b6d5-cd450071846a","Type":"ContainerStarted","Data":"02792880bca1b2fa65b14fc783db8d1adb6f7151b39c9214a8e7f1d508ba5fde"} Sep 16 07:34:36.165469 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:36.165416 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-7r4rg" podStartSLOduration=1.117246756 podStartE2EDuration="3.165400225s" totalImagesPullingTime="2.048153469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:34:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:36.164195381 +0000 UTC m=+47.747973718" watchObservedRunningTime="2026-09-16 07:34:36.165400225 +0000 UTC m=+47.749178563" Sep 16 07:34:36.240399 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:36.240363 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:36.300441 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:36.300410 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:36.694906 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:36.694870 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:37.450807 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:37.450776 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:38.596810 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:38.596775 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9852905b-319e-4378-8c40-266974260880-metrics-tls\") pod \"dns-default-qks9b\" (UID: \"9852905b-319e-4378-8c40-266974260880\") " pod="openshift-dns/dns-default-qks9b" Sep 16 07:34:38.596810 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:38.596818 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/50be9bc4-8115-41e1-afb4-51a305538a5a-registry-tls\") pod \"image-registry-9ff4cbfdc-kvcdr\" (UID: \"50be9bc4-8115-41e1-afb4-51a305538a5a\") " pod="openshift-image-registry/image-registry-9ff4cbfdc-kvcdr" Sep 16 07:34:38.597267 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:38.596933 2552 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 16 07:34:38.597267 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:38.596954 2552 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-9ff4cbfdc-kvcdr: secret "image-registry-tls" not found Sep 16 07:34:38.597267 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:38.597003 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/50be9bc4-8115-41e1-afb4-51a305538a5a-registry-tls podName:50be9bc4-8115-41e1-afb4-51a305538a5a nodeName:}" failed. No retries permitted until 2026-09-16 07:34:54.596990212 +0000 UTC m=+66.180768529 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/50be9bc4-8115-41e1-afb4-51a305538a5a-registry-tls") pod "image-registry-9ff4cbfdc-kvcdr" (UID: "50be9bc4-8115-41e1-afb4-51a305538a5a") : secret "image-registry-tls" not found Sep 16 07:34:38.597267 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:38.596933 2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 16 07:34:38.597267 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:38.597064 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9852905b-319e-4378-8c40-266974260880-metrics-tls podName:9852905b-319e-4378-8c40-266974260880 nodeName:}" failed. No retries permitted until 2026-09-16 07:34:54.597051946 +0000 UTC m=+66.180830277 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/9852905b-319e-4378-8c40-266974260880-metrics-tls") pod "dns-default-qks9b" (UID: "9852905b-319e-4378-8c40-266974260880") : secret "dns-default-metrics-tls" not found Sep 16 07:34:38.703260 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:38.703226 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:38.763764 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:38.763729 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:39.440846 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:39.440810 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:40.107693 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:40.107651 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3cc58375-e99c-4fd2-8d0c-1a652869ac2d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rw4xp\" (UID: \"3cc58375-e99c-4fd2-8d0c-1a652869ac2d\") " pod="openshift-insights/insights-runtime-extractor-rw4xp" Sep 16 07:34:40.108206 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:40.107721 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3cc58375-e99c-4fd2-8d0c-1a652869ac2d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rw4xp\" (UID: \"3cc58375-e99c-4fd2-8d0c-1a652869ac2d\") " pod="openshift-insights/insights-runtime-extractor-rw4xp" Sep 16 07:34:40.108206 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:40.107851 2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 16 07:34:40.108206 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:34:40.107910 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cc58375-e99c-4fd2-8d0c-1a652869ac2d-insights-runtime-extractor-tls podName:3cc58375-e99c-4fd2-8d0c-1a652869ac2d nodeName:}" failed. No retries permitted until 2026-09-16 07:34:48.107894183 +0000 UTC m=+59.691672520 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3cc58375-e99c-4fd2-8d0c-1a652869ac2d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rw4xp" (UID: "3cc58375-e99c-4fd2-8d0c-1a652869ac2d") : secret "insights-runtime-extractor-tls" not found Sep 16 07:34:40.108480 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:40.108455 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3cc58375-e99c-4fd2-8d0c-1a652869ac2d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rw4xp\" (UID: \"3cc58375-e99c-4fd2-8d0c-1a652869ac2d\") " pod="openshift-insights/insights-runtime-extractor-rw4xp" Sep 16 07:34:41.805942 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:41.805911 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:42.056538 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:42.056448 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:42.169603 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:42.169565 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:42.286780 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:42.286750 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:42.394444 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:42.394352 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:42.513345 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:42.513313 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:42.940823 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:42.940772 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:43.509252 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:43.509214 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:43.706617 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:43.706581 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:43.906360 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:43.906272 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:44.303344 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:44.303305 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:44.918332 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:44.918291 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:45.509024 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:45.508989 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:45.574640 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:45.574589 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:45.706498 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:45.706409 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:46.111115 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:46.111073 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:46.323408 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:46.323373 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:46.394163 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:46.394041 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:46.903322 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:46.903286 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:47.296071 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:47.296037 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:47.701970 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:47.701937 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:47.766455 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:47.766398 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:47.823052 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:47.823016 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:48.101734 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:48.101637 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:48.168089 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:48.168053 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3cc58375-e99c-4fd2-8d0c-1a652869ac2d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rw4xp\" (UID: \"3cc58375-e99c-4fd2-8d0c-1a652869ac2d\") " pod="openshift-insights/insights-runtime-extractor-rw4xp" Sep 16 07:34:48.170498 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:48.170462 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3cc58375-e99c-4fd2-8d0c-1a652869ac2d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rw4xp\" (UID: \"3cc58375-e99c-4fd2-8d0c-1a652869ac2d\") " pod="openshift-insights/insights-runtime-extractor-rw4xp" Sep 16 07:34:48.181374 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:48.181344 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:48.193241 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:48.193205 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rw4xp" Sep 16 07:34:48.310289 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:48.310248 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:48.468799 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:48.468762 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rw4xp"] Sep 16 07:34:48.472578 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:34:48.472546 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cc58375_e99c_4fd2_8d0c_1a652869ac2d.slice/crio-278d9f17c40737eece42574ece200d31f6d560c50ed496c93a283e4919262e13 WatchSource:0}: Error finding container 278d9f17c40737eece42574ece200d31f6d560c50ed496c93a283e4919262e13: Status 404 returned error can't find the container with id 278d9f17c40737eece42574ece200d31f6d560c50ed496c93a283e4919262e13 Sep 16 07:34:48.900805 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:48.900725 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:49.101926 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:49.101897 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-xc9v8" Sep 16 07:34:49.184251 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:49.184219 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rw4xp" event={"ID":"3cc58375-e99c-4fd2-8d0c-1a652869ac2d","Type":"ContainerStarted","Data":"45cdc9f6470398ed8c75a693e5a601f7b0851de5332143c856cf122a33390a59"} Sep 16 07:34:49.184390 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:49.184271 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rw4xp" event={"ID":"3cc58375-e99c-4fd2-8d0c-1a652869ac2d","Type":"ContainerStarted","Data":"278d9f17c40737eece42574ece200d31f6d560c50ed496c93a283e4919262e13"} Sep 16 07:34:49.367048 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:49.367011 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:49.467972 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:49.467934 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:49.568851 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:49.568811 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:49.702730 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:49.702695 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:49.756163 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:49.756058 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:49.810438 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:49.810404 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:49.912494 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:49.912449 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:50.115296 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:50.115216 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:50.188727 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:50.188693 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rw4xp" event={"ID":"3cc58375-e99c-4fd2-8d0c-1a652869ac2d","Type":"ContainerStarted","Data":"c63dacfb709fbbade1070fdd32126c9d6df6e65bcae851bb6e2ea9b7e9caac3a"} Sep 16 07:34:50.198652 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:50.198610 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:50.307666 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:50.307635 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:50.367652 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:50.367539 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:50.701406 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:50.701364 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:51.098171 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:51.098133 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:51.193267 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:51.193227 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rw4xp" event={"ID":"3cc58375-e99c-4fd2-8d0c-1a652869ac2d","Type":"ContainerStarted","Data":"a5bfb4a9c0807fc21e2e2612bb22be7eb4ec0011f9703bd86e8f2cbda37fd1bf"} Sep 16 07:34:51.211185 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:51.211132 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rw4xp" podStartSLOduration=17.231905209 podStartE2EDuration="19.211114933s" totalImagesPullingTime="1.979209724s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 07:34:32 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:51.210316743 +0000 UTC m=+62.794095080" watchObservedRunningTime="2026-09-16 07:34:51.211114933 +0000 UTC m=+62.794893262" Sep 16 07:34:51.315211 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:51.315180 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:51.504431 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:51.504393 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:51.563260 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:51.563221 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:52.103038 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:52.103004 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:52.153831 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:52.153801 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:52.512357 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:52.512312 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:52.701799 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:52.701765 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:52.895263 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:52.895180 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:54.619468 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:54.619424 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9852905b-319e-4378-8c40-266974260880-metrics-tls\") pod \"dns-default-qks9b\" (UID: \"9852905b-319e-4378-8c40-266974260880\") " pod="openshift-dns/dns-default-qks9b" Sep 16 07:34:54.619468 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:54.619472 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/50be9bc4-8115-41e1-afb4-51a305538a5a-registry-tls\") pod \"image-registry-9ff4cbfdc-kvcdr\" (UID: \"50be9bc4-8115-41e1-afb4-51a305538a5a\") " pod="openshift-image-registry/image-registry-9ff4cbfdc-kvcdr" Sep 16 07:34:54.619938 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:54.619499 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aca15303-6546-4428-bb32-539bb25d2486-metrics-certs\") pod \"network-metrics-daemon-94n4h\" (UID: \"aca15303-6546-4428-bb32-539bb25d2486\") " pod="openshift-multus/network-metrics-daemon-94n4h" Sep 16 07:34:54.619938 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:54.619534 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4152b2c2-a19c-44fa-96f5-dc3eb661aa9b-original-pull-secret\") pod \"global-pull-secret-syncer-wbslq\" (UID: \"4152b2c2-a19c-44fa-96f5-dc3eb661aa9b\") " pod="kube-system/global-pull-secret-syncer-wbslq" Sep 16 07:34:54.621944 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:54.621913 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9852905b-319e-4378-8c40-266974260880-metrics-tls\") pod \"dns-default-qks9b\" (UID: \"9852905b-319e-4378-8c40-266974260880\") " pod="openshift-dns/dns-default-qks9b" Sep 16 07:34:54.622073 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:54.621945 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 16 07:34:54.622073 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:54.621961 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 16 07:34:54.622073 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:54.622010 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/50be9bc4-8115-41e1-afb4-51a305538a5a-registry-tls\") pod \"image-registry-9ff4cbfdc-kvcdr\" (UID: \"50be9bc4-8115-41e1-afb4-51a305538a5a\") " pod="openshift-image-registry/image-registry-9ff4cbfdc-kvcdr" Sep 16 07:34:54.632939 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:54.632902 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4152b2c2-a19c-44fa-96f5-dc3eb661aa9b-original-pull-secret\") pod \"global-pull-secret-syncer-wbslq\" (UID: \"4152b2c2-a19c-44fa-96f5-dc3eb661aa9b\") " pod="kube-system/global-pull-secret-syncer-wbslq" Sep 16 07:34:54.632939 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:54.632924 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aca15303-6546-4428-bb32-539bb25d2486-metrics-certs\") pod \"network-metrics-daemon-94n4h\" (UID: \"aca15303-6546-4428-bb32-539bb25d2486\") " pod="openshift-multus/network-metrics-daemon-94n4h" Sep 16 07:34:54.719843 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:54.719792 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnt8f\" (UniqueName: \"kubernetes.io/projected/3c6593a0-6ec1-4945-b929-f94446ee82b1-kube-api-access-hnt8f\") pod \"network-check-target-vx6vh\" (UID: \"3c6593a0-6ec1-4945-b929-f94446ee82b1\") " pod="openshift-network-diagnostics/network-check-target-vx6vh" Sep 16 07:34:54.721985 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:54.721961 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 16 07:34:54.732566 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:54.732539 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 16 07:34:54.743732 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:54.743705 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnt8f\" (UniqueName: \"kubernetes.io/projected/3c6593a0-6ec1-4945-b929-f94446ee82b1-kube-api-access-hnt8f\") pod \"network-check-target-vx6vh\" (UID: \"3c6593a0-6ec1-4945-b929-f94446ee82b1\") " pod="openshift-network-diagnostics/network-check-target-vx6vh" Sep 16 07:34:54.847221 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:54.847190 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lz8hv\"" Sep 16 07:34:54.854873 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:54.854847 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-zf7hx\"" Sep 16 07:34:54.855645 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:54.855630 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vx6vh" Sep 16 07:34:54.855875 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:54.855858 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2msf4\"" Sep 16 07:34:54.860080 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:54.860064 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wbslq" Sep 16 07:34:54.863797 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:54.863780 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-94n4h" Sep 16 07:34:54.864986 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:54.864967 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qks9b" Sep 16 07:34:54.870827 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:54.870776 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6bpwz\"" Sep 16 07:34:54.879628 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:54.879580 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9ff4cbfdc-kvcdr" Sep 16 07:34:55.134261 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:55.134186 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vx6vh"] Sep 16 07:34:55.136784 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:55.136758 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wbslq"] Sep 16 07:34:55.137550 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:34:55.137528 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c6593a0_6ec1_4945_b929_f94446ee82b1.slice/crio-a5864c073ae5b1ee397a74f242f32e74bfdfb472002d7bd5dc5bac0e28ac5747 WatchSource:0}: Error finding container a5864c073ae5b1ee397a74f242f32e74bfdfb472002d7bd5dc5bac0e28ac5747: Status 404 returned error can't find the container with id a5864c073ae5b1ee397a74f242f32e74bfdfb472002d7bd5dc5bac0e28ac5747 Sep 16 07:34:55.139726 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:34:55.139710 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4152b2c2_a19c_44fa_96f5_dc3eb661aa9b.slice/crio-d075968ab851bf9ff6ea938348574fa342b514aef6e83297c3af29c794917d12 WatchSource:0}: Error finding container d075968ab851bf9ff6ea938348574fa342b514aef6e83297c3af29c794917d12: Status 404 returned error can't find the container with id d075968ab851bf9ff6ea938348574fa342b514aef6e83297c3af29c794917d12 Sep 16 07:34:55.180935 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:55.180903 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-94n4h"] Sep 16 07:34:55.184613 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:34:55.184576 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaca15303_6546_4428_bb32_539bb25d2486.slice/crio-8812056ab7173652616ceed09e521e2d396fa9aae28d4a5ebd5f647920d80441 WatchSource:0}: Error finding container 8812056ab7173652616ceed09e521e2d396fa9aae28d4a5ebd5f647920d80441: Status 404 returned error can't find the container with id 8812056ab7173652616ceed09e521e2d396fa9aae28d4a5ebd5f647920d80441 Sep 16 07:34:55.184804 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:55.184782 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-9ff4cbfdc-kvcdr"] Sep 16 07:34:55.185876 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:55.185783 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qks9b"] Sep 16 07:34:55.188024 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:34:55.188001 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9852905b_319e_4378_8c40_266974260880.slice/crio-0de494243bd2f9e9bc42c8de9b8083b917d2c6e1d718ec1f00b0294b6940a57e WatchSource:0}: Error finding container 0de494243bd2f9e9bc42c8de9b8083b917d2c6e1d718ec1f00b0294b6940a57e: Status 404 returned error can't find the container with id 0de494243bd2f9e9bc42c8de9b8083b917d2c6e1d718ec1f00b0294b6940a57e Sep 16 07:34:55.189226 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:34:55.189202 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50be9bc4_8115_41e1_afb4_51a305538a5a.slice/crio-778f4d7d37bfd475ecf93af37c3fbf7d1f5182c1af95052a41677ab713012ac6 WatchSource:0}: Error finding container 778f4d7d37bfd475ecf93af37c3fbf7d1f5182c1af95052a41677ab713012ac6: Status 404 returned error can't find the container with id 778f4d7d37bfd475ecf93af37c3fbf7d1f5182c1af95052a41677ab713012ac6 Sep 16 07:34:55.204759 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:55.204725 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9ff4cbfdc-kvcdr" event={"ID":"50be9bc4-8115-41e1-afb4-51a305538a5a","Type":"ContainerStarted","Data":"778f4d7d37bfd475ecf93af37c3fbf7d1f5182c1af95052a41677ab713012ac6"} Sep 16 07:34:55.206053 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:55.206029 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wbslq" event={"ID":"4152b2c2-a19c-44fa-96f5-dc3eb661aa9b","Type":"ContainerStarted","Data":"d075968ab851bf9ff6ea938348574fa342b514aef6e83297c3af29c794917d12"} Sep 16 07:34:55.207132 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:55.207093 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vx6vh" event={"ID":"3c6593a0-6ec1-4945-b929-f94446ee82b1","Type":"ContainerStarted","Data":"a5864c073ae5b1ee397a74f242f32e74bfdfb472002d7bd5dc5bac0e28ac5747"} Sep 16 07:34:55.208305 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:55.208281 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qks9b" event={"ID":"9852905b-319e-4378-8c40-266974260880","Type":"ContainerStarted","Data":"0de494243bd2f9e9bc42c8de9b8083b917d2c6e1d718ec1f00b0294b6940a57e"} Sep 16 07:34:55.209606 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:55.209584 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-94n4h" event={"ID":"aca15303-6546-4428-bb32-539bb25d2486","Type":"ContainerStarted","Data":"8812056ab7173652616ceed09e521e2d396fa9aae28d4a5ebd5f647920d80441"} Sep 16 07:34:56.224153 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.224115 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9ff4cbfdc-kvcdr" event={"ID":"50be9bc4-8115-41e1-afb4-51a305538a5a","Type":"ContainerStarted","Data":"7b06377f68ebd4e761782c675843266b3711ed1579f8a9324df57f930da073c3"} Sep 16 07:34:56.224779 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.224739 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-9ff4cbfdc-kvcdr" Sep 16 07:34:56.248590 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.248137 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-9ff4cbfdc-kvcdr" podStartSLOduration=65.248114914 podStartE2EDuration="1m5.248114914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:33:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:56.246342046 +0000 UTC m=+67.830120387" watchObservedRunningTime="2026-09-16 07:34:56.248114914 +0000 UTC m=+67.831893267" Sep 16 07:34:56.340600 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.340562 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b768d5f67-dlpr9"] Sep 16 07:34:56.344078 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.344041 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b768d5f67-dlpr9" Sep 16 07:34:56.346802 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.346723 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Sep 16 07:34:56.347068 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.347015 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 16 07:34:56.349035 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.348220 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 16 07:34:56.349035 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.348803 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 16 07:34:56.355735 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.353462 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-9ff4cbfdc-kvcdr"] Sep 16 07:34:56.356281 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.356259 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b768d5f67-dlpr9"] Sep 16 07:34:56.433371 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.433287 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nrlw\" (UniqueName: \"kubernetes.io/projected/9f47b6bb-1552-4339-95c7-d23254409d7b-kube-api-access-2nrlw\") pod \"klusterlet-addon-workmgr-5b768d5f67-dlpr9\" (UID: \"9f47b6bb-1552-4339-95c7-d23254409d7b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b768d5f67-dlpr9" Sep 16 07:34:56.433371 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.433343 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/9f47b6bb-1552-4339-95c7-d23254409d7b-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b768d5f67-dlpr9\" (UID: \"9f47b6bb-1552-4339-95c7-d23254409d7b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b768d5f67-dlpr9" Sep 16 07:34:56.433618 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.433386 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f47b6bb-1552-4339-95c7-d23254409d7b-tmp\") pod \"klusterlet-addon-workmgr-5b768d5f67-dlpr9\" (UID: \"9f47b6bb-1552-4339-95c7-d23254409d7b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b768d5f67-dlpr9" Sep 16 07:34:56.444565 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.443458 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-wvs7b"] Sep 16 07:34:56.448408 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.446674 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-wvs7b" Sep 16 07:34:56.454274 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.453584 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-7ff2t\"" Sep 16 07:34:56.454274 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.453785 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-56676c689d-5tjq7"] Sep 16 07:34:56.454274 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.453901 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 16 07:34:56.454274 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.454112 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 16 07:34:56.457374 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.457352 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56676c689d-5tjq7" Sep 16 07:34:56.462159 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.462082 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-wvs7b"] Sep 16 07:34:56.468251 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.468224 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56676c689d-5tjq7"] Sep 16 07:34:56.489215 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.489125 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:56.534308 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.534268 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nrlw\" (UniqueName: \"kubernetes.io/projected/9f47b6bb-1552-4339-95c7-d23254409d7b-kube-api-access-2nrlw\") pod \"klusterlet-addon-workmgr-5b768d5f67-dlpr9\" (UID: \"9f47b6bb-1552-4339-95c7-d23254409d7b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b768d5f67-dlpr9" Sep 16 07:34:56.534489 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.534321 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rwsj\" (UniqueName: \"kubernetes.io/projected/3c72acfc-adc3-4592-b81e-e841ea01b360-kube-api-access-2rwsj\") pod \"image-registry-56676c689d-5tjq7\" (UID: \"3c72acfc-adc3-4592-b81e-e841ea01b360\") " pod="openshift-image-registry/image-registry-56676c689d-5tjq7" Sep 16 07:34:56.534489 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.534396 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3c72acfc-adc3-4592-b81e-e841ea01b360-bound-sa-token\") pod \"image-registry-56676c689d-5tjq7\" (UID: \"3c72acfc-adc3-4592-b81e-e841ea01b360\") " pod="openshift-image-registry/image-registry-56676c689d-5tjq7" Sep 16 07:34:56.534489 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.534436 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cqw5\" (UniqueName: \"kubernetes.io/projected/eccfa20f-4a48-4588-a7fa-a82fe43e1ccc-kube-api-access-6cqw5\") pod \"downloads-76f99584f4-wvs7b\" (UID: \"eccfa20f-4a48-4588-a7fa-a82fe43e1ccc\") " pod="openshift-console/downloads-76f99584f4-wvs7b" Sep 16 07:34:56.534660 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.534508 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3c72acfc-adc3-4592-b81e-e841ea01b360-registry-certificates\") pod \"image-registry-56676c689d-5tjq7\" (UID: \"3c72acfc-adc3-4592-b81e-e841ea01b360\") " pod="openshift-image-registry/image-registry-56676c689d-5tjq7" Sep 16 07:34:56.534660 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.534541 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/9f47b6bb-1552-4339-95c7-d23254409d7b-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b768d5f67-dlpr9\" (UID: \"9f47b6bb-1552-4339-95c7-d23254409d7b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b768d5f67-dlpr9" Sep 16 07:34:56.534660 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.534576 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3c72acfc-adc3-4592-b81e-e841ea01b360-trusted-ca\") pod \"image-registry-56676c689d-5tjq7\" (UID: \"3c72acfc-adc3-4592-b81e-e841ea01b360\") " pod="openshift-image-registry/image-registry-56676c689d-5tjq7" Sep 16 07:34:56.534660 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.534605 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f47b6bb-1552-4339-95c7-d23254409d7b-tmp\") pod \"klusterlet-addon-workmgr-5b768d5f67-dlpr9\" (UID: \"9f47b6bb-1552-4339-95c7-d23254409d7b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b768d5f67-dlpr9" Sep 16 07:34:56.534660 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.534635 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3c72acfc-adc3-4592-b81e-e841ea01b360-installation-pull-secrets\") pod \"image-registry-56676c689d-5tjq7\" (UID: \"3c72acfc-adc3-4592-b81e-e841ea01b360\") " pod="openshift-image-registry/image-registry-56676c689d-5tjq7" Sep 16 07:34:56.534906 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.534807 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3c72acfc-adc3-4592-b81e-e841ea01b360-registry-tls\") pod \"image-registry-56676c689d-5tjq7\" (UID: \"3c72acfc-adc3-4592-b81e-e841ea01b360\") " pod="openshift-image-registry/image-registry-56676c689d-5tjq7" Sep 16 07:34:56.534906 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.534854 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3c72acfc-adc3-4592-b81e-e841ea01b360-image-registry-private-configuration\") pod \"image-registry-56676c689d-5tjq7\" (UID: \"3c72acfc-adc3-4592-b81e-e841ea01b360\") " pod="openshift-image-registry/image-registry-56676c689d-5tjq7" Sep 16 07:34:56.534906 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.534881 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3c72acfc-adc3-4592-b81e-e841ea01b360-ca-trust-extracted\") pod \"image-registry-56676c689d-5tjq7\" (UID: \"3c72acfc-adc3-4592-b81e-e841ea01b360\") " pod="openshift-image-registry/image-registry-56676c689d-5tjq7" Sep 16 07:34:56.535228 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.535148 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9f47b6bb-1552-4339-95c7-d23254409d7b-tmp\") pod \"klusterlet-addon-workmgr-5b768d5f67-dlpr9\" (UID: \"9f47b6bb-1552-4339-95c7-d23254409d7b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b768d5f67-dlpr9" Sep 16 07:34:56.538687 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.538248 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/9f47b6bb-1552-4339-95c7-d23254409d7b-klusterlet-config\") pod \"klusterlet-addon-workmgr-5b768d5f67-dlpr9\" (UID: \"9f47b6bb-1552-4339-95c7-d23254409d7b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b768d5f67-dlpr9" Sep 16 07:34:56.548231 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.548173 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nrlw\" (UniqueName: \"kubernetes.io/projected/9f47b6bb-1552-4339-95c7-d23254409d7b-kube-api-access-2nrlw\") pod \"klusterlet-addon-workmgr-5b768d5f67-dlpr9\" (UID: \"9f47b6bb-1552-4339-95c7-d23254409d7b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b768d5f67-dlpr9" Sep 16 07:34:56.579576 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.579537 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:56.636335 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.636295 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3c72acfc-adc3-4592-b81e-e841ea01b360-registry-tls\") pod \"image-registry-56676c689d-5tjq7\" (UID: \"3c72acfc-adc3-4592-b81e-e841ea01b360\") " pod="openshift-image-registry/image-registry-56676c689d-5tjq7" Sep 16 07:34:56.636522 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.636353 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3c72acfc-adc3-4592-b81e-e841ea01b360-image-registry-private-configuration\") pod \"image-registry-56676c689d-5tjq7\" (UID: \"3c72acfc-adc3-4592-b81e-e841ea01b360\") " pod="openshift-image-registry/image-registry-56676c689d-5tjq7" Sep 16 07:34:56.636522 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.636379 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3c72acfc-adc3-4592-b81e-e841ea01b360-ca-trust-extracted\") pod \"image-registry-56676c689d-5tjq7\" (UID: \"3c72acfc-adc3-4592-b81e-e841ea01b360\") " pod="openshift-image-registry/image-registry-56676c689d-5tjq7" Sep 16 07:34:56.636522 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.636416 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rwsj\" (UniqueName: \"kubernetes.io/projected/3c72acfc-adc3-4592-b81e-e841ea01b360-kube-api-access-2rwsj\") pod \"image-registry-56676c689d-5tjq7\" (UID: \"3c72acfc-adc3-4592-b81e-e841ea01b360\") " pod="openshift-image-registry/image-registry-56676c689d-5tjq7" Sep 16 07:34:56.636522 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.636452 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3c72acfc-adc3-4592-b81e-e841ea01b360-bound-sa-token\") pod \"image-registry-56676c689d-5tjq7\" (UID: \"3c72acfc-adc3-4592-b81e-e841ea01b360\") " pod="openshift-image-registry/image-registry-56676c689d-5tjq7" Sep 16 07:34:56.636522 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.636489 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cqw5\" (UniqueName: \"kubernetes.io/projected/eccfa20f-4a48-4588-a7fa-a82fe43e1ccc-kube-api-access-6cqw5\") pod \"downloads-76f99584f4-wvs7b\" (UID: \"eccfa20f-4a48-4588-a7fa-a82fe43e1ccc\") " pod="openshift-console/downloads-76f99584f4-wvs7b" Sep 16 07:34:56.636848 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.636527 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3c72acfc-adc3-4592-b81e-e841ea01b360-registry-certificates\") pod \"image-registry-56676c689d-5tjq7\" (UID: \"3c72acfc-adc3-4592-b81e-e841ea01b360\") " pod="openshift-image-registry/image-registry-56676c689d-5tjq7" Sep 16 07:34:56.636848 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.636561 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3c72acfc-adc3-4592-b81e-e841ea01b360-trusted-ca\") pod \"image-registry-56676c689d-5tjq7\" (UID: \"3c72acfc-adc3-4592-b81e-e841ea01b360\") " pod="openshift-image-registry/image-registry-56676c689d-5tjq7" Sep 16 07:34:56.636848 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.636593 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3c72acfc-adc3-4592-b81e-e841ea01b360-installation-pull-secrets\") pod \"image-registry-56676c689d-5tjq7\" (UID: \"3c72acfc-adc3-4592-b81e-e841ea01b360\") " pod="openshift-image-registry/image-registry-56676c689d-5tjq7" Sep 16 07:34:56.637586 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.637558 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/3c72acfc-adc3-4592-b81e-e841ea01b360-ca-trust-extracted\") pod \"image-registry-56676c689d-5tjq7\" (UID: \"3c72acfc-adc3-4592-b81e-e841ea01b360\") " pod="openshift-image-registry/image-registry-56676c689d-5tjq7" Sep 16 07:34:56.639402 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.638567 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/3c72acfc-adc3-4592-b81e-e841ea01b360-registry-certificates\") pod \"image-registry-56676c689d-5tjq7\" (UID: \"3c72acfc-adc3-4592-b81e-e841ea01b360\") " pod="openshift-image-registry/image-registry-56676c689d-5tjq7" Sep 16 07:34:56.639402 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.639359 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3c72acfc-adc3-4592-b81e-e841ea01b360-trusted-ca\") pod \"image-registry-56676c689d-5tjq7\" (UID: \"3c72acfc-adc3-4592-b81e-e841ea01b360\") " pod="openshift-image-registry/image-registry-56676c689d-5tjq7" Sep 16 07:34:56.641493 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.641457 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/3c72acfc-adc3-4592-b81e-e841ea01b360-installation-pull-secrets\") pod \"image-registry-56676c689d-5tjq7\" (UID: \"3c72acfc-adc3-4592-b81e-e841ea01b360\") " pod="openshift-image-registry/image-registry-56676c689d-5tjq7" Sep 16 07:34:56.652037 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.652010 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/3c72acfc-adc3-4592-b81e-e841ea01b360-image-registry-private-configuration\") pod \"image-registry-56676c689d-5tjq7\" (UID: \"3c72acfc-adc3-4592-b81e-e841ea01b360\") " pod="openshift-image-registry/image-registry-56676c689d-5tjq7" Sep 16 07:34:56.659722 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.659656 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rwsj\" (UniqueName: \"kubernetes.io/projected/3c72acfc-adc3-4592-b81e-e841ea01b360-kube-api-access-2rwsj\") pod \"image-registry-56676c689d-5tjq7\" (UID: \"3c72acfc-adc3-4592-b81e-e841ea01b360\") " pod="openshift-image-registry/image-registry-56676c689d-5tjq7" Sep 16 07:34:56.662090 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.662032 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cqw5\" (UniqueName: \"kubernetes.io/projected/eccfa20f-4a48-4588-a7fa-a82fe43e1ccc-kube-api-access-6cqw5\") pod \"downloads-76f99584f4-wvs7b\" (UID: \"eccfa20f-4a48-4588-a7fa-a82fe43e1ccc\") " pod="openshift-console/downloads-76f99584f4-wvs7b" Sep 16 07:34:56.662855 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.662829 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3c72acfc-adc3-4592-b81e-e841ea01b360-bound-sa-token\") pod \"image-registry-56676c689d-5tjq7\" (UID: \"3c72acfc-adc3-4592-b81e-e841ea01b360\") " pod="openshift-image-registry/image-registry-56676c689d-5tjq7" Sep 16 07:34:56.663900 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.663875 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/3c72acfc-adc3-4592-b81e-e841ea01b360-registry-tls\") pod \"image-registry-56676c689d-5tjq7\" (UID: \"3c72acfc-adc3-4592-b81e-e841ea01b360\") " pod="openshift-image-registry/image-registry-56676c689d-5tjq7" Sep 16 07:34:56.665524 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.665330 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b768d5f67-dlpr9" Sep 16 07:34:56.679264 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.679208 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:56.767595 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.767539 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-wvs7b" Sep 16 07:34:56.775245 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.774969 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56676c689d-5tjq7" Sep 16 07:34:56.778868 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.778766 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:56.972787 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:56.972752 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b768d5f67-dlpr9"] Sep 16 07:34:57.232979 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:57.232902 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-94n4h" event={"ID":"aca15303-6546-4428-bb32-539bb25d2486","Type":"ContainerStarted","Data":"dbcaf4472d159561610d0ee9d9085d9562388bac41eb5f1f732996443e3d35d6"} Sep 16 07:34:57.347609 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:34:57.347575 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f47b6bb_1552_4339_95c7_d23254409d7b.slice/crio-3a341990bb756adb229d3fcbe5ac45516b803472b1b9ebfc97f4196368571bb1 WatchSource:0}: Error finding container 3a341990bb756adb229d3fcbe5ac45516b803472b1b9ebfc97f4196368571bb1: Status 404 returned error can't find the container with id 3a341990bb756adb229d3fcbe5ac45516b803472b1b9ebfc97f4196368571bb1 Sep 16 07:34:57.356090 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:57.356033 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:57.643532 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:57.643500 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-wvs7b"] Sep 16 07:34:57.650226 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:57.650194 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56676c689d-5tjq7"] Sep 16 07:34:57.658225 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:34:57.658047 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c72acfc_adc3_4592_b81e_e841ea01b360.slice/crio-e98e1dfaba63bbca31e4863c3a820b2431316fbcb9419dd9d36682bbb3c3a4ac WatchSource:0}: Error finding container e98e1dfaba63bbca31e4863c3a820b2431316fbcb9419dd9d36682bbb3c3a4ac: Status 404 returned error can't find the container with id e98e1dfaba63bbca31e4863c3a820b2431316fbcb9419dd9d36682bbb3c3a4ac Sep 16 07:34:57.747940 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:57.747893 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:57.804341 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:57.804243 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:58.239333 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:58.239290 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-94n4h" event={"ID":"aca15303-6546-4428-bb32-539bb25d2486","Type":"ContainerStarted","Data":"ace539a037a684cafdec1bc65a1a51fead471b1d880a892cd624c9ae5e70f13b"} Sep 16 07:34:58.240645 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:58.240619 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-wvs7b" event={"ID":"eccfa20f-4a48-4588-a7fa-a82fe43e1ccc","Type":"ContainerStarted","Data":"ef8d65e08dbb56de04964c47cb1bf75f43ad283fb6fa550e93ba9cd688f90ff2"} Sep 16 07:34:58.241969 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:58.241938 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b768d5f67-dlpr9" event={"ID":"9f47b6bb-1552-4339-95c7-d23254409d7b","Type":"ContainerStarted","Data":"3a341990bb756adb229d3fcbe5ac45516b803472b1b9ebfc97f4196368571bb1"} Sep 16 07:34:58.244150 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:58.244125 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qks9b" event={"ID":"9852905b-319e-4378-8c40-266974260880","Type":"ContainerStarted","Data":"1cd908b7a62b63c39b119b3dcd48e0675b21f53f0f77de6c1937b5c79c71f2a3"} Sep 16 07:34:58.244274 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:58.244172 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qks9b" event={"ID":"9852905b-319e-4378-8c40-266974260880","Type":"ContainerStarted","Data":"4e591965c9d0aec696ac2b915fa3bbb36e6516f933b98e6bfbe2a2e923eedf48"} Sep 16 07:34:58.244274 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:58.244219 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qks9b" Sep 16 07:34:58.245711 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:58.245684 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56676c689d-5tjq7" event={"ID":"3c72acfc-adc3-4592-b81e-e841ea01b360","Type":"ContainerStarted","Data":"0cfabbc2ebe467e3458613a7eee181811efdc0acefd9dee868fd4f47fd24bb94"} Sep 16 07:34:58.245803 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:58.245733 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56676c689d-5tjq7" event={"ID":"3c72acfc-adc3-4592-b81e-e841ea01b360","Type":"ContainerStarted","Data":"e98e1dfaba63bbca31e4863c3a820b2431316fbcb9419dd9d36682bbb3c3a4ac"} Sep 16 07:34:58.245920 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:58.245897 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-56676c689d-5tjq7" Sep 16 07:34:58.255660 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:58.255594 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-94n4h" podStartSLOduration=67.841481352 podStartE2EDuration="1m9.255577488s" totalImagesPullingTime="1.414096136s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:33:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:58.254778117 +0000 UTC m=+69.838556455" watchObservedRunningTime="2026-09-16 07:34:58.255577488 +0000 UTC m=+69.839355826" Sep 16 07:34:58.270580 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:58.270522 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-qks9b" podStartSLOduration=34.052497499 podStartE2EDuration="36.27050343s" totalImagesPullingTime="2.218005931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:34:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:58.269742855 +0000 UTC m=+69.853521193" watchObservedRunningTime="2026-09-16 07:34:58.27050343 +0000 UTC m=+69.854281769" Sep 16 07:34:58.290179 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:58.290093 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-56676c689d-5tjq7" podStartSLOduration=2.290073541 podStartE2EDuration="2.290073541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:34:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:34:58.289022642 +0000 UTC m=+69.872800985" watchObservedRunningTime="2026-09-16 07:34:58.290073541 +0000 UTC m=+69.873851878" Sep 16 07:34:58.555054 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:58.554953 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:58.954252 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:58.954201 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:59.006088 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:59.006052 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:59.132529 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:59.132486 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:34:59.182719 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:34:59.182676 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:00.046437 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:00.046394 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:00.154527 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:00.154488 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:00.246141 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:00.246084 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rrcsn"] Sep 16 07:35:00.252583 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:00.252536 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rrcsn" Sep 16 07:35:00.256116 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:00.256069 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 16 07:35:00.256269 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:00.256069 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-rvcvf\"" Sep 16 07:35:00.264746 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:00.264717 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rrcsn"] Sep 16 07:35:00.273071 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:00.273027 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:00.361150 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:00.361056 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:00.370912 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:00.370880 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2b09201b-f78c-4de1-91ff-b583cab8316e-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-rrcsn\" (UID: \"2b09201b-f78c-4de1-91ff-b583cab8316e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rrcsn" Sep 16 07:35:00.471477 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:00.471445 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2b09201b-f78c-4de1-91ff-b583cab8316e-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-rrcsn\" (UID: \"2b09201b-f78c-4de1-91ff-b583cab8316e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rrcsn" Sep 16 07:35:00.474634 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:00.474604 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2b09201b-f78c-4de1-91ff-b583cab8316e-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-rrcsn\" (UID: \"2b09201b-f78c-4de1-91ff-b583cab8316e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rrcsn" Sep 16 07:35:00.564543 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:00.564319 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rrcsn" Sep 16 07:35:00.879554 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:00.879515 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rrcsn"] Sep 16 07:35:00.884582 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:35:00.884549 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b09201b_f78c_4de1_91ff_b583cab8316e.slice/crio-2c3375c44f1ca67a69e49ce0ba247b2b2bf1ce86f046478d62bfc430a307ce72 WatchSource:0}: Error finding container 2c3375c44f1ca67a69e49ce0ba247b2b2bf1ce86f046478d62bfc430a307ce72: Status 404 returned error can't find the container with id 2c3375c44f1ca67a69e49ce0ba247b2b2bf1ce86f046478d62bfc430a307ce72 Sep 16 07:35:01.260078 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:01.260040 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wbslq" event={"ID":"4152b2c2-a19c-44fa-96f5-dc3eb661aa9b","Type":"ContainerStarted","Data":"91237c16b66db49891c7c3c29d07a6cf5a8ca6342c8b412db83aafaa6c27d5fa"} Sep 16 07:35:01.262305 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:01.262278 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vx6vh" event={"ID":"3c6593a0-6ec1-4945-b929-f94446ee82b1","Type":"ContainerStarted","Data":"5206585a82fc004e56f0880fa0543ea302d7fe6ec0c03e8bad8b6d4ffab41653"} Sep 16 07:35:01.262433 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:01.262357 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vx6vh" Sep 16 07:35:01.263856 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:01.263823 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rrcsn" event={"ID":"2b09201b-f78c-4de1-91ff-b583cab8316e","Type":"ContainerStarted","Data":"2c3375c44f1ca67a69e49ce0ba247b2b2bf1ce86f046478d62bfc430a307ce72"} Sep 16 07:35:01.275966 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:01.275232 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-wbslq" podStartSLOduration=66.989144863 podStartE2EDuration="1m12.275211404s" totalImagesPullingTime="5.286066541s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 07:33:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:35:01.273762033 +0000 UTC m=+72.857540374" watchObservedRunningTime="2026-09-16 07:35:01.275211404 +0000 UTC m=+72.858989761" Sep 16 07:35:01.288681 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:01.288610 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vx6vh" podStartSLOduration=67.000341095 podStartE2EDuration="1m12.288591522s" totalImagesPullingTime="5.288250427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:33:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:35:01.287404451 +0000 UTC m=+72.871182819" watchObservedRunningTime="2026-09-16 07:35:01.288591522 +0000 UTC m=+72.872369860" Sep 16 07:35:02.137838 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:02.137798 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:02.538957 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:02.538920 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:02.713736 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:02.713695 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:04.276946 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:04.276905 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b768d5f67-dlpr9" event={"ID":"9f47b6bb-1552-4339-95c7-d23254409d7b","Type":"ContainerStarted","Data":"e5e1d92603080bb141d99adf935ce591f7539a0c5fbf1f4e48c2d89f91fae923"} Sep 16 07:35:04.277365 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:04.277064 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b768d5f67-dlpr9" Sep 16 07:35:04.278702 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:04.278678 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rrcsn" event={"ID":"2b09201b-f78c-4de1-91ff-b583cab8316e","Type":"ContainerStarted","Data":"6a0f9144ccb9ade4f2caf4700d29900bbf98bbe9e340ebb48f7f1bba96893eed"} Sep 16 07:35:04.278895 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:04.278867 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rrcsn" Sep 16 07:35:04.279062 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:04.279039 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b768d5f67-dlpr9" Sep 16 07:35:04.283591 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:04.283567 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rrcsn" Sep 16 07:35:04.291052 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:04.290995 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5b768d5f67-dlpr9" podStartSLOduration=1.5513539 podStartE2EDuration="8.290976029s" totalImagesPullingTime="6.739622129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:34:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:35:04.290407749 +0000 UTC m=+75.874186110" watchObservedRunningTime="2026-09-16 07:35:04.290976029 +0000 UTC m=+75.874754385" Sep 16 07:35:04.317723 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:04.317666 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-rrcsn" podStartSLOduration=1.131848297 podStartE2EDuration="4.317644012s" totalImagesPullingTime="3.185795715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:35:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:35:04.316804639 +0000 UTC m=+75.900583006" watchObservedRunningTime="2026-09-16 07:35:04.317644012 +0000 UTC m=+75.901422353" Sep 16 07:35:04.712774 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:04.712738 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:04.772084 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:04.772038 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:05.269884 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:05.269843 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:05.293375 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:05.293341 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-cnsvs"] Sep 16 07:35:05.296898 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:05.296867 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-cnsvs" Sep 16 07:35:05.300290 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:05.300090 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 16 07:35:05.300660 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:05.300635 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 16 07:35:05.301198 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:05.301179 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 16 07:35:05.301287 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:05.301215 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 16 07:35:05.301287 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:05.301239 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 16 07:35:05.301667 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:05.301648 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-h6dcb\"" Sep 16 07:35:05.307927 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:05.307901 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-cnsvs"] Sep 16 07:35:05.339504 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:05.339450 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:05.415686 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:05.415651 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7dbc90e5-a266-40cd-b099-35ec42755ec4-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-cnsvs\" (UID: \"7dbc90e5-a266-40cd-b099-35ec42755ec4\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cnsvs" Sep 16 07:35:05.415879 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:05.415739 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7dbc90e5-a266-40cd-b099-35ec42755ec4-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-cnsvs\" (UID: \"7dbc90e5-a266-40cd-b099-35ec42755ec4\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cnsvs" Sep 16 07:35:05.415957 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:05.415936 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7dbc90e5-a266-40cd-b099-35ec42755ec4-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-cnsvs\" (UID: \"7dbc90e5-a266-40cd-b099-35ec42755ec4\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cnsvs" Sep 16 07:35:05.416043 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:05.416019 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzfqn\" (UniqueName: \"kubernetes.io/projected/7dbc90e5-a266-40cd-b099-35ec42755ec4-kube-api-access-nzfqn\") pod \"prometheus-operator-68bcd7f987-cnsvs\" (UID: \"7dbc90e5-a266-40cd-b099-35ec42755ec4\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cnsvs" Sep 16 07:35:05.517006 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:05.516954 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7dbc90e5-a266-40cd-b099-35ec42755ec4-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-cnsvs\" (UID: \"7dbc90e5-a266-40cd-b099-35ec42755ec4\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cnsvs" Sep 16 07:35:05.517221 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:05.517035 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7dbc90e5-a266-40cd-b099-35ec42755ec4-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-cnsvs\" (UID: \"7dbc90e5-a266-40cd-b099-35ec42755ec4\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cnsvs" Sep 16 07:35:05.517221 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:05.517088 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzfqn\" (UniqueName: \"kubernetes.io/projected/7dbc90e5-a266-40cd-b099-35ec42755ec4-kube-api-access-nzfqn\") pod \"prometheus-operator-68bcd7f987-cnsvs\" (UID: \"7dbc90e5-a266-40cd-b099-35ec42755ec4\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cnsvs" Sep 16 07:35:05.517221 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:05.517158 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7dbc90e5-a266-40cd-b099-35ec42755ec4-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-cnsvs\" (UID: \"7dbc90e5-a266-40cd-b099-35ec42755ec4\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cnsvs" Sep 16 07:35:05.518030 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:05.517967 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7dbc90e5-a266-40cd-b099-35ec42755ec4-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-cnsvs\" (UID: \"7dbc90e5-a266-40cd-b099-35ec42755ec4\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cnsvs" Sep 16 07:35:05.519955 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:05.519902 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7dbc90e5-a266-40cd-b099-35ec42755ec4-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-cnsvs\" (UID: \"7dbc90e5-a266-40cd-b099-35ec42755ec4\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cnsvs" Sep 16 07:35:05.520134 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:05.520116 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7dbc90e5-a266-40cd-b099-35ec42755ec4-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-cnsvs\" (UID: \"7dbc90e5-a266-40cd-b099-35ec42755ec4\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cnsvs" Sep 16 07:35:05.524956 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:05.524920 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzfqn\" (UniqueName: \"kubernetes.io/projected/7dbc90e5-a266-40cd-b099-35ec42755ec4-kube-api-access-nzfqn\") pod \"prometheus-operator-68bcd7f987-cnsvs\" (UID: \"7dbc90e5-a266-40cd-b099-35ec42755ec4\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-cnsvs" Sep 16 07:35:05.550597 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:05.550556 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:05.609570 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:05.609535 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-cnsvs" Sep 16 07:35:05.888948 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:05.888862 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-cnsvs"] Sep 16 07:35:05.893409 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:35:05.893275 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dbc90e5_a266_40cd_b099_35ec42755ec4.slice/crio-11cc632190d533ea2f798817fccbbee98f2af0091346142c270d9912e1c065ba WatchSource:0}: Error finding container 11cc632190d533ea2f798817fccbbee98f2af0091346142c270d9912e1c065ba: Status 404 returned error can't find the container with id 11cc632190d533ea2f798817fccbbee98f2af0091346142c270d9912e1c065ba Sep 16 07:35:06.240456 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:06.240368 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:06.287971 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:06.287932 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-cnsvs" event={"ID":"7dbc90e5-a266-40cd-b099-35ec42755ec4","Type":"ContainerStarted","Data":"11cc632190d533ea2f798817fccbbee98f2af0091346142c270d9912e1c065ba"} Sep 16 07:35:06.333670 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:06.333637 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:07.653728 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:07.653624 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:08.251991 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:08.251958 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qks9b" Sep 16 07:35:08.299654 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:08.299547 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-cnsvs" event={"ID":"7dbc90e5-a266-40cd-b099-35ec42755ec4","Type":"ContainerStarted","Data":"9d645a1bc01ac8f9e6d53eb050a1a02d0a65ecdf30502807e345eeac3e903513"} Sep 16 07:35:08.299654 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:08.299613 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-cnsvs" event={"ID":"7dbc90e5-a266-40cd-b099-35ec42755ec4","Type":"ContainerStarted","Data":"873d6cbca2f70519ef56bfb53fb18c7372fc9d6ec65c0c132c294c6f04b68a50"} Sep 16 07:35:08.315332 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:08.315256 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-cnsvs" podStartSLOduration=1.833991557 podStartE2EDuration="3.31523204s" totalImagesPullingTime="1.481240483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:35:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:35:08.31435781 +0000 UTC m=+79.898136150" watchObservedRunningTime="2026-09-16 07:35:08.31523204 +0000 UTC m=+79.899010381" Sep 16 07:35:09.351117 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:09.351065 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:09.400718 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:09.400660 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:10.399439 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.399405 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:10.619920 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.619885 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:10.635717 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.635685 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-pjv4h"] Sep 16 07:35:10.640262 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.640235 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pjv4h" Sep 16 07:35:10.643950 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.643875 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-8k8qf\"" Sep 16 07:35:10.644263 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.644245 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 16 07:35:10.644547 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.644531 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 16 07:35:10.646773 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.646693 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-b5g4j"] Sep 16 07:35:10.652250 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.652184 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-pjv4h"] Sep 16 07:35:10.652373 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.652328 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-b5g4j" Sep 16 07:35:10.654336 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.654305 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-kqtql\"" Sep 16 07:35:10.654937 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.654683 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 16 07:35:10.654937 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.654710 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 16 07:35:10.654937 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.654685 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 16 07:35:10.708602 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.708568 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:10.762198 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.762158 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/998d1cf1-cf6f-405e-807d-eb18e33f2e54-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-pjv4h\" (UID: \"998d1cf1-cf6f-405e-807d-eb18e33f2e54\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pjv4h" Sep 16 07:35:10.762386 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.762221 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/998d1cf1-cf6f-405e-807d-eb18e33f2e54-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-pjv4h\" (UID: \"998d1cf1-cf6f-405e-807d-eb18e33f2e54\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pjv4h" Sep 16 07:35:10.762386 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.762247 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c41268ea-998a-4cb5-8763-9780490c6432-sys\") pod \"node-exporter-b5g4j\" (UID: \"c41268ea-998a-4cb5-8763-9780490c6432\") " pod="openshift-monitoring/node-exporter-b5g4j" Sep 16 07:35:10.762386 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.762272 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c41268ea-998a-4cb5-8763-9780490c6432-root\") pod \"node-exporter-b5g4j\" (UID: \"c41268ea-998a-4cb5-8763-9780490c6432\") " pod="openshift-monitoring/node-exporter-b5g4j" Sep 16 07:35:10.762386 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.762301 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c41268ea-998a-4cb5-8763-9780490c6432-node-exporter-textfile\") pod \"node-exporter-b5g4j\" (UID: \"c41268ea-998a-4cb5-8763-9780490c6432\") " pod="openshift-monitoring/node-exporter-b5g4j" Sep 16 07:35:10.762386 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.762324 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc7cr\" (UniqueName: \"kubernetes.io/projected/c41268ea-998a-4cb5-8763-9780490c6432-kube-api-access-rc7cr\") pod \"node-exporter-b5g4j\" (UID: \"c41268ea-998a-4cb5-8763-9780490c6432\") " pod="openshift-monitoring/node-exporter-b5g4j" Sep 16 07:35:10.762386 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.762355 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c41268ea-998a-4cb5-8763-9780490c6432-node-exporter-wtmp\") pod \"node-exporter-b5g4j\" (UID: \"c41268ea-998a-4cb5-8763-9780490c6432\") " pod="openshift-monitoring/node-exporter-b5g4j" Sep 16 07:35:10.762663 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.762409 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/998d1cf1-cf6f-405e-807d-eb18e33f2e54-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-pjv4h\" (UID: \"998d1cf1-cf6f-405e-807d-eb18e33f2e54\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pjv4h" Sep 16 07:35:10.762663 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.762443 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c41268ea-998a-4cb5-8763-9780490c6432-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b5g4j\" (UID: \"c41268ea-998a-4cb5-8763-9780490c6432\") " pod="openshift-monitoring/node-exporter-b5g4j" Sep 16 07:35:10.762663 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.762507 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c41268ea-998a-4cb5-8763-9780490c6432-metrics-client-ca\") pod \"node-exporter-b5g4j\" (UID: \"c41268ea-998a-4cb5-8763-9780490c6432\") " pod="openshift-monitoring/node-exporter-b5g4j" Sep 16 07:35:10.762663 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.762536 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlh9z\" (UniqueName: \"kubernetes.io/projected/998d1cf1-cf6f-405e-807d-eb18e33f2e54-kube-api-access-hlh9z\") pod \"openshift-state-metrics-67dcf8c979-pjv4h\" (UID: \"998d1cf1-cf6f-405e-807d-eb18e33f2e54\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pjv4h" Sep 16 07:35:10.762663 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.762567 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c41268ea-998a-4cb5-8763-9780490c6432-node-exporter-tls\") pod \"node-exporter-b5g4j\" (UID: \"c41268ea-998a-4cb5-8763-9780490c6432\") " pod="openshift-monitoring/node-exporter-b5g4j" Sep 16 07:35:10.762663 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.762606 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c41268ea-998a-4cb5-8763-9780490c6432-node-exporter-accelerators-collector-config\") pod \"node-exporter-b5g4j\" (UID: \"c41268ea-998a-4cb5-8763-9780490c6432\") " pod="openshift-monitoring/node-exporter-b5g4j" Sep 16 07:35:10.787459 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.787402 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:10.834974 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.834936 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:10.864140 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.864086 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c41268ea-998a-4cb5-8763-9780490c6432-node-exporter-accelerators-collector-config\") pod \"node-exporter-b5g4j\" (UID: \"c41268ea-998a-4cb5-8763-9780490c6432\") " pod="openshift-monitoring/node-exporter-b5g4j" Sep 16 07:35:10.864329 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.864177 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/998d1cf1-cf6f-405e-807d-eb18e33f2e54-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-pjv4h\" (UID: \"998d1cf1-cf6f-405e-807d-eb18e33f2e54\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pjv4h" Sep 16 07:35:10.864329 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.864219 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/998d1cf1-cf6f-405e-807d-eb18e33f2e54-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-pjv4h\" (UID: \"998d1cf1-cf6f-405e-807d-eb18e33f2e54\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pjv4h" Sep 16 07:35:10.864329 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.864241 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c41268ea-998a-4cb5-8763-9780490c6432-sys\") pod \"node-exporter-b5g4j\" (UID: \"c41268ea-998a-4cb5-8763-9780490c6432\") " pod="openshift-monitoring/node-exporter-b5g4j" Sep 16 07:35:10.864329 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.864263 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c41268ea-998a-4cb5-8763-9780490c6432-root\") pod \"node-exporter-b5g4j\" (UID: \"c41268ea-998a-4cb5-8763-9780490c6432\") " pod="openshift-monitoring/node-exporter-b5g4j" Sep 16 07:35:10.864329 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.864290 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c41268ea-998a-4cb5-8763-9780490c6432-node-exporter-textfile\") pod \"node-exporter-b5g4j\" (UID: \"c41268ea-998a-4cb5-8763-9780490c6432\") " pod="openshift-monitoring/node-exporter-b5g4j" Sep 16 07:35:10.864329 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.864311 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc7cr\" (UniqueName: \"kubernetes.io/projected/c41268ea-998a-4cb5-8763-9780490c6432-kube-api-access-rc7cr\") pod \"node-exporter-b5g4j\" (UID: \"c41268ea-998a-4cb5-8763-9780490c6432\") " pod="openshift-monitoring/node-exporter-b5g4j" Sep 16 07:35:10.864856 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.864342 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c41268ea-998a-4cb5-8763-9780490c6432-node-exporter-wtmp\") pod \"node-exporter-b5g4j\" (UID: \"c41268ea-998a-4cb5-8763-9780490c6432\") " pod="openshift-monitoring/node-exporter-b5g4j" Sep 16 07:35:10.864856 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.864371 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/998d1cf1-cf6f-405e-807d-eb18e33f2e54-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-pjv4h\" (UID: \"998d1cf1-cf6f-405e-807d-eb18e33f2e54\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pjv4h" Sep 16 07:35:10.864856 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.864394 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c41268ea-998a-4cb5-8763-9780490c6432-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b5g4j\" (UID: \"c41268ea-998a-4cb5-8763-9780490c6432\") " pod="openshift-monitoring/node-exporter-b5g4j" Sep 16 07:35:10.864856 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.864417 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c41268ea-998a-4cb5-8763-9780490c6432-metrics-client-ca\") pod \"node-exporter-b5g4j\" (UID: \"c41268ea-998a-4cb5-8763-9780490c6432\") " pod="openshift-monitoring/node-exporter-b5g4j" Sep 16 07:35:10.864856 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.864442 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlh9z\" (UniqueName: \"kubernetes.io/projected/998d1cf1-cf6f-405e-807d-eb18e33f2e54-kube-api-access-hlh9z\") pod \"openshift-state-metrics-67dcf8c979-pjv4h\" (UID: \"998d1cf1-cf6f-405e-807d-eb18e33f2e54\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pjv4h" Sep 16 07:35:10.864856 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.864480 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c41268ea-998a-4cb5-8763-9780490c6432-node-exporter-tls\") pod \"node-exporter-b5g4j\" (UID: \"c41268ea-998a-4cb5-8763-9780490c6432\") " pod="openshift-monitoring/node-exporter-b5g4j" Sep 16 07:35:10.864856 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:35:10.864595 2552 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 16 07:35:10.864856 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.864754 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c41268ea-998a-4cb5-8763-9780490c6432-node-exporter-accelerators-collector-config\") pod \"node-exporter-b5g4j\" (UID: \"c41268ea-998a-4cb5-8763-9780490c6432\") " pod="openshift-monitoring/node-exporter-b5g4j" Sep 16 07:35:10.865242 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.864889 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c41268ea-998a-4cb5-8763-9780490c6432-sys\") pod \"node-exporter-b5g4j\" (UID: \"c41268ea-998a-4cb5-8763-9780490c6432\") " pod="openshift-monitoring/node-exporter-b5g4j" Sep 16 07:35:10.865242 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.864909 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c41268ea-998a-4cb5-8763-9780490c6432-node-exporter-wtmp\") pod \"node-exporter-b5g4j\" (UID: \"c41268ea-998a-4cb5-8763-9780490c6432\") " pod="openshift-monitoring/node-exporter-b5g4j" Sep 16 07:35:10.865242 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.864953 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c41268ea-998a-4cb5-8763-9780490c6432-root\") pod \"node-exporter-b5g4j\" (UID: \"c41268ea-998a-4cb5-8763-9780490c6432\") " pod="openshift-monitoring/node-exporter-b5g4j" Sep 16 07:35:10.865242 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.865224 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c41268ea-998a-4cb5-8763-9780490c6432-node-exporter-textfile\") pod \"node-exporter-b5g4j\" (UID: \"c41268ea-998a-4cb5-8763-9780490c6432\") " pod="openshift-monitoring/node-exporter-b5g4j" Sep 16 07:35:10.865553 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.865530 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/998d1cf1-cf6f-405e-807d-eb18e33f2e54-metrics-client-ca\") pod \"openshift-state-metrics-67dcf8c979-pjv4h\" (UID: \"998d1cf1-cf6f-405e-807d-eb18e33f2e54\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pjv4h" Sep 16 07:35:10.865696 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:35:10.865605 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c41268ea-998a-4cb5-8763-9780490c6432-node-exporter-tls podName:c41268ea-998a-4cb5-8763-9780490c6432 nodeName:}" failed. No retries permitted until 2026-09-16 07:35:11.364648717 +0000 UTC m=+82.948427047 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/c41268ea-998a-4cb5-8763-9780490c6432-node-exporter-tls") pod "node-exporter-b5g4j" (UID: "c41268ea-998a-4cb5-8763-9780490c6432") : secret "node-exporter-tls" not found Sep 16 07:35:10.865696 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.865632 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c41268ea-998a-4cb5-8763-9780490c6432-metrics-client-ca\") pod \"node-exporter-b5g4j\" (UID: \"c41268ea-998a-4cb5-8763-9780490c6432\") " pod="openshift-monitoring/node-exporter-b5g4j" Sep 16 07:35:10.868721 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.868694 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/998d1cf1-cf6f-405e-807d-eb18e33f2e54-openshift-state-metrics-tls\") pod \"openshift-state-metrics-67dcf8c979-pjv4h\" (UID: \"998d1cf1-cf6f-405e-807d-eb18e33f2e54\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pjv4h" Sep 16 07:35:10.869710 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.869156 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c41268ea-998a-4cb5-8763-9780490c6432-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-b5g4j\" (UID: \"c41268ea-998a-4cb5-8763-9780490c6432\") " pod="openshift-monitoring/node-exporter-b5g4j" Sep 16 07:35:10.869972 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.869942 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/998d1cf1-cf6f-405e-807d-eb18e33f2e54-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-67dcf8c979-pjv4h\" (UID: \"998d1cf1-cf6f-405e-807d-eb18e33f2e54\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pjv4h" Sep 16 07:35:10.877564 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.877534 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc7cr\" (UniqueName: \"kubernetes.io/projected/c41268ea-998a-4cb5-8763-9780490c6432-kube-api-access-rc7cr\") pod \"node-exporter-b5g4j\" (UID: \"c41268ea-998a-4cb5-8763-9780490c6432\") " pod="openshift-monitoring/node-exporter-b5g4j" Sep 16 07:35:10.877728 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.877611 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlh9z\" (UniqueName: \"kubernetes.io/projected/998d1cf1-cf6f-405e-807d-eb18e33f2e54-kube-api-access-hlh9z\") pod \"openshift-state-metrics-67dcf8c979-pjv4h\" (UID: \"998d1cf1-cf6f-405e-807d-eb18e33f2e54\") " pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pjv4h" Sep 16 07:35:10.899218 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.899178 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:10.959238 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:10.959200 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pjv4h" Sep 16 07:35:11.005799 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:11.005646 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:11.179512 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:11.179474 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:11.369090 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:11.369012 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c41268ea-998a-4cb5-8763-9780490c6432-node-exporter-tls\") pod \"node-exporter-b5g4j\" (UID: \"c41268ea-998a-4cb5-8763-9780490c6432\") " pod="openshift-monitoring/node-exporter-b5g4j" Sep 16 07:35:11.372141 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:11.372088 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c41268ea-998a-4cb5-8763-9780490c6432-node-exporter-tls\") pod \"node-exporter-b5g4j\" (UID: \"c41268ea-998a-4cb5-8763-9780490c6432\") " pod="openshift-monitoring/node-exporter-b5g4j" Sep 16 07:35:11.567082 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:11.567042 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-b5g4j" Sep 16 07:35:11.622811 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:11.622719 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:11.703489 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:11.703457 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:11.835291 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:11.835251 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:13.607930 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:13.607878 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:13.708589 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:13.708548 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:13.808420 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:13.808380 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:13.939713 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:13.939671 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:13.998912 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:13.998859 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:14.131080 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:14.131025 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:14.209970 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:14.209848 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:15.414799 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:15.414743 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:15.515784 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:15.515747 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:15.944049 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:15.943995 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:16.920850 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:16.920815 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 16 07:35:16.926002 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:16.925973 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:16.940217 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:16.940184 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:16.943492 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:16.943466 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 16 07:35:16.943914 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:16.943889 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 16 07:35:16.944358 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:16.944154 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-ftk58\"" Sep 16 07:35:16.944358 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:16.944242 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 16 07:35:16.944358 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:16.944280 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 16 07:35:16.944584 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:16.944413 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 16 07:35:16.944584 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:16.944434 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 16 07:35:16.946006 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:16.944928 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 16 07:35:16.946006 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:16.945286 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 16 07:35:16.946006 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:16.945521 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 16 07:35:16.946230 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:16.946129 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 16 07:35:16.946628 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:16.946430 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 16 07:35:16.946628 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:16.946455 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 16 07:35:16.946628 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:16.946490 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 16 07:35:16.946628 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:16.946517 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1p9u7c0karkug\"" Sep 16 07:35:16.946628 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:16.946459 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 16 07:35:16.948067 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:16.948032 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 16 07:35:16.948313 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:16.948283 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 16 07:35:17.020918 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.020874 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.021122 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.020930 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9d0d367-0a54-4d53-9a93-1721072b0b89-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.021122 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.020997 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9d0d367-0a54-4d53-9a93-1721072b0b89-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.021122 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.021041 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e9d0d367-0a54-4d53-9a93-1721072b0b89-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.021122 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.021078 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9d0d367-0a54-4d53-9a93-1721072b0b89-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.021122 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.021124 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9d0d367-0a54-4d53-9a93-1721072b0b89-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.021354 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.021140 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ktc7\" (UniqueName: \"kubernetes.io/projected/e9d0d367-0a54-4d53-9a93-1721072b0b89-kube-api-access-6ktc7\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.021354 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.021158 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e9d0d367-0a54-4d53-9a93-1721072b0b89-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.021354 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.021186 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.021354 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.021214 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.021354 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.021240 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e9d0d367-0a54-4d53-9a93-1721072b0b89-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.021354 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.021270 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e9d0d367-0a54-4d53-9a93-1721072b0b89-config-out\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.021354 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.021293 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.021354 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.021352 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-config\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.021661 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.021369 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.021661 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.021391 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e9d0d367-0a54-4d53-9a93-1721072b0b89-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.021661 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.021426 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e9d0d367-0a54-4d53-9a93-1721072b0b89-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.021661 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.021458 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.021661 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.021484 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.021661 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.021511 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-web-config\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.122068 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.122031 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.122299 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.122077 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-web-config\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.122299 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.122129 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.122299 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.122149 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9d0d367-0a54-4d53-9a93-1721072b0b89-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.122299 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.122197 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9d0d367-0a54-4d53-9a93-1721072b0b89-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.122299 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.122221 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e9d0d367-0a54-4d53-9a93-1721072b0b89-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.122299 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.122273 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9d0d367-0a54-4d53-9a93-1721072b0b89-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.122601 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.122316 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9d0d367-0a54-4d53-9a93-1721072b0b89-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.122601 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.122350 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ktc7\" (UniqueName: \"kubernetes.io/projected/e9d0d367-0a54-4d53-9a93-1721072b0b89-kube-api-access-6ktc7\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.122601 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.122373 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e9d0d367-0a54-4d53-9a93-1721072b0b89-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.122601 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.122401 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.122601 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.122422 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.122601 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.122448 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e9d0d367-0a54-4d53-9a93-1721072b0b89-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.122601 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.122474 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e9d0d367-0a54-4d53-9a93-1721072b0b89-config-out\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.122601 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.122499 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.122601 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.122545 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-config\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.122601 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.122576 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.123053 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.122608 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e9d0d367-0a54-4d53-9a93-1721072b0b89-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.123053 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.122649 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e9d0d367-0a54-4d53-9a93-1721072b0b89-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.123053 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.122680 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.124123 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.123803 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e9d0d367-0a54-4d53-9a93-1721072b0b89-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.126613 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.126577 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.127924 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.127865 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.128417 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.128379 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.128570 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.128547 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-config\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.129421 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.129361 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-web-config\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.129421 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.129382 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.130010 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.129973 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e9d0d367-0a54-4d53-9a93-1721072b0b89-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.131858 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.130807 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9d0d367-0a54-4d53-9a93-1721072b0b89-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.131858 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.130976 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9d0d367-0a54-4d53-9a93-1721072b0b89-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.131858 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.131490 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9d0d367-0a54-4d53-9a93-1721072b0b89-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.131858 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.131816 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9d0d367-0a54-4d53-9a93-1721072b0b89-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.132240 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.132216 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e9d0d367-0a54-4d53-9a93-1721072b0b89-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.132972 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.132949 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e9d0d367-0a54-4d53-9a93-1721072b0b89-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.133352 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.133298 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e9d0d367-0a54-4d53-9a93-1721072b0b89-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.133964 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.133918 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.134250 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.134227 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e9d0d367-0a54-4d53-9a93-1721072b0b89-config-out\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.134887 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.134864 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.135706 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.135644 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.141273 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.141249 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ktc7\" (UniqueName: \"kubernetes.io/projected/e9d0d367-0a54-4d53-9a93-1721072b0b89-kube-api-access-6ktc7\") pod \"prometheus-k8s-0\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.257179 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.256994 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:17.559298 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:35:17.559264 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc41268ea_998a_4cb5_8763_9780490c6432.slice/crio-2c21274eb6361a0d6b4fb0e3e98e566c9ae894c8551578d41bc9373c8038ab75 WatchSource:0}: Error finding container 2c21274eb6361a0d6b4fb0e3e98e566c9ae894c8551578d41bc9373c8038ab75: Status 404 returned error can't find the container with id 2c21274eb6361a0d6b4fb0e3e98e566c9ae894c8551578d41bc9373c8038ab75 Sep 16 07:35:17.824889 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.824863 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-67dcf8c979-pjv4h"] Sep 16 07:35:17.843548 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:17.843524 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 16 07:35:17.918865 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:35:17.918827 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod998d1cf1_cf6f_405e_807d_eb18e33f2e54.slice/crio-9b13c7d6c9eaeddaa26126c3013adc1ed576c3c14ae137d5c024008b4843564c WatchSource:0}: Error finding container 9b13c7d6c9eaeddaa26126c3013adc1ed576c3c14ae137d5c024008b4843564c: Status 404 returned error can't find the container with id 9b13c7d6c9eaeddaa26126c3013adc1ed576c3c14ae137d5c024008b4843564c Sep 16 07:35:17.919548 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:35:17.919517 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9d0d367_0a54_4d53_9a93_1721072b0b89.slice/crio-6cd87879d8001fbdeab253303985e507fec04c0b48bdaa4a1f1b93d3f978f769 WatchSource:0}: Error finding container 6cd87879d8001fbdeab253303985e507fec04c0b48bdaa4a1f1b93d3f978f769: Status 404 returned error can't find the container with id 6cd87879d8001fbdeab253303985e507fec04c0b48bdaa4a1f1b93d3f978f769 Sep 16 07:35:18.092891 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:18.092698 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:18.192190 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:18.192151 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:18.252781 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:18.252747 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-9ff4cbfdc-kvcdr" Sep 16 07:35:18.341909 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:18.341826 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b5g4j" event={"ID":"c41268ea-998a-4cb5-8763-9780490c6432","Type":"ContainerStarted","Data":"2c21274eb6361a0d6b4fb0e3e98e566c9ae894c8551578d41bc9373c8038ab75"} Sep 16 07:35:18.343271 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:18.343239 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-wvs7b" event={"ID":"eccfa20f-4a48-4588-a7fa-a82fe43e1ccc","Type":"ContainerStarted","Data":"24d6b3c8fb89af7ea12d15c00dd89df0e77ea5ba3471a3b18e9b1d64e8a887a5"} Sep 16 07:35:18.343435 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:18.343410 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-wvs7b" Sep 16 07:35:18.344677 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:18.344656 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e9d0d367-0a54-4d53-9a93-1721072b0b89","Type":"ContainerStarted","Data":"6cd87879d8001fbdeab253303985e507fec04c0b48bdaa4a1f1b93d3f978f769"} Sep 16 07:35:18.345141 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:18.345094 2552 patch_prober.go:28] interesting pod/downloads-76f99584f4-wvs7b container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.12:8080/\": dial tcp 10.133.0.12:8080: connect: connection refused" start-of-body= Sep 16 07:35:18.345220 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:18.345181 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-76f99584f4-wvs7b" podUID="eccfa20f-4a48-4588-a7fa-a82fe43e1ccc" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.12:8080/\": dial tcp 10.133.0.12:8080: connect: connection refused" Sep 16 07:35:18.346590 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:18.346570 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pjv4h" event={"ID":"998d1cf1-cf6f-405e-807d-eb18e33f2e54","Type":"ContainerStarted","Data":"eccb63eb12150dfdee42439b57191251fe52c64b9639fa63e7540d6f12ebfa9f"} Sep 16 07:35:18.346664 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:18.346615 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pjv4h" event={"ID":"998d1cf1-cf6f-405e-807d-eb18e33f2e54","Type":"ContainerStarted","Data":"952a6b17740f0899a949a48091a86f6ae9b715aeafd51e8502feb854b4002efd"} Sep 16 07:35:18.346664 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:18.346636 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pjv4h" event={"ID":"998d1cf1-cf6f-405e-807d-eb18e33f2e54","Type":"ContainerStarted","Data":"9b13c7d6c9eaeddaa26126c3013adc1ed576c3c14ae137d5c024008b4843564c"} Sep 16 07:35:18.358414 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:18.358364 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-wvs7b" podStartSLOduration=2.021077892 podStartE2EDuration="22.35834687s" totalImagesPullingTime="20.337268978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:34:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:35:18.357688876 +0000 UTC m=+89.941467214" watchObservedRunningTime="2026-09-16 07:35:18.35834687 +0000 UTC m=+89.942125207" Sep 16 07:35:19.257090 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:19.256472 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-56676c689d-5tjq7" Sep 16 07:35:19.368178 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:19.368137 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-wvs7b" Sep 16 07:35:19.389204 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:19.389160 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:19.440117 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:19.440065 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:20.074012 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:20.073864 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:20.143531 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:20.143447 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:20.362590 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:20.361831 2552 generic.go:356] "Generic (PLEG): container finished" podID="c41268ea-998a-4cb5-8763-9780490c6432" containerID="68c6323d66bccbf33a18a7937ed6a48ea9152f7be7824fcfdde427c8f1038290" exitCode=0 Sep 16 07:35:20.362964 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:20.362686 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b5g4j" event={"ID":"c41268ea-998a-4cb5-8763-9780490c6432","Type":"ContainerDied","Data":"68c6323d66bccbf33a18a7937ed6a48ea9152f7be7824fcfdde427c8f1038290"} Sep 16 07:35:21.370239 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:21.370207 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b5g4j" event={"ID":"c41268ea-998a-4cb5-8763-9780490c6432","Type":"ContainerStarted","Data":"9156f533b978e5771ce78ade9a6a01efd7debf7989c5c24a6c59333d9fa6aab1"} Sep 16 07:35:21.370693 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:21.370272 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-b5g4j" event={"ID":"c41268ea-998a-4cb5-8763-9780490c6432","Type":"ContainerStarted","Data":"ac118992f269a099abce715a6783e8615d93ade9fc3c9a58f097f2a44d4aeb4a"} Sep 16 07:35:21.372077 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:21.372037 2552 generic.go:356] "Generic (PLEG): container finished" podID="e9d0d367-0a54-4d53-9a93-1721072b0b89" containerID="ee4977c84cfaf35bfed2d2569dadde5c1c599858d172d26e5278c6c01497c290" exitCode=0 Sep 16 07:35:21.372202 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:21.372080 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e9d0d367-0a54-4d53-9a93-1721072b0b89","Type":"ContainerDied","Data":"ee4977c84cfaf35bfed2d2569dadde5c1c599858d172d26e5278c6c01497c290"} Sep 16 07:35:21.375354 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:21.375326 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pjv4h" event={"ID":"998d1cf1-cf6f-405e-807d-eb18e33f2e54","Type":"ContainerStarted","Data":"6c0e14a6cb157111359308c7a76de10195671260c7fe5ee08289e922aa2d8c43"} Sep 16 07:35:21.389268 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:21.389206 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-b5g4j" podStartSLOduration=9.254325131 podStartE2EDuration="11.389188916s" totalImagesPullingTime="2.038244936s" totalInitContainerRuntime="96.618849ms" isStatefulPod=true podCreationTimestamp="2026-09-16 07:35:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:35:21.387593117 +0000 UTC m=+92.971371456" watchObservedRunningTime="2026-09-16 07:35:21.389188916 +0000 UTC m=+92.972967257" Sep 16 07:35:21.403426 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:21.403368 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-67dcf8c979-pjv4h" podStartSLOduration=8.879467705 podStartE2EDuration="11.40334833s" totalImagesPullingTime="2.523880625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:35:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:35:21.402626235 +0000 UTC m=+92.986404574" watchObservedRunningTime="2026-09-16 07:35:21.40334833 +0000 UTC m=+92.987126699" Sep 16 07:35:23.039519 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:23.039431 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:23.145665 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:23.145511 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:23.240058 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:23.240007 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:23.267682 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:23.267643 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-9ff4cbfdc-kvcdr" podUID="50be9bc4-8115-41e1-afb4-51a305538a5a" containerName="registry" containerID="cri-o://7b06377f68ebd4e761782c675843266b3711ed1579f8a9324df57f930da073c3" gracePeriod=30 Sep 16 07:35:23.388937 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:23.388896 2552 generic.go:356] "Generic (PLEG): container finished" podID="50be9bc4-8115-41e1-afb4-51a305538a5a" containerID="7b06377f68ebd4e761782c675843266b3711ed1579f8a9324df57f930da073c3" exitCode=0 Sep 16 07:35:23.389315 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:23.389007 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9ff4cbfdc-kvcdr" event={"ID":"50be9bc4-8115-41e1-afb4-51a305538a5a","Type":"ContainerDied","Data":"7b06377f68ebd4e761782c675843266b3711ed1579f8a9324df57f930da073c3"} Sep 16 07:35:23.582766 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:23.582727 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9ff4cbfdc-kvcdr" Sep 16 07:35:23.693127 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:23.692156 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/50be9bc4-8115-41e1-afb4-51a305538a5a-registry-tls\") pod \"50be9bc4-8115-41e1-afb4-51a305538a5a\" (UID: \"50be9bc4-8115-41e1-afb4-51a305538a5a\") " Sep 16 07:35:23.693127 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:23.692248 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/50be9bc4-8115-41e1-afb4-51a305538a5a-image-registry-private-configuration\") pod \"50be9bc4-8115-41e1-afb4-51a305538a5a\" (UID: \"50be9bc4-8115-41e1-afb4-51a305538a5a\") " Sep 16 07:35:23.693127 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:23.692287 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/50be9bc4-8115-41e1-afb4-51a305538a5a-ca-trust-extracted\") pod \"50be9bc4-8115-41e1-afb4-51a305538a5a\" (UID: \"50be9bc4-8115-41e1-afb4-51a305538a5a\") " Sep 16 07:35:23.693127 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:23.692330 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/50be9bc4-8115-41e1-afb4-51a305538a5a-installation-pull-secrets\") pod \"50be9bc4-8115-41e1-afb4-51a305538a5a\" (UID: \"50be9bc4-8115-41e1-afb4-51a305538a5a\") " Sep 16 07:35:23.693127 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:23.692399 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdztr\" (UniqueName: \"kubernetes.io/projected/50be9bc4-8115-41e1-afb4-51a305538a5a-kube-api-access-qdztr\") pod \"50be9bc4-8115-41e1-afb4-51a305538a5a\" (UID: \"50be9bc4-8115-41e1-afb4-51a305538a5a\") " Sep 16 07:35:23.693127 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:23.692432 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/50be9bc4-8115-41e1-afb4-51a305538a5a-trusted-ca\") pod \"50be9bc4-8115-41e1-afb4-51a305538a5a\" (UID: \"50be9bc4-8115-41e1-afb4-51a305538a5a\") " Sep 16 07:35:23.693127 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:23.692453 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/50be9bc4-8115-41e1-afb4-51a305538a5a-bound-sa-token\") pod \"50be9bc4-8115-41e1-afb4-51a305538a5a\" (UID: \"50be9bc4-8115-41e1-afb4-51a305538a5a\") " Sep 16 07:35:23.693127 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:23.692486 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/50be9bc4-8115-41e1-afb4-51a305538a5a-registry-certificates\") pod \"50be9bc4-8115-41e1-afb4-51a305538a5a\" (UID: \"50be9bc4-8115-41e1-afb4-51a305538a5a\") " Sep 16 07:35:23.693663 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:23.693433 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50be9bc4-8115-41e1-afb4-51a305538a5a-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "50be9bc4-8115-41e1-afb4-51a305538a5a" (UID: "50be9bc4-8115-41e1-afb4-51a305538a5a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:35:23.693875 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:23.693845 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50be9bc4-8115-41e1-afb4-51a305538a5a-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "50be9bc4-8115-41e1-afb4-51a305538a5a" (UID: "50be9bc4-8115-41e1-afb4-51a305538a5a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:35:23.697336 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:23.696674 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50be9bc4-8115-41e1-afb4-51a305538a5a-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "50be9bc4-8115-41e1-afb4-51a305538a5a" (UID: "50be9bc4-8115-41e1-afb4-51a305538a5a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:35:23.697336 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:23.696962 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50be9bc4-8115-41e1-afb4-51a305538a5a-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "50be9bc4-8115-41e1-afb4-51a305538a5a" (UID: "50be9bc4-8115-41e1-afb4-51a305538a5a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:35:23.697336 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:23.697234 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50be9bc4-8115-41e1-afb4-51a305538a5a-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "50be9bc4-8115-41e1-afb4-51a305538a5a" (UID: "50be9bc4-8115-41e1-afb4-51a305538a5a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:35:23.697785 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:23.697697 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50be9bc4-8115-41e1-afb4-51a305538a5a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "50be9bc4-8115-41e1-afb4-51a305538a5a" (UID: "50be9bc4-8115-41e1-afb4-51a305538a5a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:35:23.698594 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:23.698472 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50be9bc4-8115-41e1-afb4-51a305538a5a-kube-api-access-qdztr" (OuterVolumeSpecName: "kube-api-access-qdztr") pod "50be9bc4-8115-41e1-afb4-51a305538a5a" (UID: "50be9bc4-8115-41e1-afb4-51a305538a5a"). InnerVolumeSpecName "kube-api-access-qdztr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:35:23.705256 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:23.705225 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50be9bc4-8115-41e1-afb4-51a305538a5a-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "50be9bc4-8115-41e1-afb4-51a305538a5a" (UID: "50be9bc4-8115-41e1-afb4-51a305538a5a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:35:23.793676 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:23.793632 2552 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/50be9bc4-8115-41e1-afb4-51a305538a5a-installation-pull-secrets\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:35:23.793676 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:23.793669 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdztr\" (UniqueName: \"kubernetes.io/projected/50be9bc4-8115-41e1-afb4-51a305538a5a-kube-api-access-qdztr\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:35:23.793676 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:23.793684 2552 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/50be9bc4-8115-41e1-afb4-51a305538a5a-trusted-ca\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:35:23.793971 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:23.793696 2552 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/50be9bc4-8115-41e1-afb4-51a305538a5a-bound-sa-token\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:35:23.793971 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:23.793708 2552 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/50be9bc4-8115-41e1-afb4-51a305538a5a-registry-certificates\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:35:23.793971 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:23.793720 2552 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/50be9bc4-8115-41e1-afb4-51a305538a5a-registry-tls\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:35:23.793971 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:23.793733 2552 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/50be9bc4-8115-41e1-afb4-51a305538a5a-image-registry-private-configuration\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:35:23.793971 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:23.793746 2552 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/50be9bc4-8115-41e1-afb4-51a305538a5a-ca-trust-extracted\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:35:24.399210 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:24.399172 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-9ff4cbfdc-kvcdr" event={"ID":"50be9bc4-8115-41e1-afb4-51a305538a5a","Type":"ContainerDied","Data":"778f4d7d37bfd475ecf93af37c3fbf7d1f5182c1af95052a41677ab713012ac6"} Sep 16 07:35:24.399757 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:24.399240 2552 scope.go:117] "RemoveContainer" containerID="7b06377f68ebd4e761782c675843266b3711ed1579f8a9324df57f930da073c3" Sep 16 07:35:24.399757 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:24.399299 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-9ff4cbfdc-kvcdr" Sep 16 07:35:24.426220 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:24.426177 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-9ff4cbfdc-kvcdr"] Sep 16 07:35:24.437143 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:24.437053 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-9ff4cbfdc-kvcdr"] Sep 16 07:35:24.939340 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:24.938967 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50be9bc4-8115-41e1-afb4-51a305538a5a" path="/var/lib/kubelet/pods/50be9bc4-8115-41e1-afb4-51a305538a5a/volumes" Sep 16 07:35:25.187873 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:25.187829 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:25.242539 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:25.242402 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:26.128634 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:26.128549 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:26.228388 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:26.228332 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:26.412544 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:26.412453 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e9d0d367-0a54-4d53-9a93-1721072b0b89","Type":"ContainerStarted","Data":"af3184c588433b88f5583bb186c02e5c5142ddff86460c3b83822a1a8a33ccdf"} Sep 16 07:35:26.412751 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:26.412545 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e9d0d367-0a54-4d53-9a93-1721072b0b89","Type":"ContainerStarted","Data":"d0ca0fa64eecb52c0bb358e0aa1c0a9283cfe1d52a301ff02a38ec827f96117f"} Sep 16 07:35:29.344611 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:29.344574 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:29.428593 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:29.428558 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e9d0d367-0a54-4d53-9a93-1721072b0b89","Type":"ContainerStarted","Data":"1485fcfa9b29747ab9bd61c51731fb276fa438d606eaddd06fbf5748d2b79f5f"} Sep 16 07:35:29.428762 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:29.428653 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e9d0d367-0a54-4d53-9a93-1721072b0b89","Type":"ContainerStarted","Data":"06b82993f95b5ea4df0dcb1a0d42d739a63f2ca530572b38af228910dd4ffdd1"} Sep 16 07:35:29.428762 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:29.428680 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e9d0d367-0a54-4d53-9a93-1721072b0b89","Type":"ContainerStarted","Data":"8c9986675788ef785b7af85b8926f210f1ad0f7b0cbc8643504cca48d7f3e579"} Sep 16 07:35:29.428762 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:29.428707 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e9d0d367-0a54-4d53-9a93-1721072b0b89","Type":"ContainerStarted","Data":"ffbaad20a66e7eb73abf245c7723a9676a14bae01819e0e116e2041933ff1362"} Sep 16 07:35:29.442449 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:29.442388 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:29.455094 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:29.455035 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.402571637 podStartE2EDuration="13.455016178s" totalImagesPullingTime="9.025560109s" totalInitContainerRuntime="26.884432ms" isStatefulPod=false podCreationTimestamp="2026-09-16 07:35:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:35:29.452860881 +0000 UTC m=+101.036639218" watchObservedRunningTime="2026-09-16 07:35:29.455016178 +0000 UTC m=+101.038794517" Sep 16 07:35:30.304440 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:30.304400 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:30.405012 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:30.404977 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:31.158570 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:31.158535 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:32.105116 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:32.105070 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:32.208588 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:32.208548 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:32.257362 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:32.257333 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:35:32.270890 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:32.270861 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vx6vh" Sep 16 07:35:34.713917 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:34.713875 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:34.770437 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:34.770399 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:49.710313 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:49.710277 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:49.770082 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:49.770043 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:53.794784 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:53.794716 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:53.896032 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:53.896003 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:53.947819 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:53.947780 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:35:56.843112 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:35:56.843067 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:36:17.257791 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:17.257746 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:17.277790 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:17.277762 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:17.625950 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:17.625861 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:29.921559 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:29.921514 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:36:30.317621 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:30.317522 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:36:35.304051 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.304010 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 16 07:36:35.305046 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.305012 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e9d0d367-0a54-4d53-9a93-1721072b0b89" containerName="prometheus" containerID="cri-o://d0ca0fa64eecb52c0bb358e0aa1c0a9283cfe1d52a301ff02a38ec827f96117f" gracePeriod=600 Sep 16 07:36:35.305515 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.305476 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e9d0d367-0a54-4d53-9a93-1721072b0b89" containerName="config-reloader" containerID="cri-o://af3184c588433b88f5583bb186c02e5c5142ddff86460c3b83822a1a8a33ccdf" gracePeriod=600 Sep 16 07:36:35.305741 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.305406 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e9d0d367-0a54-4d53-9a93-1721072b0b89" containerName="kube-rbac-proxy-thanos" containerID="cri-o://1485fcfa9b29747ab9bd61c51731fb276fa438d606eaddd06fbf5748d2b79f5f" gracePeriod=600 Sep 16 07:36:35.305828 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.305425 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e9d0d367-0a54-4d53-9a93-1721072b0b89" containerName="thanos-sidecar" containerID="cri-o://ffbaad20a66e7eb73abf245c7723a9676a14bae01819e0e116e2041933ff1362" gracePeriod=600 Sep 16 07:36:35.305880 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.305442 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e9d0d367-0a54-4d53-9a93-1721072b0b89" containerName="kube-rbac-proxy-web" containerID="cri-o://8c9986675788ef785b7af85b8926f210f1ad0f7b0cbc8643504cca48d7f3e579" gracePeriod=600 Sep 16 07:36:35.305930 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.305232 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e9d0d367-0a54-4d53-9a93-1721072b0b89" containerName="kube-rbac-proxy" containerID="cri-o://06b82993f95b5ea4df0dcb1a0d42d739a63f2ca530572b38af228910dd4ffdd1" gracePeriod=600 Sep 16 07:36:35.555048 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.554973 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.575808 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.575766 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9d0d367-0a54-4d53-9a93-1721072b0b89-prometheus-trusted-ca-bundle\") pod \"e9d0d367-0a54-4d53-9a93-1721072b0b89\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " Sep 16 07:36:35.575996 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.575876 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-secret-metrics-client-certs\") pod \"e9d0d367-0a54-4d53-9a93-1721072b0b89\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " Sep 16 07:36:35.575996 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.575911 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-secret-kube-rbac-proxy\") pod \"e9d0d367-0a54-4d53-9a93-1721072b0b89\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " Sep 16 07:36:35.575996 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.575931 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9d0d367-0a54-4d53-9a93-1721072b0b89-configmap-kubelet-serving-ca-bundle\") pod \"e9d0d367-0a54-4d53-9a93-1721072b0b89\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " Sep 16 07:36:35.575996 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.575970 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"e9d0d367-0a54-4d53-9a93-1721072b0b89\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " Sep 16 07:36:35.575996 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.575994 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e9d0d367-0a54-4d53-9a93-1721072b0b89-config-out\") pod \"e9d0d367-0a54-4d53-9a93-1721072b0b89\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " Sep 16 07:36:35.576288 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.576011 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-config\") pod \"e9d0d367-0a54-4d53-9a93-1721072b0b89\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " Sep 16 07:36:35.576288 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.576048 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9d0d367-0a54-4d53-9a93-1721072b0b89-configmap-serving-certs-ca-bundle\") pod \"e9d0d367-0a54-4d53-9a93-1721072b0b89\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " Sep 16 07:36:35.576288 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.576107 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-thanos-prometheus-http-client-file\") pod \"e9d0d367-0a54-4d53-9a93-1721072b0b89\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " Sep 16 07:36:35.576288 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.576139 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-secret-grpc-tls\") pod \"e9d0d367-0a54-4d53-9a93-1721072b0b89\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " Sep 16 07:36:35.576288 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.576167 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"e9d0d367-0a54-4d53-9a93-1721072b0b89\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " Sep 16 07:36:35.576288 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.576191 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-secret-prometheus-k8s-tls\") pod \"e9d0d367-0a54-4d53-9a93-1721072b0b89\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " Sep 16 07:36:35.576288 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.576225 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e9d0d367-0a54-4d53-9a93-1721072b0b89-tls-assets\") pod \"e9d0d367-0a54-4d53-9a93-1721072b0b89\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " Sep 16 07:36:35.576288 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.576249 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e9d0d367-0a54-4d53-9a93-1721072b0b89-prometheus-k8s-rulefiles-2\") pod \"e9d0d367-0a54-4d53-9a93-1721072b0b89\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " Sep 16 07:36:35.576288 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.576282 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6ktc7\" (UniqueName: \"kubernetes.io/projected/e9d0d367-0a54-4d53-9a93-1721072b0b89-kube-api-access-6ktc7\") pod \"e9d0d367-0a54-4d53-9a93-1721072b0b89\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " Sep 16 07:36:35.576730 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.576313 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9d0d367-0a54-4d53-9a93-1721072b0b89-configmap-metrics-client-ca\") pod \"e9d0d367-0a54-4d53-9a93-1721072b0b89\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " Sep 16 07:36:35.576730 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.576342 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e9d0d367-0a54-4d53-9a93-1721072b0b89-prometheus-k8s-rulefiles-1\") pod \"e9d0d367-0a54-4d53-9a93-1721072b0b89\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " Sep 16 07:36:35.576730 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.576379 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-web-config\") pod \"e9d0d367-0a54-4d53-9a93-1721072b0b89\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " Sep 16 07:36:35.576730 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.576411 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e9d0d367-0a54-4d53-9a93-1721072b0b89-prometheus-k8s-db\") pod \"e9d0d367-0a54-4d53-9a93-1721072b0b89\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " Sep 16 07:36:35.576730 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.576219 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9d0d367-0a54-4d53-9a93-1721072b0b89-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "e9d0d367-0a54-4d53-9a93-1721072b0b89" (UID: "e9d0d367-0a54-4d53-9a93-1721072b0b89"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:36:35.576730 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.576416 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9d0d367-0a54-4d53-9a93-1721072b0b89-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "e9d0d367-0a54-4d53-9a93-1721072b0b89" (UID: "e9d0d367-0a54-4d53-9a93-1721072b0b89"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:36:35.576730 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.576632 2552 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9d0d367-0a54-4d53-9a93-1721072b0b89-prometheus-trusted-ca-bundle\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:36:35.576730 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.576651 2552 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9d0d367-0a54-4d53-9a93-1721072b0b89-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:36:35.577744 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.577485 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9d0d367-0a54-4d53-9a93-1721072b0b89-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "e9d0d367-0a54-4d53-9a93-1721072b0b89" (UID: "e9d0d367-0a54-4d53-9a93-1721072b0b89"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:36:35.577744 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.577514 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9d0d367-0a54-4d53-9a93-1721072b0b89-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "e9d0d367-0a54-4d53-9a93-1721072b0b89" (UID: "e9d0d367-0a54-4d53-9a93-1721072b0b89"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:36:35.577744 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.577604 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9d0d367-0a54-4d53-9a93-1721072b0b89-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "e9d0d367-0a54-4d53-9a93-1721072b0b89" (UID: "e9d0d367-0a54-4d53-9a93-1721072b0b89"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:36:35.579000 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.578949 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9d0d367-0a54-4d53-9a93-1721072b0b89-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "e9d0d367-0a54-4d53-9a93-1721072b0b89" (UID: "e9d0d367-0a54-4d53-9a93-1721072b0b89"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:36:35.579135 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.579074 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9d0d367-0a54-4d53-9a93-1721072b0b89-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "e9d0d367-0a54-4d53-9a93-1721072b0b89" (UID: "e9d0d367-0a54-4d53-9a93-1721072b0b89"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:36:35.579206 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.579180 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "e9d0d367-0a54-4d53-9a93-1721072b0b89" (UID: "e9d0d367-0a54-4d53-9a93-1721072b0b89"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:36:35.580036 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.579796 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "e9d0d367-0a54-4d53-9a93-1721072b0b89" (UID: "e9d0d367-0a54-4d53-9a93-1721072b0b89"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:36:35.580036 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.579899 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "e9d0d367-0a54-4d53-9a93-1721072b0b89" (UID: "e9d0d367-0a54-4d53-9a93-1721072b0b89"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:36:35.580789 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.580606 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9d0d367-0a54-4d53-9a93-1721072b0b89-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "e9d0d367-0a54-4d53-9a93-1721072b0b89" (UID: "e9d0d367-0a54-4d53-9a93-1721072b0b89"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:36:35.581191 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.581066 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9d0d367-0a54-4d53-9a93-1721072b0b89-kube-api-access-6ktc7" (OuterVolumeSpecName: "kube-api-access-6ktc7") pod "e9d0d367-0a54-4d53-9a93-1721072b0b89" (UID: "e9d0d367-0a54-4d53-9a93-1721072b0b89"). InnerVolumeSpecName "kube-api-access-6ktc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:36:35.581474 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.581439 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "e9d0d367-0a54-4d53-9a93-1721072b0b89" (UID: "e9d0d367-0a54-4d53-9a93-1721072b0b89"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:36:35.582173 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.582132 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "e9d0d367-0a54-4d53-9a93-1721072b0b89" (UID: "e9d0d367-0a54-4d53-9a93-1721072b0b89"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:36:35.582992 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.582930 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "e9d0d367-0a54-4d53-9a93-1721072b0b89" (UID: "e9d0d367-0a54-4d53-9a93-1721072b0b89"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:36:35.582992 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.582954 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9d0d367-0a54-4d53-9a93-1721072b0b89-config-out" (OuterVolumeSpecName: "config-out") pod "e9d0d367-0a54-4d53-9a93-1721072b0b89" (UID: "e9d0d367-0a54-4d53-9a93-1721072b0b89"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:36:35.583779 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.583645 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-config" (OuterVolumeSpecName: "config") pod "e9d0d367-0a54-4d53-9a93-1721072b0b89" (UID: "e9d0d367-0a54-4d53-9a93-1721072b0b89"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:36:35.583779 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.583723 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "e9d0d367-0a54-4d53-9a93-1721072b0b89" (UID: "e9d0d367-0a54-4d53-9a93-1721072b0b89"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:36:35.599088 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.598935 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-web-config" (OuterVolumeSpecName: "web-config") pod "e9d0d367-0a54-4d53-9a93-1721072b0b89" (UID: "e9d0d367-0a54-4d53-9a93-1721072b0b89"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:36:35.677335 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.677300 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e9d0d367-0a54-4d53-9a93-1721072b0b89-prometheus-k8s-rulefiles-0\") pod \"e9d0d367-0a54-4d53-9a93-1721072b0b89\" (UID: \"e9d0d367-0a54-4d53-9a93-1721072b0b89\") " Sep 16 07:36:35.677529 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.677483 2552 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e9d0d367-0a54-4d53-9a93-1721072b0b89-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:36:35.677529 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.677502 2552 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-thanos-prometheus-http-client-file\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:36:35.677529 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.677517 2552 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-secret-grpc-tls\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:36:35.677888 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.677531 2552 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:36:35.677888 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.677544 2552 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-secret-prometheus-k8s-tls\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:36:35.677888 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.677557 2552 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e9d0d367-0a54-4d53-9a93-1721072b0b89-tls-assets\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:36:35.677888 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.677570 2552 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e9d0d367-0a54-4d53-9a93-1721072b0b89-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:36:35.677888 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.677584 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ktc7\" (UniqueName: \"kubernetes.io/projected/e9d0d367-0a54-4d53-9a93-1721072b0b89-kube-api-access-6ktc7\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:36:35.677888 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.677597 2552 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e9d0d367-0a54-4d53-9a93-1721072b0b89-configmap-metrics-client-ca\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:36:35.677888 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.677613 2552 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e9d0d367-0a54-4d53-9a93-1721072b0b89-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:36:35.677888 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.677626 2552 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-web-config\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:36:35.677888 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.677639 2552 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e9d0d367-0a54-4d53-9a93-1721072b0b89-prometheus-k8s-db\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:36:35.677888 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.677653 2552 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-secret-metrics-client-certs\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:36:35.677888 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.677666 2552 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-secret-kube-rbac-proxy\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:36:35.677888 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.677679 2552 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:36:35.677888 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.677693 2552 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e9d0d367-0a54-4d53-9a93-1721072b0b89-config-out\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:36:35.677888 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.677706 2552 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/e9d0d367-0a54-4d53-9a93-1721072b0b89-config\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:36:35.678553 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.678374 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9d0d367-0a54-4d53-9a93-1721072b0b89-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "e9d0d367-0a54-4d53-9a93-1721072b0b89" (UID: "e9d0d367-0a54-4d53-9a93-1721072b0b89"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:36:35.683791 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.683761 2552 generic.go:356] "Generic (PLEG): container finished" podID="e9d0d367-0a54-4d53-9a93-1721072b0b89" containerID="1485fcfa9b29747ab9bd61c51731fb276fa438d606eaddd06fbf5748d2b79f5f" exitCode=0 Sep 16 07:36:35.683791 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.683784 2552 generic.go:356] "Generic (PLEG): container finished" podID="e9d0d367-0a54-4d53-9a93-1721072b0b89" containerID="06b82993f95b5ea4df0dcb1a0d42d739a63f2ca530572b38af228910dd4ffdd1" exitCode=0 Sep 16 07:36:35.683791 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.683790 2552 generic.go:356] "Generic (PLEG): container finished" podID="e9d0d367-0a54-4d53-9a93-1721072b0b89" containerID="8c9986675788ef785b7af85b8926f210f1ad0f7b0cbc8643504cca48d7f3e579" exitCode=0 Sep 16 07:36:35.683791 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.683796 2552 generic.go:356] "Generic (PLEG): container finished" podID="e9d0d367-0a54-4d53-9a93-1721072b0b89" containerID="ffbaad20a66e7eb73abf245c7723a9676a14bae01819e0e116e2041933ff1362" exitCode=0 Sep 16 07:36:35.684088 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.683801 2552 generic.go:356] "Generic (PLEG): container finished" podID="e9d0d367-0a54-4d53-9a93-1721072b0b89" containerID="af3184c588433b88f5583bb186c02e5c5142ddff86460c3b83822a1a8a33ccdf" exitCode=0 Sep 16 07:36:35.684088 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.683806 2552 generic.go:356] "Generic (PLEG): container finished" podID="e9d0d367-0a54-4d53-9a93-1721072b0b89" containerID="d0ca0fa64eecb52c0bb358e0aa1c0a9283cfe1d52a301ff02a38ec827f96117f" exitCode=0 Sep 16 07:36:35.684088 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.683814 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e9d0d367-0a54-4d53-9a93-1721072b0b89","Type":"ContainerDied","Data":"1485fcfa9b29747ab9bd61c51731fb276fa438d606eaddd06fbf5748d2b79f5f"} Sep 16 07:36:35.684088 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.683852 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.684088 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.683890 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e9d0d367-0a54-4d53-9a93-1721072b0b89","Type":"ContainerDied","Data":"06b82993f95b5ea4df0dcb1a0d42d739a63f2ca530572b38af228910dd4ffdd1"} Sep 16 07:36:35.684088 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.683918 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e9d0d367-0a54-4d53-9a93-1721072b0b89","Type":"ContainerDied","Data":"8c9986675788ef785b7af85b8926f210f1ad0f7b0cbc8643504cca48d7f3e579"} Sep 16 07:36:35.684088 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.683942 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e9d0d367-0a54-4d53-9a93-1721072b0b89","Type":"ContainerDied","Data":"ffbaad20a66e7eb73abf245c7723a9676a14bae01819e0e116e2041933ff1362"} Sep 16 07:36:35.684088 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.683956 2552 scope.go:117] "RemoveContainer" containerID="1485fcfa9b29747ab9bd61c51731fb276fa438d606eaddd06fbf5748d2b79f5f" Sep 16 07:36:35.684088 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.683967 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e9d0d367-0a54-4d53-9a93-1721072b0b89","Type":"ContainerDied","Data":"af3184c588433b88f5583bb186c02e5c5142ddff86460c3b83822a1a8a33ccdf"} Sep 16 07:36:35.684088 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.683993 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e9d0d367-0a54-4d53-9a93-1721072b0b89","Type":"ContainerDied","Data":"d0ca0fa64eecb52c0bb358e0aa1c0a9283cfe1d52a301ff02a38ec827f96117f"} Sep 16 07:36:35.684088 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.684018 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e9d0d367-0a54-4d53-9a93-1721072b0b89","Type":"ContainerDied","Data":"6cd87879d8001fbdeab253303985e507fec04c0b48bdaa4a1f1b93d3f978f769"} Sep 16 07:36:35.693050 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.693026 2552 scope.go:117] "RemoveContainer" containerID="06b82993f95b5ea4df0dcb1a0d42d739a63f2ca530572b38af228910dd4ffdd1" Sep 16 07:36:35.700946 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.700925 2552 scope.go:117] "RemoveContainer" containerID="8c9986675788ef785b7af85b8926f210f1ad0f7b0cbc8643504cca48d7f3e579" Sep 16 07:36:35.707005 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.706982 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 16 07:36:35.708563 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.708542 2552 scope.go:117] "RemoveContainer" containerID="ffbaad20a66e7eb73abf245c7723a9676a14bae01819e0e116e2041933ff1362" Sep 16 07:36:35.713823 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.713802 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 16 07:36:35.715934 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.715913 2552 scope.go:117] "RemoveContainer" containerID="af3184c588433b88f5583bb186c02e5c5142ddff86460c3b83822a1a8a33ccdf" Sep 16 07:36:35.722769 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.722749 2552 scope.go:117] "RemoveContainer" containerID="d0ca0fa64eecb52c0bb358e0aa1c0a9283cfe1d52a301ff02a38ec827f96117f" Sep 16 07:36:35.730115 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.730078 2552 scope.go:117] "RemoveContainer" containerID="ee4977c84cfaf35bfed2d2569dadde5c1c599858d172d26e5278c6c01497c290" Sep 16 07:36:35.737341 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.737317 2552 scope.go:117] "RemoveContainer" containerID="1485fcfa9b29747ab9bd61c51731fb276fa438d606eaddd06fbf5748d2b79f5f" Sep 16 07:36:35.737644 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:36:35.737621 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1485fcfa9b29747ab9bd61c51731fb276fa438d606eaddd06fbf5748d2b79f5f\": container with ID starting with 1485fcfa9b29747ab9bd61c51731fb276fa438d606eaddd06fbf5748d2b79f5f not found: ID does not exist" containerID="1485fcfa9b29747ab9bd61c51731fb276fa438d606eaddd06fbf5748d2b79f5f" Sep 16 07:36:35.737705 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.737654 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1485fcfa9b29747ab9bd61c51731fb276fa438d606eaddd06fbf5748d2b79f5f"} err="failed to get container status \"1485fcfa9b29747ab9bd61c51731fb276fa438d606eaddd06fbf5748d2b79f5f\": rpc error: code = NotFound desc = could not find container \"1485fcfa9b29747ab9bd61c51731fb276fa438d606eaddd06fbf5748d2b79f5f\": container with ID starting with 1485fcfa9b29747ab9bd61c51731fb276fa438d606eaddd06fbf5748d2b79f5f not found: ID does not exist" Sep 16 07:36:35.737705 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.737688 2552 scope.go:117] "RemoveContainer" containerID="06b82993f95b5ea4df0dcb1a0d42d739a63f2ca530572b38af228910dd4ffdd1" Sep 16 07:36:35.737945 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:36:35.737928 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06b82993f95b5ea4df0dcb1a0d42d739a63f2ca530572b38af228910dd4ffdd1\": container with ID starting with 06b82993f95b5ea4df0dcb1a0d42d739a63f2ca530572b38af228910dd4ffdd1 not found: ID does not exist" containerID="06b82993f95b5ea4df0dcb1a0d42d739a63f2ca530572b38af228910dd4ffdd1" Sep 16 07:36:35.737984 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.737952 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06b82993f95b5ea4df0dcb1a0d42d739a63f2ca530572b38af228910dd4ffdd1"} err="failed to get container status \"06b82993f95b5ea4df0dcb1a0d42d739a63f2ca530572b38af228910dd4ffdd1\": rpc error: code = NotFound desc = could not find container \"06b82993f95b5ea4df0dcb1a0d42d739a63f2ca530572b38af228910dd4ffdd1\": container with ID starting with 06b82993f95b5ea4df0dcb1a0d42d739a63f2ca530572b38af228910dd4ffdd1 not found: ID does not exist" Sep 16 07:36:35.737984 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.737967 2552 scope.go:117] "RemoveContainer" containerID="8c9986675788ef785b7af85b8926f210f1ad0f7b0cbc8643504cca48d7f3e579" Sep 16 07:36:35.738251 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:36:35.738231 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c9986675788ef785b7af85b8926f210f1ad0f7b0cbc8643504cca48d7f3e579\": container with ID starting with 8c9986675788ef785b7af85b8926f210f1ad0f7b0cbc8643504cca48d7f3e579 not found: ID does not exist" containerID="8c9986675788ef785b7af85b8926f210f1ad0f7b0cbc8643504cca48d7f3e579" Sep 16 07:36:35.738312 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.738253 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c9986675788ef785b7af85b8926f210f1ad0f7b0cbc8643504cca48d7f3e579"} err="failed to get container status \"8c9986675788ef785b7af85b8926f210f1ad0f7b0cbc8643504cca48d7f3e579\": rpc error: code = NotFound desc = could not find container \"8c9986675788ef785b7af85b8926f210f1ad0f7b0cbc8643504cca48d7f3e579\": container with ID starting with 8c9986675788ef785b7af85b8926f210f1ad0f7b0cbc8643504cca48d7f3e579 not found: ID does not exist" Sep 16 07:36:35.738312 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.738265 2552 scope.go:117] "RemoveContainer" containerID="ffbaad20a66e7eb73abf245c7723a9676a14bae01819e0e116e2041933ff1362" Sep 16 07:36:35.738516 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:36:35.738501 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffbaad20a66e7eb73abf245c7723a9676a14bae01819e0e116e2041933ff1362\": container with ID starting with ffbaad20a66e7eb73abf245c7723a9676a14bae01819e0e116e2041933ff1362 not found: ID does not exist" containerID="ffbaad20a66e7eb73abf245c7723a9676a14bae01819e0e116e2041933ff1362" Sep 16 07:36:35.738551 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.738519 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffbaad20a66e7eb73abf245c7723a9676a14bae01819e0e116e2041933ff1362"} err="failed to get container status \"ffbaad20a66e7eb73abf245c7723a9676a14bae01819e0e116e2041933ff1362\": rpc error: code = NotFound desc = could not find container \"ffbaad20a66e7eb73abf245c7723a9676a14bae01819e0e116e2041933ff1362\": container with ID starting with ffbaad20a66e7eb73abf245c7723a9676a14bae01819e0e116e2041933ff1362 not found: ID does not exist" Sep 16 07:36:35.738551 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.738542 2552 scope.go:117] "RemoveContainer" containerID="af3184c588433b88f5583bb186c02e5c5142ddff86460c3b83822a1a8a33ccdf" Sep 16 07:36:35.738769 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:36:35.738750 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af3184c588433b88f5583bb186c02e5c5142ddff86460c3b83822a1a8a33ccdf\": container with ID starting with af3184c588433b88f5583bb186c02e5c5142ddff86460c3b83822a1a8a33ccdf not found: ID does not exist" containerID="af3184c588433b88f5583bb186c02e5c5142ddff86460c3b83822a1a8a33ccdf" Sep 16 07:36:35.738803 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.738777 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af3184c588433b88f5583bb186c02e5c5142ddff86460c3b83822a1a8a33ccdf"} err="failed to get container status \"af3184c588433b88f5583bb186c02e5c5142ddff86460c3b83822a1a8a33ccdf\": rpc error: code = NotFound desc = could not find container \"af3184c588433b88f5583bb186c02e5c5142ddff86460c3b83822a1a8a33ccdf\": container with ID starting with af3184c588433b88f5583bb186c02e5c5142ddff86460c3b83822a1a8a33ccdf not found: ID does not exist" Sep 16 07:36:35.738803 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.738797 2552 scope.go:117] "RemoveContainer" containerID="d0ca0fa64eecb52c0bb358e0aa1c0a9283cfe1d52a301ff02a38ec827f96117f" Sep 16 07:36:35.739022 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:36:35.739003 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0ca0fa64eecb52c0bb358e0aa1c0a9283cfe1d52a301ff02a38ec827f96117f\": container with ID starting with d0ca0fa64eecb52c0bb358e0aa1c0a9283cfe1d52a301ff02a38ec827f96117f not found: ID does not exist" containerID="d0ca0fa64eecb52c0bb358e0aa1c0a9283cfe1d52a301ff02a38ec827f96117f" Sep 16 07:36:35.739059 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.739026 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0ca0fa64eecb52c0bb358e0aa1c0a9283cfe1d52a301ff02a38ec827f96117f"} err="failed to get container status \"d0ca0fa64eecb52c0bb358e0aa1c0a9283cfe1d52a301ff02a38ec827f96117f\": rpc error: code = NotFound desc = could not find container \"d0ca0fa64eecb52c0bb358e0aa1c0a9283cfe1d52a301ff02a38ec827f96117f\": container with ID starting with d0ca0fa64eecb52c0bb358e0aa1c0a9283cfe1d52a301ff02a38ec827f96117f not found: ID does not exist" Sep 16 07:36:35.739059 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.739038 2552 scope.go:117] "RemoveContainer" containerID="ee4977c84cfaf35bfed2d2569dadde5c1c599858d172d26e5278c6c01497c290" Sep 16 07:36:35.739271 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:36:35.739257 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee4977c84cfaf35bfed2d2569dadde5c1c599858d172d26e5278c6c01497c290\": container with ID starting with ee4977c84cfaf35bfed2d2569dadde5c1c599858d172d26e5278c6c01497c290 not found: ID does not exist" containerID="ee4977c84cfaf35bfed2d2569dadde5c1c599858d172d26e5278c6c01497c290" Sep 16 07:36:35.739306 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.739274 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee4977c84cfaf35bfed2d2569dadde5c1c599858d172d26e5278c6c01497c290"} err="failed to get container status \"ee4977c84cfaf35bfed2d2569dadde5c1c599858d172d26e5278c6c01497c290\": rpc error: code = NotFound desc = could not find container \"ee4977c84cfaf35bfed2d2569dadde5c1c599858d172d26e5278c6c01497c290\": container with ID starting with ee4977c84cfaf35bfed2d2569dadde5c1c599858d172d26e5278c6c01497c290 not found: ID does not exist" Sep 16 07:36:35.739306 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.739285 2552 scope.go:117] "RemoveContainer" containerID="1485fcfa9b29747ab9bd61c51731fb276fa438d606eaddd06fbf5748d2b79f5f" Sep 16 07:36:35.739503 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.739488 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1485fcfa9b29747ab9bd61c51731fb276fa438d606eaddd06fbf5748d2b79f5f"} err="failed to get container status \"1485fcfa9b29747ab9bd61c51731fb276fa438d606eaddd06fbf5748d2b79f5f\": rpc error: code = NotFound desc = could not find container \"1485fcfa9b29747ab9bd61c51731fb276fa438d606eaddd06fbf5748d2b79f5f\": container with ID starting with 1485fcfa9b29747ab9bd61c51731fb276fa438d606eaddd06fbf5748d2b79f5f not found: ID does not exist" Sep 16 07:36:35.739538 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.739504 2552 scope.go:117] "RemoveContainer" containerID="06b82993f95b5ea4df0dcb1a0d42d739a63f2ca530572b38af228910dd4ffdd1" Sep 16 07:36:35.739751 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.739734 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06b82993f95b5ea4df0dcb1a0d42d739a63f2ca530572b38af228910dd4ffdd1"} err="failed to get container status \"06b82993f95b5ea4df0dcb1a0d42d739a63f2ca530572b38af228910dd4ffdd1\": rpc error: code = NotFound desc = could not find container \"06b82993f95b5ea4df0dcb1a0d42d739a63f2ca530572b38af228910dd4ffdd1\": container with ID starting with 06b82993f95b5ea4df0dcb1a0d42d739a63f2ca530572b38af228910dd4ffdd1 not found: ID does not exist" Sep 16 07:36:35.739789 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.739752 2552 scope.go:117] "RemoveContainer" containerID="8c9986675788ef785b7af85b8926f210f1ad0f7b0cbc8643504cca48d7f3e579" Sep 16 07:36:35.739968 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.739951 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c9986675788ef785b7af85b8926f210f1ad0f7b0cbc8643504cca48d7f3e579"} err="failed to get container status \"8c9986675788ef785b7af85b8926f210f1ad0f7b0cbc8643504cca48d7f3e579\": rpc error: code = NotFound desc = could not find container \"8c9986675788ef785b7af85b8926f210f1ad0f7b0cbc8643504cca48d7f3e579\": container with ID starting with 8c9986675788ef785b7af85b8926f210f1ad0f7b0cbc8643504cca48d7f3e579 not found: ID does not exist" Sep 16 07:36:35.740010 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.739968 2552 scope.go:117] "RemoveContainer" containerID="ffbaad20a66e7eb73abf245c7723a9676a14bae01819e0e116e2041933ff1362" Sep 16 07:36:35.740192 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.740173 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffbaad20a66e7eb73abf245c7723a9676a14bae01819e0e116e2041933ff1362"} err="failed to get container status \"ffbaad20a66e7eb73abf245c7723a9676a14bae01819e0e116e2041933ff1362\": rpc error: code = NotFound desc = could not find container \"ffbaad20a66e7eb73abf245c7723a9676a14bae01819e0e116e2041933ff1362\": container with ID starting with ffbaad20a66e7eb73abf245c7723a9676a14bae01819e0e116e2041933ff1362 not found: ID does not exist" Sep 16 07:36:35.740239 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.740195 2552 scope.go:117] "RemoveContainer" containerID="af3184c588433b88f5583bb186c02e5c5142ddff86460c3b83822a1a8a33ccdf" Sep 16 07:36:35.740416 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.740401 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af3184c588433b88f5583bb186c02e5c5142ddff86460c3b83822a1a8a33ccdf"} err="failed to get container status \"af3184c588433b88f5583bb186c02e5c5142ddff86460c3b83822a1a8a33ccdf\": rpc error: code = NotFound desc = could not find container \"af3184c588433b88f5583bb186c02e5c5142ddff86460c3b83822a1a8a33ccdf\": container with ID starting with af3184c588433b88f5583bb186c02e5c5142ddff86460c3b83822a1a8a33ccdf not found: ID does not exist" Sep 16 07:36:35.740459 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.740417 2552 scope.go:117] "RemoveContainer" containerID="d0ca0fa64eecb52c0bb358e0aa1c0a9283cfe1d52a301ff02a38ec827f96117f" Sep 16 07:36:35.740635 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.740620 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0ca0fa64eecb52c0bb358e0aa1c0a9283cfe1d52a301ff02a38ec827f96117f"} err="failed to get container status \"d0ca0fa64eecb52c0bb358e0aa1c0a9283cfe1d52a301ff02a38ec827f96117f\": rpc error: code = NotFound desc = could not find container \"d0ca0fa64eecb52c0bb358e0aa1c0a9283cfe1d52a301ff02a38ec827f96117f\": container with ID starting with d0ca0fa64eecb52c0bb358e0aa1c0a9283cfe1d52a301ff02a38ec827f96117f not found: ID does not exist" Sep 16 07:36:35.740635 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.740634 2552 scope.go:117] "RemoveContainer" containerID="ee4977c84cfaf35bfed2d2569dadde5c1c599858d172d26e5278c6c01497c290" Sep 16 07:36:35.740817 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.740802 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee4977c84cfaf35bfed2d2569dadde5c1c599858d172d26e5278c6c01497c290"} err="failed to get container status \"ee4977c84cfaf35bfed2d2569dadde5c1c599858d172d26e5278c6c01497c290\": rpc error: code = NotFound desc = could not find container \"ee4977c84cfaf35bfed2d2569dadde5c1c599858d172d26e5278c6c01497c290\": container with ID starting with ee4977c84cfaf35bfed2d2569dadde5c1c599858d172d26e5278c6c01497c290 not found: ID does not exist" Sep 16 07:36:35.740858 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.740816 2552 scope.go:117] "RemoveContainer" containerID="1485fcfa9b29747ab9bd61c51731fb276fa438d606eaddd06fbf5748d2b79f5f" Sep 16 07:36:35.741043 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.741023 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1485fcfa9b29747ab9bd61c51731fb276fa438d606eaddd06fbf5748d2b79f5f"} err="failed to get container status \"1485fcfa9b29747ab9bd61c51731fb276fa438d606eaddd06fbf5748d2b79f5f\": rpc error: code = NotFound desc = could not find container \"1485fcfa9b29747ab9bd61c51731fb276fa438d606eaddd06fbf5748d2b79f5f\": container with ID starting with 1485fcfa9b29747ab9bd61c51731fb276fa438d606eaddd06fbf5748d2b79f5f not found: ID does not exist" Sep 16 07:36:35.741135 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.741045 2552 scope.go:117] "RemoveContainer" containerID="06b82993f95b5ea4df0dcb1a0d42d739a63f2ca530572b38af228910dd4ffdd1" Sep 16 07:36:35.741318 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.741300 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06b82993f95b5ea4df0dcb1a0d42d739a63f2ca530572b38af228910dd4ffdd1"} err="failed to get container status \"06b82993f95b5ea4df0dcb1a0d42d739a63f2ca530572b38af228910dd4ffdd1\": rpc error: code = NotFound desc = could not find container \"06b82993f95b5ea4df0dcb1a0d42d739a63f2ca530572b38af228910dd4ffdd1\": container with ID starting with 06b82993f95b5ea4df0dcb1a0d42d739a63f2ca530572b38af228910dd4ffdd1 not found: ID does not exist" Sep 16 07:36:35.741368 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.741318 2552 scope.go:117] "RemoveContainer" containerID="8c9986675788ef785b7af85b8926f210f1ad0f7b0cbc8643504cca48d7f3e579" Sep 16 07:36:35.741548 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.741529 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c9986675788ef785b7af85b8926f210f1ad0f7b0cbc8643504cca48d7f3e579"} err="failed to get container status \"8c9986675788ef785b7af85b8926f210f1ad0f7b0cbc8643504cca48d7f3e579\": rpc error: code = NotFound desc = could not find container \"8c9986675788ef785b7af85b8926f210f1ad0f7b0cbc8643504cca48d7f3e579\": container with ID starting with 8c9986675788ef785b7af85b8926f210f1ad0f7b0cbc8643504cca48d7f3e579 not found: ID does not exist" Sep 16 07:36:35.741598 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.741548 2552 scope.go:117] "RemoveContainer" containerID="ffbaad20a66e7eb73abf245c7723a9676a14bae01819e0e116e2041933ff1362" Sep 16 07:36:35.741766 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.741750 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffbaad20a66e7eb73abf245c7723a9676a14bae01819e0e116e2041933ff1362"} err="failed to get container status \"ffbaad20a66e7eb73abf245c7723a9676a14bae01819e0e116e2041933ff1362\": rpc error: code = NotFound desc = could not find container \"ffbaad20a66e7eb73abf245c7723a9676a14bae01819e0e116e2041933ff1362\": container with ID starting with ffbaad20a66e7eb73abf245c7723a9676a14bae01819e0e116e2041933ff1362 not found: ID does not exist" Sep 16 07:36:35.741807 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.741766 2552 scope.go:117] "RemoveContainer" containerID="af3184c588433b88f5583bb186c02e5c5142ddff86460c3b83822a1a8a33ccdf" Sep 16 07:36:35.741987 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.741968 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af3184c588433b88f5583bb186c02e5c5142ddff86460c3b83822a1a8a33ccdf"} err="failed to get container status \"af3184c588433b88f5583bb186c02e5c5142ddff86460c3b83822a1a8a33ccdf\": rpc error: code = NotFound desc = could not find container \"af3184c588433b88f5583bb186c02e5c5142ddff86460c3b83822a1a8a33ccdf\": container with ID starting with af3184c588433b88f5583bb186c02e5c5142ddff86460c3b83822a1a8a33ccdf not found: ID does not exist" Sep 16 07:36:35.742031 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.741987 2552 scope.go:117] "RemoveContainer" containerID="d0ca0fa64eecb52c0bb358e0aa1c0a9283cfe1d52a301ff02a38ec827f96117f" Sep 16 07:36:35.742233 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.742215 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0ca0fa64eecb52c0bb358e0aa1c0a9283cfe1d52a301ff02a38ec827f96117f"} err="failed to get container status \"d0ca0fa64eecb52c0bb358e0aa1c0a9283cfe1d52a301ff02a38ec827f96117f\": rpc error: code = NotFound desc = could not find container \"d0ca0fa64eecb52c0bb358e0aa1c0a9283cfe1d52a301ff02a38ec827f96117f\": container with ID starting with d0ca0fa64eecb52c0bb358e0aa1c0a9283cfe1d52a301ff02a38ec827f96117f not found: ID does not exist" Sep 16 07:36:35.742273 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.742233 2552 scope.go:117] "RemoveContainer" containerID="ee4977c84cfaf35bfed2d2569dadde5c1c599858d172d26e5278c6c01497c290" Sep 16 07:36:35.742448 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.742432 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee4977c84cfaf35bfed2d2569dadde5c1c599858d172d26e5278c6c01497c290"} err="failed to get container status \"ee4977c84cfaf35bfed2d2569dadde5c1c599858d172d26e5278c6c01497c290\": rpc error: code = NotFound desc = could not find container \"ee4977c84cfaf35bfed2d2569dadde5c1c599858d172d26e5278c6c01497c290\": container with ID starting with ee4977c84cfaf35bfed2d2569dadde5c1c599858d172d26e5278c6c01497c290 not found: ID does not exist" Sep 16 07:36:35.742485 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.742448 2552 scope.go:117] "RemoveContainer" containerID="1485fcfa9b29747ab9bd61c51731fb276fa438d606eaddd06fbf5748d2b79f5f" Sep 16 07:36:35.742645 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.742626 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1485fcfa9b29747ab9bd61c51731fb276fa438d606eaddd06fbf5748d2b79f5f"} err="failed to get container status \"1485fcfa9b29747ab9bd61c51731fb276fa438d606eaddd06fbf5748d2b79f5f\": rpc error: code = NotFound desc = could not find container \"1485fcfa9b29747ab9bd61c51731fb276fa438d606eaddd06fbf5748d2b79f5f\": container with ID starting with 1485fcfa9b29747ab9bd61c51731fb276fa438d606eaddd06fbf5748d2b79f5f not found: ID does not exist" Sep 16 07:36:35.742687 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.742647 2552 scope.go:117] "RemoveContainer" containerID="06b82993f95b5ea4df0dcb1a0d42d739a63f2ca530572b38af228910dd4ffdd1" Sep 16 07:36:35.742861 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.742845 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06b82993f95b5ea4df0dcb1a0d42d739a63f2ca530572b38af228910dd4ffdd1"} err="failed to get container status \"06b82993f95b5ea4df0dcb1a0d42d739a63f2ca530572b38af228910dd4ffdd1\": rpc error: code = NotFound desc = could not find container \"06b82993f95b5ea4df0dcb1a0d42d739a63f2ca530572b38af228910dd4ffdd1\": container with ID starting with 06b82993f95b5ea4df0dcb1a0d42d739a63f2ca530572b38af228910dd4ffdd1 not found: ID does not exist" Sep 16 07:36:35.742861 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.742860 2552 scope.go:117] "RemoveContainer" containerID="8c9986675788ef785b7af85b8926f210f1ad0f7b0cbc8643504cca48d7f3e579" Sep 16 07:36:35.743068 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.743052 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c9986675788ef785b7af85b8926f210f1ad0f7b0cbc8643504cca48d7f3e579"} err="failed to get container status \"8c9986675788ef785b7af85b8926f210f1ad0f7b0cbc8643504cca48d7f3e579\": rpc error: code = NotFound desc = could not find container \"8c9986675788ef785b7af85b8926f210f1ad0f7b0cbc8643504cca48d7f3e579\": container with ID starting with 8c9986675788ef785b7af85b8926f210f1ad0f7b0cbc8643504cca48d7f3e579 not found: ID does not exist" Sep 16 07:36:35.743142 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.743068 2552 scope.go:117] "RemoveContainer" containerID="ffbaad20a66e7eb73abf245c7723a9676a14bae01819e0e116e2041933ff1362" Sep 16 07:36:35.743318 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.743303 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffbaad20a66e7eb73abf245c7723a9676a14bae01819e0e116e2041933ff1362"} err="failed to get container status \"ffbaad20a66e7eb73abf245c7723a9676a14bae01819e0e116e2041933ff1362\": rpc error: code = NotFound desc = could not find container \"ffbaad20a66e7eb73abf245c7723a9676a14bae01819e0e116e2041933ff1362\": container with ID starting with ffbaad20a66e7eb73abf245c7723a9676a14bae01819e0e116e2041933ff1362 not found: ID does not exist" Sep 16 07:36:35.743365 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.743318 2552 scope.go:117] "RemoveContainer" containerID="af3184c588433b88f5583bb186c02e5c5142ddff86460c3b83822a1a8a33ccdf" Sep 16 07:36:35.743532 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.743515 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af3184c588433b88f5583bb186c02e5c5142ddff86460c3b83822a1a8a33ccdf"} err="failed to get container status \"af3184c588433b88f5583bb186c02e5c5142ddff86460c3b83822a1a8a33ccdf\": rpc error: code = NotFound desc = could not find container \"af3184c588433b88f5583bb186c02e5c5142ddff86460c3b83822a1a8a33ccdf\": container with ID starting with af3184c588433b88f5583bb186c02e5c5142ddff86460c3b83822a1a8a33ccdf not found: ID does not exist" Sep 16 07:36:35.743575 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.743533 2552 scope.go:117] "RemoveContainer" containerID="d0ca0fa64eecb52c0bb358e0aa1c0a9283cfe1d52a301ff02a38ec827f96117f" Sep 16 07:36:35.743722 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.743705 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0ca0fa64eecb52c0bb358e0aa1c0a9283cfe1d52a301ff02a38ec827f96117f"} err="failed to get container status \"d0ca0fa64eecb52c0bb358e0aa1c0a9283cfe1d52a301ff02a38ec827f96117f\": rpc error: code = NotFound desc = could not find container \"d0ca0fa64eecb52c0bb358e0aa1c0a9283cfe1d52a301ff02a38ec827f96117f\": container with ID starting with d0ca0fa64eecb52c0bb358e0aa1c0a9283cfe1d52a301ff02a38ec827f96117f not found: ID does not exist" Sep 16 07:36:35.743764 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.743722 2552 scope.go:117] "RemoveContainer" containerID="ee4977c84cfaf35bfed2d2569dadde5c1c599858d172d26e5278c6c01497c290" Sep 16 07:36:35.743909 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.743894 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee4977c84cfaf35bfed2d2569dadde5c1c599858d172d26e5278c6c01497c290"} err="failed to get container status \"ee4977c84cfaf35bfed2d2569dadde5c1c599858d172d26e5278c6c01497c290\": rpc error: code = NotFound desc = could not find container \"ee4977c84cfaf35bfed2d2569dadde5c1c599858d172d26e5278c6c01497c290\": container with ID starting with ee4977c84cfaf35bfed2d2569dadde5c1c599858d172d26e5278c6c01497c290 not found: ID does not exist" Sep 16 07:36:35.743949 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.743909 2552 scope.go:117] "RemoveContainer" containerID="1485fcfa9b29747ab9bd61c51731fb276fa438d606eaddd06fbf5748d2b79f5f" Sep 16 07:36:35.744162 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.744091 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1485fcfa9b29747ab9bd61c51731fb276fa438d606eaddd06fbf5748d2b79f5f"} err="failed to get container status \"1485fcfa9b29747ab9bd61c51731fb276fa438d606eaddd06fbf5748d2b79f5f\": rpc error: code = NotFound desc = could not find container \"1485fcfa9b29747ab9bd61c51731fb276fa438d606eaddd06fbf5748d2b79f5f\": container with ID starting with 1485fcfa9b29747ab9bd61c51731fb276fa438d606eaddd06fbf5748d2b79f5f not found: ID does not exist" Sep 16 07:36:35.744162 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.744141 2552 scope.go:117] "RemoveContainer" containerID="06b82993f95b5ea4df0dcb1a0d42d739a63f2ca530572b38af228910dd4ffdd1" Sep 16 07:36:35.744776 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.744750 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06b82993f95b5ea4df0dcb1a0d42d739a63f2ca530572b38af228910dd4ffdd1"} err="failed to get container status \"06b82993f95b5ea4df0dcb1a0d42d739a63f2ca530572b38af228910dd4ffdd1\": rpc error: code = NotFound desc = could not find container \"06b82993f95b5ea4df0dcb1a0d42d739a63f2ca530572b38af228910dd4ffdd1\": container with ID starting with 06b82993f95b5ea4df0dcb1a0d42d739a63f2ca530572b38af228910dd4ffdd1 not found: ID does not exist" Sep 16 07:36:35.745002 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.744806 2552 scope.go:117] "RemoveContainer" containerID="8c9986675788ef785b7af85b8926f210f1ad0f7b0cbc8643504cca48d7f3e579" Sep 16 07:36:35.745184 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.745163 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c9986675788ef785b7af85b8926f210f1ad0f7b0cbc8643504cca48d7f3e579"} err="failed to get container status \"8c9986675788ef785b7af85b8926f210f1ad0f7b0cbc8643504cca48d7f3e579\": rpc error: code = NotFound desc = could not find container \"8c9986675788ef785b7af85b8926f210f1ad0f7b0cbc8643504cca48d7f3e579\": container with ID starting with 8c9986675788ef785b7af85b8926f210f1ad0f7b0cbc8643504cca48d7f3e579 not found: ID does not exist" Sep 16 07:36:35.745274 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.745193 2552 scope.go:117] "RemoveContainer" containerID="ffbaad20a66e7eb73abf245c7723a9676a14bae01819e0e116e2041933ff1362" Sep 16 07:36:35.745451 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.745433 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffbaad20a66e7eb73abf245c7723a9676a14bae01819e0e116e2041933ff1362"} err="failed to get container status \"ffbaad20a66e7eb73abf245c7723a9676a14bae01819e0e116e2041933ff1362\": rpc error: code = NotFound desc = could not find container \"ffbaad20a66e7eb73abf245c7723a9676a14bae01819e0e116e2041933ff1362\": container with ID starting with ffbaad20a66e7eb73abf245c7723a9676a14bae01819e0e116e2041933ff1362 not found: ID does not exist" Sep 16 07:36:35.745498 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.745452 2552 scope.go:117] "RemoveContainer" containerID="af3184c588433b88f5583bb186c02e5c5142ddff86460c3b83822a1a8a33ccdf" Sep 16 07:36:35.745705 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.745680 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af3184c588433b88f5583bb186c02e5c5142ddff86460c3b83822a1a8a33ccdf"} err="failed to get container status \"af3184c588433b88f5583bb186c02e5c5142ddff86460c3b83822a1a8a33ccdf\": rpc error: code = NotFound desc = could not find container \"af3184c588433b88f5583bb186c02e5c5142ddff86460c3b83822a1a8a33ccdf\": container with ID starting with af3184c588433b88f5583bb186c02e5c5142ddff86460c3b83822a1a8a33ccdf not found: ID does not exist" Sep 16 07:36:35.745762 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.745707 2552 scope.go:117] "RemoveContainer" containerID="d0ca0fa64eecb52c0bb358e0aa1c0a9283cfe1d52a301ff02a38ec827f96117f" Sep 16 07:36:35.745977 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.745953 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0ca0fa64eecb52c0bb358e0aa1c0a9283cfe1d52a301ff02a38ec827f96117f"} err="failed to get container status \"d0ca0fa64eecb52c0bb358e0aa1c0a9283cfe1d52a301ff02a38ec827f96117f\": rpc error: code = NotFound desc = could not find container \"d0ca0fa64eecb52c0bb358e0aa1c0a9283cfe1d52a301ff02a38ec827f96117f\": container with ID starting with d0ca0fa64eecb52c0bb358e0aa1c0a9283cfe1d52a301ff02a38ec827f96117f not found: ID does not exist" Sep 16 07:36:35.746020 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.745977 2552 scope.go:117] "RemoveContainer" containerID="ee4977c84cfaf35bfed2d2569dadde5c1c599858d172d26e5278c6c01497c290" Sep 16 07:36:35.746223 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.746207 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee4977c84cfaf35bfed2d2569dadde5c1c599858d172d26e5278c6c01497c290"} err="failed to get container status \"ee4977c84cfaf35bfed2d2569dadde5c1c599858d172d26e5278c6c01497c290\": rpc error: code = NotFound desc = could not find container \"ee4977c84cfaf35bfed2d2569dadde5c1c599858d172d26e5278c6c01497c290\": container with ID starting with ee4977c84cfaf35bfed2d2569dadde5c1c599858d172d26e5278c6c01497c290 not found: ID does not exist" Sep 16 07:36:35.746223 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.746222 2552 scope.go:117] "RemoveContainer" containerID="1485fcfa9b29747ab9bd61c51731fb276fa438d606eaddd06fbf5748d2b79f5f" Sep 16 07:36:35.746428 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.746416 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1485fcfa9b29747ab9bd61c51731fb276fa438d606eaddd06fbf5748d2b79f5f"} err="failed to get container status \"1485fcfa9b29747ab9bd61c51731fb276fa438d606eaddd06fbf5748d2b79f5f\": rpc error: code = NotFound desc = could not find container \"1485fcfa9b29747ab9bd61c51731fb276fa438d606eaddd06fbf5748d2b79f5f\": container with ID starting with 1485fcfa9b29747ab9bd61c51731fb276fa438d606eaddd06fbf5748d2b79f5f not found: ID does not exist" Sep 16 07:36:35.746475 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.746428 2552 scope.go:117] "RemoveContainer" containerID="06b82993f95b5ea4df0dcb1a0d42d739a63f2ca530572b38af228910dd4ffdd1" Sep 16 07:36:35.746519 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.746488 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 16 07:36:35.746630 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.746614 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06b82993f95b5ea4df0dcb1a0d42d739a63f2ca530572b38af228910dd4ffdd1"} err="failed to get container status \"06b82993f95b5ea4df0dcb1a0d42d739a63f2ca530572b38af228910dd4ffdd1\": rpc error: code = NotFound desc = could not find container \"06b82993f95b5ea4df0dcb1a0d42d739a63f2ca530572b38af228910dd4ffdd1\": container with ID starting with 06b82993f95b5ea4df0dcb1a0d42d739a63f2ca530572b38af228910dd4ffdd1 not found: ID does not exist" Sep 16 07:36:35.746681 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.746630 2552 scope.go:117] "RemoveContainer" containerID="8c9986675788ef785b7af85b8926f210f1ad0f7b0cbc8643504cca48d7f3e579" Sep 16 07:36:35.746773 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.746760 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9d0d367-0a54-4d53-9a93-1721072b0b89" containerName="config-reloader" Sep 16 07:36:35.746813 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.746775 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9d0d367-0a54-4d53-9a93-1721072b0b89" containerName="config-reloader" Sep 16 07:36:35.746813 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.746791 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9d0d367-0a54-4d53-9a93-1721072b0b89" containerName="prometheus" Sep 16 07:36:35.746813 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.746802 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9d0d367-0a54-4d53-9a93-1721072b0b89" containerName="prometheus" Sep 16 07:36:35.746813 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.746811 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9d0d367-0a54-4d53-9a93-1721072b0b89" containerName="kube-rbac-proxy-web" Sep 16 07:36:35.746925 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.746818 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9d0d367-0a54-4d53-9a93-1721072b0b89" containerName="kube-rbac-proxy-web" Sep 16 07:36:35.746925 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.746826 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9d0d367-0a54-4d53-9a93-1721072b0b89" containerName="kube-rbac-proxy" Sep 16 07:36:35.746925 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.746831 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9d0d367-0a54-4d53-9a93-1721072b0b89" containerName="kube-rbac-proxy" Sep 16 07:36:35.746925 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.746837 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9d0d367-0a54-4d53-9a93-1721072b0b89" containerName="kube-rbac-proxy-thanos" Sep 16 07:36:35.746925 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.746856 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9d0d367-0a54-4d53-9a93-1721072b0b89" containerName="kube-rbac-proxy-thanos" Sep 16 07:36:35.746925 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.746865 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50be9bc4-8115-41e1-afb4-51a305538a5a" containerName="registry" Sep 16 07:36:35.746925 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.746871 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="50be9bc4-8115-41e1-afb4-51a305538a5a" containerName="registry" Sep 16 07:36:35.746925 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.746808 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c9986675788ef785b7af85b8926f210f1ad0f7b0cbc8643504cca48d7f3e579"} err="failed to get container status \"8c9986675788ef785b7af85b8926f210f1ad0f7b0cbc8643504cca48d7f3e579\": rpc error: code = NotFound desc = could not find container \"8c9986675788ef785b7af85b8926f210f1ad0f7b0cbc8643504cca48d7f3e579\": container with ID starting with 8c9986675788ef785b7af85b8926f210f1ad0f7b0cbc8643504cca48d7f3e579 not found: ID does not exist" Sep 16 07:36:35.746925 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.746891 2552 scope.go:117] "RemoveContainer" containerID="ffbaad20a66e7eb73abf245c7723a9676a14bae01819e0e116e2041933ff1362" Sep 16 07:36:35.746925 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.746879 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9d0d367-0a54-4d53-9a93-1721072b0b89" containerName="thanos-sidecar" Sep 16 07:36:35.747258 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.746930 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9d0d367-0a54-4d53-9a93-1721072b0b89" containerName="thanos-sidecar" Sep 16 07:36:35.747258 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.746942 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9d0d367-0a54-4d53-9a93-1721072b0b89" containerName="init-config-reloader" Sep 16 07:36:35.747258 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.746952 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9d0d367-0a54-4d53-9a93-1721072b0b89" containerName="init-config-reloader" Sep 16 07:36:35.747258 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.747025 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9d0d367-0a54-4d53-9a93-1721072b0b89" containerName="kube-rbac-proxy-web" Sep 16 07:36:35.747258 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.747035 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9d0d367-0a54-4d53-9a93-1721072b0b89" containerName="kube-rbac-proxy" Sep 16 07:36:35.747258 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.747045 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9d0d367-0a54-4d53-9a93-1721072b0b89" containerName="thanos-sidecar" Sep 16 07:36:35.747258 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.747054 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="50be9bc4-8115-41e1-afb4-51a305538a5a" containerName="registry" Sep 16 07:36:35.747258 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.747062 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9d0d367-0a54-4d53-9a93-1721072b0b89" containerName="prometheus" Sep 16 07:36:35.747258 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.747072 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9d0d367-0a54-4d53-9a93-1721072b0b89" containerName="config-reloader" Sep 16 07:36:35.747258 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.747080 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9d0d367-0a54-4d53-9a93-1721072b0b89" containerName="kube-rbac-proxy-thanos" Sep 16 07:36:35.747258 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.747149 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffbaad20a66e7eb73abf245c7723a9676a14bae01819e0e116e2041933ff1362"} err="failed to get container status \"ffbaad20a66e7eb73abf245c7723a9676a14bae01819e0e116e2041933ff1362\": rpc error: code = NotFound desc = could not find container \"ffbaad20a66e7eb73abf245c7723a9676a14bae01819e0e116e2041933ff1362\": container with ID starting with ffbaad20a66e7eb73abf245c7723a9676a14bae01819e0e116e2041933ff1362 not found: ID does not exist" Sep 16 07:36:35.747258 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.747164 2552 scope.go:117] "RemoveContainer" containerID="af3184c588433b88f5583bb186c02e5c5142ddff86460c3b83822a1a8a33ccdf" Sep 16 07:36:35.747676 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.747385 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af3184c588433b88f5583bb186c02e5c5142ddff86460c3b83822a1a8a33ccdf"} err="failed to get container status \"af3184c588433b88f5583bb186c02e5c5142ddff86460c3b83822a1a8a33ccdf\": rpc error: code = NotFound desc = could not find container \"af3184c588433b88f5583bb186c02e5c5142ddff86460c3b83822a1a8a33ccdf\": container with ID starting with af3184c588433b88f5583bb186c02e5c5142ddff86460c3b83822a1a8a33ccdf not found: ID does not exist" Sep 16 07:36:35.747676 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.747404 2552 scope.go:117] "RemoveContainer" containerID="d0ca0fa64eecb52c0bb358e0aa1c0a9283cfe1d52a301ff02a38ec827f96117f" Sep 16 07:36:35.747676 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.747628 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0ca0fa64eecb52c0bb358e0aa1c0a9283cfe1d52a301ff02a38ec827f96117f"} err="failed to get container status \"d0ca0fa64eecb52c0bb358e0aa1c0a9283cfe1d52a301ff02a38ec827f96117f\": rpc error: code = NotFound desc = could not find container \"d0ca0fa64eecb52c0bb358e0aa1c0a9283cfe1d52a301ff02a38ec827f96117f\": container with ID starting with d0ca0fa64eecb52c0bb358e0aa1c0a9283cfe1d52a301ff02a38ec827f96117f not found: ID does not exist" Sep 16 07:36:35.747676 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.747652 2552 scope.go:117] "RemoveContainer" containerID="ee4977c84cfaf35bfed2d2569dadde5c1c599858d172d26e5278c6c01497c290" Sep 16 07:36:35.747922 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.747904 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee4977c84cfaf35bfed2d2569dadde5c1c599858d172d26e5278c6c01497c290"} err="failed to get container status \"ee4977c84cfaf35bfed2d2569dadde5c1c599858d172d26e5278c6c01497c290\": rpc error: code = NotFound desc = could not find container \"ee4977c84cfaf35bfed2d2569dadde5c1c599858d172d26e5278c6c01497c290\": container with ID starting with ee4977c84cfaf35bfed2d2569dadde5c1c599858d172d26e5278c6c01497c290 not found: ID does not exist" Sep 16 07:36:35.752197 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.752179 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.754436 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.754414 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Sep 16 07:36:35.755092 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.754871 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Sep 16 07:36:35.755204 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.755182 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-1p9u7c0karkug\"" Sep 16 07:36:35.755897 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.755877 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-ftk58\"" Sep 16 07:36:35.755961 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.755922 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Sep 16 07:36:35.755961 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.755929 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Sep 16 07:36:35.756416 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.756395 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Sep 16 07:36:35.756528 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.756510 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Sep 16 07:36:35.756587 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.756543 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Sep 16 07:36:35.756770 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.756751 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Sep 16 07:36:35.756913 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.756760 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Sep 16 07:36:35.757003 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.756519 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Sep 16 07:36:35.757341 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.757321 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Sep 16 07:36:35.757698 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.757679 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Sep 16 07:36:35.757698 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.757731 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Sep 16 07:36:35.763763 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.763582 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Sep 16 07:36:35.765947 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.765917 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 16 07:36:35.766191 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.766175 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Sep 16 07:36:35.777956 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.777921 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.777956 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.777954 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.778159 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.777976 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.778159 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.777994 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-web-config\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.778159 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.778063 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.778159 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.778117 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.778159 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.778146 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-config-out\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.778313 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.778171 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.778313 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.778191 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.778313 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.778229 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.778313 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.778299 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.778425 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.778324 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.778425 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.778354 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.778425 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.778369 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hppnm\" (UniqueName: \"kubernetes.io/projected/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-kube-api-access-hppnm\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.778425 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.778386 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.778540 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.778433 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-config\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.778540 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.778455 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.778540 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.778473 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.778540 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.778497 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.778540 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.778524 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.778712 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.778574 2552 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e9d0d367-0a54-4d53-9a93-1721072b0b89-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:36:35.879047 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.878949 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.879047 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.878996 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.879047 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.879022 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.879047 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.879039 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hppnm\" (UniqueName: \"kubernetes.io/projected/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-kube-api-access-hppnm\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.879387 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.879056 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.879387 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.879082 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-config\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.879387 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.879238 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.879387 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.879268 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.879387 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.879296 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.879387 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.879319 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.879387 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.879344 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.879387 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.879365 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.879387 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.879390 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.879817 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.879424 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-web-config\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.879817 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.879461 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.879817 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.879496 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.879817 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.879520 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-config-out\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.879817 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.879548 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.879817 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.879570 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.879817 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.879602 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.880272 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.879932 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.880272 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.880037 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.880556 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.880483 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.881270 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.880729 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.881270 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.880809 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.882521 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.881626 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.882521 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.882187 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.882521 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.882310 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.882521 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.882448 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.882794 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.882529 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.883443 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.883397 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-config-out\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.884007 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.883965 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.884121 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.884006 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.884408 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.884381 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.884489 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.884468 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-config\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.884535 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.884504 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.884535 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.884521 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.884824 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.884806 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-web-config\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.885400 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.885380 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:35.886465 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:35.886448 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hppnm\" (UniqueName: \"kubernetes.io/projected/c8d25c0d-6cda-4dd3-997d-a9012823ce7a-kube-api-access-hppnm\") pod \"prometheus-k8s-0\" (UID: \"c8d25c0d-6cda-4dd3-997d-a9012823ce7a\") " pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:36.066695 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:36.066658 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:36.351969 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:36.351937 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Sep 16 07:36:36.354452 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:36:36.354425 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8d25c0d_6cda_4dd3_997d_a9012823ce7a.slice/crio-69557db45bae49f738d8bf6c7de6940a230a9d85ac9efd426dfcfddc451f1677 WatchSource:0}: Error finding container 69557db45bae49f738d8bf6c7de6940a230a9d85ac9efd426dfcfddc451f1677: Status 404 returned error can't find the container with id 69557db45bae49f738d8bf6c7de6940a230a9d85ac9efd426dfcfddc451f1677 Sep 16 07:36:36.690243 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:36.690207 2552 generic.go:356] "Generic (PLEG): container finished" podID="c8d25c0d-6cda-4dd3-997d-a9012823ce7a" containerID="63da3886b1a1d6f501b0eb3386b475b60d3e664a11ff7ea6743728e67be0a323" exitCode=0 Sep 16 07:36:36.690412 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:36.690270 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c8d25c0d-6cda-4dd3-997d-a9012823ce7a","Type":"ContainerDied","Data":"63da3886b1a1d6f501b0eb3386b475b60d3e664a11ff7ea6743728e67be0a323"} Sep 16 07:36:36.690412 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:36.690327 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c8d25c0d-6cda-4dd3-997d-a9012823ce7a","Type":"ContainerStarted","Data":"69557db45bae49f738d8bf6c7de6940a230a9d85ac9efd426dfcfddc451f1677"} Sep 16 07:36:36.937289 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:36.937252 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9d0d367-0a54-4d53-9a93-1721072b0b89" path="/var/lib/kubelet/pods/e9d0d367-0a54-4d53-9a93-1721072b0b89/volumes" Sep 16 07:36:37.698056 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:37.698019 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c8d25c0d-6cda-4dd3-997d-a9012823ce7a","Type":"ContainerStarted","Data":"063b49d6c505ffd95d7590b51328921f641298c7b73f650ebbafe686fceb1f9a"} Sep 16 07:36:37.698559 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:37.698154 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c8d25c0d-6cda-4dd3-997d-a9012823ce7a","Type":"ContainerStarted","Data":"c0a36bcff38ba7b36c508e598642c2e6bbe3b69b6fb7e465c6a6cf7bb9b2811c"} Sep 16 07:36:37.698559 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:37.698201 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c8d25c0d-6cda-4dd3-997d-a9012823ce7a","Type":"ContainerStarted","Data":"5b153d82e4a5587b5dae76471fc23d35f2f93aeba38ac25dc1e6141f90f2fb95"} Sep 16 07:36:37.698559 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:37.698247 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c8d25c0d-6cda-4dd3-997d-a9012823ce7a","Type":"ContainerStarted","Data":"8ed56f853523990fd2f5692bf8ea8909c6d712264678a1f0bd7545179b083768"} Sep 16 07:36:37.698559 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:37.698283 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c8d25c0d-6cda-4dd3-997d-a9012823ce7a","Type":"ContainerStarted","Data":"5e0c2f8b84af5dd017bc9db78a6b03bd8594d01dbb992841b805bc0810b8d597"} Sep 16 07:36:37.698559 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:37.698324 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c8d25c0d-6cda-4dd3-997d-a9012823ce7a","Type":"ContainerStarted","Data":"97f20fda4b6e05e0913cb0de96ceaa54769913a6ea9fdd72e41545f8e8b41f78"} Sep 16 07:36:37.725209 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:37.725148 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.705627864 podStartE2EDuration="2.725127215s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.499351ms" isStatefulPod=false podCreationTimestamp="2026-09-16 07:36:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:36:37.723975652 +0000 UTC m=+169.307754013" watchObservedRunningTime="2026-09-16 07:36:37.725127215 +0000 UTC m=+169.308905577" Sep 16 07:36:38.638834 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:38.638797 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:36:41.066867 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:41.066771 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:36:49.571947 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:49.571905 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:36:49.623877 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:49.623845 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:36:49.675760 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:36:49.675729 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:37:27.794121 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:37:27.794069 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:37:27.894628 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:37:27.894589 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:37:31.132729 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:37:31.132690 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:37:36.067816 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:37:36.067771 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:37:36.083568 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:37:36.083540 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:37:36.936248 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:37:36.936222 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Sep 16 07:37:45.137481 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:37:45.137433 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:37:45.188121 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:37:45.188062 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:37:49.579061 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:37:49.579013 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:37:49.632810 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:37:49.632772 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:37:49.735415 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:37:49.735382 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:37:49.787730 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:37:49.787698 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:37:50.542421 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:37:50.542382 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:37:50.592363 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:37:50.592321 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:37:57.058945 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:37:57.058916 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:37:57.112116 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:37:57.112074 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:37:59.994979 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:37:59.994950 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:38:00.045897 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:38:00.045867 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:38:00.868635 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:38:00.868608 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:38:00.922465 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:38:00.922441 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:38:00.983151 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:38:00.983114 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:38:04.717062 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:38:04.717029 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:38:04.818296 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:38:04.818260 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:38:04.918815 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:38:04.918787 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:38:05.120029 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:38:05.119936 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:38:48.816116 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:38:48.816065 2552 kubelet.go:1648] "Image garbage collection succeeded" Sep 16 07:39:03.652974 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:39:03.652932 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:39:04.899780 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:39:04.899744 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:39:30.563396 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:39:30.563360 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:39:31.921189 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:39:31.921145 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:39:34.157008 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:39:34.156974 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:39:34.209040 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:39:34.209000 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:39:36.039905 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:39:36.039869 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:39:36.139254 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:39:36.139223 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:39:36.452385 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:39:36.452349 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:39:36.561469 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:39:36.561434 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:39:38.056353 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:39:38.056318 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:39:38.441661 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:39:38.441631 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:39:41.162065 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:39:41.161970 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:39:41.216752 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:39:41.216708 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:39:44.242941 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:39:44.242905 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:39:44.295816 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:39:44.295777 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:39:46.644026 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:39:46.643987 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:39:48.644123 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:39:48.644064 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:39:48.744913 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:39:48.744866 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:39:53.931286 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:39:53.931251 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:39:54.034469 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:39:54.034439 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:39:55.615962 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:39:55.615917 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:07.507740 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:07.507700 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:07.852431 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:07.852339 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:07.900614 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:07.900578 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:08.453695 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:08.453663 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:08.506393 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:08.506345 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:11.613273 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:11.613229 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:11.705235 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:11.705204 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:12.224197 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:12.224161 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:12.449976 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:12.449944 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:12.555156 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:12.555047 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:12.602015 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:12.601975 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-dcb658945-hbqzx"] Sep 16 07:40:12.605357 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:12.605337 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-hbqzx" Sep 16 07:40:12.607741 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:12.607719 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 16 07:40:12.607741 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:12.607727 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 16 07:40:12.607949 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:12.607823 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 16 07:40:12.608023 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:12.608006 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 16 07:40:12.608370 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:12.608353 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-cljxh\"" Sep 16 07:40:12.617219 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:12.617197 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-dcb658945-hbqzx"] Sep 16 07:40:12.667417 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:12.667377 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c4734177-10ea-4111-bf3e-840fa34f0a6c-webhook-cert\") pod \"opendatahub-operator-controller-manager-dcb658945-hbqzx\" (UID: \"c4734177-10ea-4111-bf3e-840fa34f0a6c\") " pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-hbqzx" Sep 16 07:40:12.667593 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:12.667427 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c4734177-10ea-4111-bf3e-840fa34f0a6c-manifests\") pod \"opendatahub-operator-controller-manager-dcb658945-hbqzx\" (UID: \"c4734177-10ea-4111-bf3e-840fa34f0a6c\") " pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-hbqzx" Sep 16 07:40:12.667593 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:12.667452 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrkc4\" (UniqueName: \"kubernetes.io/projected/c4734177-10ea-4111-bf3e-840fa34f0a6c-kube-api-access-jrkc4\") pod \"opendatahub-operator-controller-manager-dcb658945-hbqzx\" (UID: \"c4734177-10ea-4111-bf3e-840fa34f0a6c\") " pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-hbqzx" Sep 16 07:40:12.667593 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:12.667488 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4734177-10ea-4111-bf3e-840fa34f0a6c-tmp\") pod \"opendatahub-operator-controller-manager-dcb658945-hbqzx\" (UID: \"c4734177-10ea-4111-bf3e-840fa34f0a6c\") " pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-hbqzx" Sep 16 07:40:12.667593 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:12.667514 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c4734177-10ea-4111-bf3e-840fa34f0a6c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dcb658945-hbqzx\" (UID: \"c4734177-10ea-4111-bf3e-840fa34f0a6c\") " pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-hbqzx" Sep 16 07:40:12.713544 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:12.713510 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:12.768607 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:12.768567 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c4734177-10ea-4111-bf3e-840fa34f0a6c-manifests\") pod \"opendatahub-operator-controller-manager-dcb658945-hbqzx\" (UID: \"c4734177-10ea-4111-bf3e-840fa34f0a6c\") " pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-hbqzx" Sep 16 07:40:12.768607 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:12.768607 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrkc4\" (UniqueName: \"kubernetes.io/projected/c4734177-10ea-4111-bf3e-840fa34f0a6c-kube-api-access-jrkc4\") pod \"opendatahub-operator-controller-manager-dcb658945-hbqzx\" (UID: \"c4734177-10ea-4111-bf3e-840fa34f0a6c\") " pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-hbqzx" Sep 16 07:40:12.768857 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:12.768638 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4734177-10ea-4111-bf3e-840fa34f0a6c-tmp\") pod \"opendatahub-operator-controller-manager-dcb658945-hbqzx\" (UID: \"c4734177-10ea-4111-bf3e-840fa34f0a6c\") " pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-hbqzx" Sep 16 07:40:12.768857 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:12.768668 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c4734177-10ea-4111-bf3e-840fa34f0a6c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dcb658945-hbqzx\" (UID: \"c4734177-10ea-4111-bf3e-840fa34f0a6c\") " pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-hbqzx" Sep 16 07:40:12.768857 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:12.768748 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c4734177-10ea-4111-bf3e-840fa34f0a6c-webhook-cert\") pod \"opendatahub-operator-controller-manager-dcb658945-hbqzx\" (UID: \"c4734177-10ea-4111-bf3e-840fa34f0a6c\") " pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-hbqzx" Sep 16 07:40:12.769064 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:12.769044 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c4734177-10ea-4111-bf3e-840fa34f0a6c-manifests\") pod \"opendatahub-operator-controller-manager-dcb658945-hbqzx\" (UID: \"c4734177-10ea-4111-bf3e-840fa34f0a6c\") " pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-hbqzx" Sep 16 07:40:12.771089 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:12.771059 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4734177-10ea-4111-bf3e-840fa34f0a6c-tmp\") pod \"opendatahub-operator-controller-manager-dcb658945-hbqzx\" (UID: \"c4734177-10ea-4111-bf3e-840fa34f0a6c\") " pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-hbqzx" Sep 16 07:40:12.771306 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:12.771286 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c4734177-10ea-4111-bf3e-840fa34f0a6c-webhook-cert\") pod \"opendatahub-operator-controller-manager-dcb658945-hbqzx\" (UID: \"c4734177-10ea-4111-bf3e-840fa34f0a6c\") " pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-hbqzx" Sep 16 07:40:12.771383 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:12.771284 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c4734177-10ea-4111-bf3e-840fa34f0a6c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dcb658945-hbqzx\" (UID: \"c4734177-10ea-4111-bf3e-840fa34f0a6c\") " pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-hbqzx" Sep 16 07:40:12.782036 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:12.782007 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrkc4\" (UniqueName: \"kubernetes.io/projected/c4734177-10ea-4111-bf3e-840fa34f0a6c-kube-api-access-jrkc4\") pod \"opendatahub-operator-controller-manager-dcb658945-hbqzx\" (UID: \"c4734177-10ea-4111-bf3e-840fa34f0a6c\") " pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-hbqzx" Sep 16 07:40:12.810799 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:12.810708 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:12.876970 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:12.876932 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:12.915803 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:12.915762 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-hbqzx" Sep 16 07:40:12.980498 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:12.980448 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:13.077651 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:13.077565 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:13.168820 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:13.168783 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:13.208337 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:40:13.208308 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4734177_10ea_4111_bf3e_840fa34f0a6c.slice/crio-a8e9198191f30c2f898b422a2b84e2d8a8f23b7748a7093700bb27a5e69ad4d8 WatchSource:0}: Error finding container a8e9198191f30c2f898b422a2b84e2d8a8f23b7748a7093700bb27a5e69ad4d8: Status 404 returned error can't find the container with id a8e9198191f30c2f898b422a2b84e2d8a8f23b7748a7093700bb27a5e69ad4d8 Sep 16 07:40:13.209942 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:13.209921 2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 07:40:13.220234 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:13.220202 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-dcb658945-hbqzx"] Sep 16 07:40:13.509494 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:13.509457 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-hbqzx" event={"ID":"c4734177-10ea-4111-bf3e-840fa34f0a6c","Type":"ContainerStarted","Data":"a8e9198191f30c2f898b422a2b84e2d8a8f23b7748a7093700bb27a5e69ad4d8"} Sep 16 07:40:16.524417 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:16.524378 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-hbqzx" event={"ID":"c4734177-10ea-4111-bf3e-840fa34f0a6c","Type":"ContainerStarted","Data":"345a568c2f9d42f5e332d315b4b7c50df46b5abded53a34f6847f89ee62fc154"} Sep 16 07:40:17.530066 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:17.530023 2552 generic.go:356] "Generic (PLEG): container finished" podID="c4734177-10ea-4111-bf3e-840fa34f0a6c" containerID="345a568c2f9d42f5e332d315b4b7c50df46b5abded53a34f6847f89ee62fc154" exitCode=0 Sep 16 07:40:17.530066 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:17.530063 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-hbqzx" event={"ID":"c4734177-10ea-4111-bf3e-840fa34f0a6c","Type":"ContainerDied","Data":"345a568c2f9d42f5e332d315b4b7c50df46b5abded53a34f6847f89ee62fc154"} Sep 16 07:40:18.536402 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:18.536361 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-hbqzx" event={"ID":"c4734177-10ea-4111-bf3e-840fa34f0a6c","Type":"ContainerStarted","Data":"4b2a403d850e6bb2eed993fc65d532a5fbf48c966e7477441242994c2ef337cf"} Sep 16 07:40:18.536776 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:18.536465 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-hbqzx" Sep 16 07:40:18.557607 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:18.557556 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-hbqzx" podStartSLOduration=2.193481922 podStartE2EDuration="6.557537229s" totalImagesPullingTime="3.260995783s" totalInitContainerRuntime="1.103059524s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:40:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:40:18.555667144 +0000 UTC m=+390.139445483" watchObservedRunningTime="2026-09-16 07:40:18.557537229 +0000 UTC m=+390.141315585" Sep 16 07:40:18.589794 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:18.589762 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:18.690836 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:18.690803 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:18.742634 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:18.742603 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:21.699777 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:21.699740 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:21.811358 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:21.811324 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:24.330379 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:24.330342 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:24.433695 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:24.433666 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:27.933705 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:27.933674 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:27.983856 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:27.983820 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:28.359437 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:28.359333 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:28.414351 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:28.414322 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:29.542430 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:29.542401 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-dcb658945-hbqzx" Sep 16 07:40:32.948970 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:32.948932 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:33.080629 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:33.080600 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:33.348196 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:33.348090 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:33.379128 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:33.379071 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7b6965477b-dxhrr"] Sep 16 07:40:33.393383 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:33.393345 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7b6965477b-dxhrr" Sep 16 07:40:33.396548 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:33.396405 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 16 07:40:33.396548 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:33.396500 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-mkp79\"" Sep 16 07:40:33.396769 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:33.396591 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 16 07:40:33.396769 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:33.396692 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 16 07:40:33.397181 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:33.397158 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 16 07:40:33.397421 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:33.397396 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7b6965477b-dxhrr"] Sep 16 07:40:33.426680 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:33.426637 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50f94726-d0ba-41b4-9288-cff938220e36-tls-certs\") pod \"kube-auth-proxy-7b6965477b-dxhrr\" (UID: \"50f94726-d0ba-41b4-9288-cff938220e36\") " pod="openshift-ingress/kube-auth-proxy-7b6965477b-dxhrr" Sep 16 07:40:33.426870 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:33.426700 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c9mz\" (UniqueName: \"kubernetes.io/projected/50f94726-d0ba-41b4-9288-cff938220e36-kube-api-access-9c9mz\") pod \"kube-auth-proxy-7b6965477b-dxhrr\" (UID: \"50f94726-d0ba-41b4-9288-cff938220e36\") " pod="openshift-ingress/kube-auth-proxy-7b6965477b-dxhrr" Sep 16 07:40:33.426870 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:33.426748 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50f94726-d0ba-41b4-9288-cff938220e36-tmp\") pod \"kube-auth-proxy-7b6965477b-dxhrr\" (UID: \"50f94726-d0ba-41b4-9288-cff938220e36\") " pod="openshift-ingress/kube-auth-proxy-7b6965477b-dxhrr" Sep 16 07:40:33.527500 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:33.527460 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c9mz\" (UniqueName: \"kubernetes.io/projected/50f94726-d0ba-41b4-9288-cff938220e36-kube-api-access-9c9mz\") pod \"kube-auth-proxy-7b6965477b-dxhrr\" (UID: \"50f94726-d0ba-41b4-9288-cff938220e36\") " pod="openshift-ingress/kube-auth-proxy-7b6965477b-dxhrr" Sep 16 07:40:33.527703 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:33.527521 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50f94726-d0ba-41b4-9288-cff938220e36-tmp\") pod \"kube-auth-proxy-7b6965477b-dxhrr\" (UID: \"50f94726-d0ba-41b4-9288-cff938220e36\") " pod="openshift-ingress/kube-auth-proxy-7b6965477b-dxhrr" Sep 16 07:40:33.527703 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:33.527559 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50f94726-d0ba-41b4-9288-cff938220e36-tls-certs\") pod \"kube-auth-proxy-7b6965477b-dxhrr\" (UID: \"50f94726-d0ba-41b4-9288-cff938220e36\") " pod="openshift-ingress/kube-auth-proxy-7b6965477b-dxhrr" Sep 16 07:40:33.529883 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:33.529854 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50f94726-d0ba-41b4-9288-cff938220e36-tmp\") pod \"kube-auth-proxy-7b6965477b-dxhrr\" (UID: \"50f94726-d0ba-41b4-9288-cff938220e36\") " pod="openshift-ingress/kube-auth-proxy-7b6965477b-dxhrr" Sep 16 07:40:33.530113 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:33.530081 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50f94726-d0ba-41b4-9288-cff938220e36-tls-certs\") pod \"kube-auth-proxy-7b6965477b-dxhrr\" (UID: \"50f94726-d0ba-41b4-9288-cff938220e36\") " pod="openshift-ingress/kube-auth-proxy-7b6965477b-dxhrr" Sep 16 07:40:33.536381 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:33.536352 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c9mz\" (UniqueName: \"kubernetes.io/projected/50f94726-d0ba-41b4-9288-cff938220e36-kube-api-access-9c9mz\") pod \"kube-auth-proxy-7b6965477b-dxhrr\" (UID: \"50f94726-d0ba-41b4-9288-cff938220e36\") " pod="openshift-ingress/kube-auth-proxy-7b6965477b-dxhrr" Sep 16 07:40:33.565558 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:33.565528 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:33.704532 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:33.704499 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7b6965477b-dxhrr" Sep 16 07:40:33.987744 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:33.987661 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7b6965477b-dxhrr"] Sep 16 07:40:33.991324 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:40:33.991294 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f94726_d0ba_41b4_9288_cff938220e36.slice/crio-a0daecb54997f3a539a3d58a91a1d29c11894acffd105ac41a1aeccb9e38055f WatchSource:0}: Error finding container a0daecb54997f3a539a3d58a91a1d29c11894acffd105ac41a1aeccb9e38055f: Status 404 returned error can't find the container with id a0daecb54997f3a539a3d58a91a1d29c11894acffd105ac41a1aeccb9e38055f Sep 16 07:40:34.543125 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:34.543066 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:34.599485 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:34.599447 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7b6965477b-dxhrr" event={"ID":"50f94726-d0ba-41b4-9288-cff938220e36","Type":"ContainerStarted","Data":"a0daecb54997f3a539a3d58a91a1d29c11894acffd105ac41a1aeccb9e38055f"} Sep 16 07:40:34.648722 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:34.648684 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:35.429203 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:35.429150 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:37.613815 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:37.613779 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7b6965477b-dxhrr" event={"ID":"50f94726-d0ba-41b4-9288-cff938220e36","Type":"ContainerStarted","Data":"0c31bcde9c825cfdd5da2a0a112f37ab2a1b7dca5114ac75c496d01d831426f8"} Sep 16 07:40:37.629485 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:37.629413 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7b6965477b-dxhrr" podStartSLOduration=1.9610799 podStartE2EDuration="4.629392354s" totalImagesPullingTime="2.668312454s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:40:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:40:37.628344185 +0000 UTC m=+409.212122524" watchObservedRunningTime="2026-09-16 07:40:37.629392354 +0000 UTC m=+409.213170693" Sep 16 07:40:39.046250 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:39.046213 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:39.149460 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:39.149426 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:39.638154 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:39.638119 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:39.824433 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:39.824397 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:40.121563 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:40.121531 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:42.473430 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:42.473398 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:42.528177 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:42.528147 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:43.873792 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:43.873743 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:43.971094 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:43.971062 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:44.071465 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:44.071426 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:44.173958 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:44.173928 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:44.372359 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:44.372304 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:44.378207 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:44.378169 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-6fw9g"] Sep 16 07:40:44.381676 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:44.381656 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-6fw9g" Sep 16 07:40:44.383893 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:44.383865 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-tmvt4\"" Sep 16 07:40:44.384038 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:44.383972 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Sep 16 07:40:44.392430 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:44.392398 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-6fw9g"] Sep 16 07:40:44.470584 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:44.470496 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:44.523453 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:44.523421 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/962c76dc-bb7c-45bf-b4c1-fc2968fa707c-tls-certs\") pod \"model-serving-api-54d89bd5fc-6fw9g\" (UID: \"962c76dc-bb7c-45bf-b4c1-fc2968fa707c\") " pod="opendatahub/model-serving-api-54d89bd5fc-6fw9g" Sep 16 07:40:44.523453 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:44.523459 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wk58\" (UniqueName: \"kubernetes.io/projected/962c76dc-bb7c-45bf-b4c1-fc2968fa707c-kube-api-access-6wk58\") pod \"model-serving-api-54d89bd5fc-6fw9g\" (UID: \"962c76dc-bb7c-45bf-b4c1-fc2968fa707c\") " pod="opendatahub/model-serving-api-54d89bd5fc-6fw9g" Sep 16 07:40:44.558585 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:44.558541 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:44.624813 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:44.624772 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/962c76dc-bb7c-45bf-b4c1-fc2968fa707c-tls-certs\") pod \"model-serving-api-54d89bd5fc-6fw9g\" (UID: \"962c76dc-bb7c-45bf-b4c1-fc2968fa707c\") " pod="opendatahub/model-serving-api-54d89bd5fc-6fw9g" Sep 16 07:40:44.624813 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:44.624814 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wk58\" (UniqueName: \"kubernetes.io/projected/962c76dc-bb7c-45bf-b4c1-fc2968fa707c-kube-api-access-6wk58\") pod \"model-serving-api-54d89bd5fc-6fw9g\" (UID: \"962c76dc-bb7c-45bf-b4c1-fc2968fa707c\") " pod="opendatahub/model-serving-api-54d89bd5fc-6fw9g" Sep 16 07:40:44.627259 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:44.627228 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/962c76dc-bb7c-45bf-b4c1-fc2968fa707c-tls-certs\") pod \"model-serving-api-54d89bd5fc-6fw9g\" (UID: \"962c76dc-bb7c-45bf-b4c1-fc2968fa707c\") " pod="opendatahub/model-serving-api-54d89bd5fc-6fw9g" Sep 16 07:40:44.631761 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:44.631732 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wk58\" (UniqueName: \"kubernetes.io/projected/962c76dc-bb7c-45bf-b4c1-fc2968fa707c-kube-api-access-6wk58\") pod \"model-serving-api-54d89bd5fc-6fw9g\" (UID: \"962c76dc-bb7c-45bf-b4c1-fc2968fa707c\") " pod="opendatahub/model-serving-api-54d89bd5fc-6fw9g" Sep 16 07:40:44.691371 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:44.691331 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-6fw9g" Sep 16 07:40:44.963422 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:44.963383 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-6fw9g"] Sep 16 07:40:44.966624 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:40:44.966594 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod962c76dc_bb7c_45bf_b4c1_fc2968fa707c.slice/crio-3abd38bbda6a75f26615a631b9daa127ed3ef31455fd93ec01dd9d4cc449606b WatchSource:0}: Error finding container 3abd38bbda6a75f26615a631b9daa127ed3ef31455fd93ec01dd9d4cc449606b: Status 404 returned error can't find the container with id 3abd38bbda6a75f26615a631b9daa127ed3ef31455fd93ec01dd9d4cc449606b Sep 16 07:40:45.645735 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:45.645695 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-6fw9g" event={"ID":"962c76dc-bb7c-45bf-b4c1-fc2968fa707c","Type":"ContainerStarted","Data":"3abd38bbda6a75f26615a631b9daa127ed3ef31455fd93ec01dd9d4cc449606b"} Sep 16 07:40:46.651610 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:46.651571 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-6fw9g" event={"ID":"962c76dc-bb7c-45bf-b4c1-fc2968fa707c","Type":"ContainerStarted","Data":"d619adb09d0cfef79681af460164f96b2a169d72d8d6474e33e754c1fbd5a2cd"} Sep 16 07:40:46.652136 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:46.651696 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-6fw9g" Sep 16 07:40:46.668546 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:46.668487 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-6fw9g" podStartSLOduration=1.109738872 podStartE2EDuration="2.668469244s" totalImagesPullingTime="1.558730372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:40:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:40:46.666512946 +0000 UTC m=+418.250291284" watchObservedRunningTime="2026-09-16 07:40:46.668469244 +0000 UTC m=+418.252247581" Sep 16 07:40:46.734512 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:46.734476 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:46.791536 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:46.791450 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:49.273656 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:49.273616 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6f78fc697-hxrsc"] Sep 16 07:40:49.278533 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:49.278504 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6f78fc697-hxrsc" Sep 16 07:40:49.280785 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:49.280751 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Sep 16 07:40:49.280910 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:49.280885 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Sep 16 07:40:49.281795 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:49.281767 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Sep 16 07:40:49.281896 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:49.281779 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Sep 16 07:40:49.281896 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:49.281821 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Sep 16 07:40:49.281978 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:49.281918 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-hprlw\"" Sep 16 07:40:49.286192 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:49.285817 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6f78fc697-hxrsc"] Sep 16 07:40:49.365871 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:49.365827 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5602b217-00e1-4d62-b1e1-982a88eb3f5a-manager-config\") pod \"lws-controller-manager-6f78fc697-hxrsc\" (UID: \"5602b217-00e1-4d62-b1e1-982a88eb3f5a\") " pod="openshift-lws-operator/lws-controller-manager-6f78fc697-hxrsc" Sep 16 07:40:49.365871 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:49.365875 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5602b217-00e1-4d62-b1e1-982a88eb3f5a-cert\") pod \"lws-controller-manager-6f78fc697-hxrsc\" (UID: \"5602b217-00e1-4d62-b1e1-982a88eb3f5a\") " pod="openshift-lws-operator/lws-controller-manager-6f78fc697-hxrsc" Sep 16 07:40:49.366087 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:49.365935 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5602b217-00e1-4d62-b1e1-982a88eb3f5a-metrics-cert\") pod \"lws-controller-manager-6f78fc697-hxrsc\" (UID: \"5602b217-00e1-4d62-b1e1-982a88eb3f5a\") " pod="openshift-lws-operator/lws-controller-manager-6f78fc697-hxrsc" Sep 16 07:40:49.366087 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:49.366059 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrcdr\" (UniqueName: \"kubernetes.io/projected/5602b217-00e1-4d62-b1e1-982a88eb3f5a-kube-api-access-mrcdr\") pod \"lws-controller-manager-6f78fc697-hxrsc\" (UID: \"5602b217-00e1-4d62-b1e1-982a88eb3f5a\") " pod="openshift-lws-operator/lws-controller-manager-6f78fc697-hxrsc" Sep 16 07:40:49.467398 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:49.467358 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrcdr\" (UniqueName: \"kubernetes.io/projected/5602b217-00e1-4d62-b1e1-982a88eb3f5a-kube-api-access-mrcdr\") pod \"lws-controller-manager-6f78fc697-hxrsc\" (UID: \"5602b217-00e1-4d62-b1e1-982a88eb3f5a\") " pod="openshift-lws-operator/lws-controller-manager-6f78fc697-hxrsc" Sep 16 07:40:49.467398 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:49.467403 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5602b217-00e1-4d62-b1e1-982a88eb3f5a-manager-config\") pod \"lws-controller-manager-6f78fc697-hxrsc\" (UID: \"5602b217-00e1-4d62-b1e1-982a88eb3f5a\") " pod="openshift-lws-operator/lws-controller-manager-6f78fc697-hxrsc" Sep 16 07:40:49.467663 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:49.467430 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5602b217-00e1-4d62-b1e1-982a88eb3f5a-cert\") pod \"lws-controller-manager-6f78fc697-hxrsc\" (UID: \"5602b217-00e1-4d62-b1e1-982a88eb3f5a\") " pod="openshift-lws-operator/lws-controller-manager-6f78fc697-hxrsc" Sep 16 07:40:49.467663 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:49.467469 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5602b217-00e1-4d62-b1e1-982a88eb3f5a-metrics-cert\") pod \"lws-controller-manager-6f78fc697-hxrsc\" (UID: \"5602b217-00e1-4d62-b1e1-982a88eb3f5a\") " pod="openshift-lws-operator/lws-controller-manager-6f78fc697-hxrsc" Sep 16 07:40:49.468049 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:49.468019 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/5602b217-00e1-4d62-b1e1-982a88eb3f5a-manager-config\") pod \"lws-controller-manager-6f78fc697-hxrsc\" (UID: \"5602b217-00e1-4d62-b1e1-982a88eb3f5a\") " pod="openshift-lws-operator/lws-controller-manager-6f78fc697-hxrsc" Sep 16 07:40:49.469953 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:49.469928 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5602b217-00e1-4d62-b1e1-982a88eb3f5a-metrics-cert\") pod \"lws-controller-manager-6f78fc697-hxrsc\" (UID: \"5602b217-00e1-4d62-b1e1-982a88eb3f5a\") " pod="openshift-lws-operator/lws-controller-manager-6f78fc697-hxrsc" Sep 16 07:40:49.470085 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:49.470065 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5602b217-00e1-4d62-b1e1-982a88eb3f5a-cert\") pod \"lws-controller-manager-6f78fc697-hxrsc\" (UID: \"5602b217-00e1-4d62-b1e1-982a88eb3f5a\") " pod="openshift-lws-operator/lws-controller-manager-6f78fc697-hxrsc" Sep 16 07:40:49.475424 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:49.475396 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrcdr\" (UniqueName: \"kubernetes.io/projected/5602b217-00e1-4d62-b1e1-982a88eb3f5a-kube-api-access-mrcdr\") pod \"lws-controller-manager-6f78fc697-hxrsc\" (UID: \"5602b217-00e1-4d62-b1e1-982a88eb3f5a\") " pod="openshift-lws-operator/lws-controller-manager-6f78fc697-hxrsc" Sep 16 07:40:49.589467 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:49.589361 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6f78fc697-hxrsc" Sep 16 07:40:49.863124 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:49.863005 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6f78fc697-hxrsc"] Sep 16 07:40:49.867004 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:40:49.866967 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5602b217_00e1_4d62_b1e1_982a88eb3f5a.slice/crio-a7d1fd44dc5cb22aa179aa6d48252c646c41dd09141286af258aade23737ce09 WatchSource:0}: Error finding container a7d1fd44dc5cb22aa179aa6d48252c646c41dd09141286af258aade23737ce09: Status 404 returned error can't find the container with id a7d1fd44dc5cb22aa179aa6d48252c646c41dd09141286af258aade23737ce09 Sep 16 07:40:50.671076 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:50.671034 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6f78fc697-hxrsc" event={"ID":"5602b217-00e1-4d62-b1e1-982a88eb3f5a","Type":"ContainerStarted","Data":"a7d1fd44dc5cb22aa179aa6d48252c646c41dd09141286af258aade23737ce09"} Sep 16 07:40:52.681969 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:52.681928 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6f78fc697-hxrsc" event={"ID":"5602b217-00e1-4d62-b1e1-982a88eb3f5a","Type":"ContainerStarted","Data":"aed55dfda355d718e8bfe4b8116464f714401b42d7c6c136f7fa975927dc724a"} Sep 16 07:40:52.682391 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:52.682004 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6f78fc697-hxrsc" Sep 16 07:40:52.701377 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:52.701322 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6f78fc697-hxrsc" podStartSLOduration=1.520109347 podStartE2EDuration="3.701302961s" totalImagesPullingTime="2.181193614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:40:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:40:52.699494749 +0000 UTC m=+424.283273084" watchObservedRunningTime="2026-09-16 07:40:52.701302961 +0000 UTC m=+424.285081298" Sep 16 07:40:53.894802 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:53.894763 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:53.977663 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:53.977616 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:54.233444 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:54.233410 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:54.328258 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:54.328223 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:54.425769 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:54.425734 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:54.586224 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:54.586138 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:40:57.659793 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:40:57.659756 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-6fw9g" Sep 16 07:41:03.689521 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:41:03.689486 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6f78fc697-hxrsc" Sep 16 07:41:04.711369 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:41:04.711335 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:41:04.827243 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:41:04.827199 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:41:44.886704 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:41:44.886665 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:41:44.954479 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:41:44.954445 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:42:04.930617 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:04.930577 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:42:07.009430 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:07.009393 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:42:07.739889 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:07.739843 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:42:09.551593 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:09.551549 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:42:09.784717 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:09.784681 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:42:09.940421 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:09.940376 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:42:10.552395 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:10.552364 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:42:11.079791 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:11.079755 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:42:12.016236 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:12.016187 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:42:12.118833 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:12.118800 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:42:13.737142 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:13.737110 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:42:14.115312 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:14.115214 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:42:15.978908 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:15.978873 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:42:16.088689 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:16.088657 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:42:17.535330 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:17.535300 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:42:17.639440 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:17.639390 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:42:20.796643 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:20.796601 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-8wgwz"] Sep 16 07:42:20.799913 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:20.799894 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8wgwz" Sep 16 07:42:20.803432 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:20.803404 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-rz7z2\"" Sep 16 07:42:20.803958 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:20.803933 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:42:20.804376 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:20.804345 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Sep 16 07:42:20.804376 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:20.804361 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Sep 16 07:42:20.804524 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:20.804385 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 16 07:42:20.804524 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:20.804368 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 16 07:42:20.816333 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:20.816302 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-8wgwz"] Sep 16 07:42:20.832255 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:20.832220 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2362570e-47ce-4c9c-bbac-382833e94ae0-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-8wgwz\" (UID: \"2362570e-47ce-4c9c-bbac-382833e94ae0\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8wgwz" Sep 16 07:42:20.832422 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:20.832268 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc85s\" (UniqueName: \"kubernetes.io/projected/2362570e-47ce-4c9c-bbac-382833e94ae0-kube-api-access-nc85s\") pod \"kuadrant-console-plugin-fbdd4f445-8wgwz\" (UID: \"2362570e-47ce-4c9c-bbac-382833e94ae0\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8wgwz" Sep 16 07:42:20.832422 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:20.832338 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2362570e-47ce-4c9c-bbac-382833e94ae0-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-8wgwz\" (UID: \"2362570e-47ce-4c9c-bbac-382833e94ae0\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8wgwz" Sep 16 07:42:20.932926 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:20.932894 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2362570e-47ce-4c9c-bbac-382833e94ae0-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-8wgwz\" (UID: \"2362570e-47ce-4c9c-bbac-382833e94ae0\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8wgwz" Sep 16 07:42:20.933122 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:20.932945 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc85s\" (UniqueName: \"kubernetes.io/projected/2362570e-47ce-4c9c-bbac-382833e94ae0-kube-api-access-nc85s\") pod \"kuadrant-console-plugin-fbdd4f445-8wgwz\" (UID: \"2362570e-47ce-4c9c-bbac-382833e94ae0\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8wgwz" Sep 16 07:42:20.933122 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:20.932970 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2362570e-47ce-4c9c-bbac-382833e94ae0-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-8wgwz\" (UID: \"2362570e-47ce-4c9c-bbac-382833e94ae0\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8wgwz" Sep 16 07:42:20.933267 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:42:20.933148 2552 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Sep 16 07:42:20.933267 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:42:20.933223 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2362570e-47ce-4c9c-bbac-382833e94ae0-plugin-serving-cert podName:2362570e-47ce-4c9c-bbac-382833e94ae0 nodeName:}" failed. No retries permitted until 2026-09-16 07:42:21.433203513 +0000 UTC m=+513.016981831 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/2362570e-47ce-4c9c-bbac-382833e94ae0-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-8wgwz" (UID: "2362570e-47ce-4c9c-bbac-382833e94ae0") : secret "plugin-serving-cert" not found Sep 16 07:42:20.933624 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:20.933601 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2362570e-47ce-4c9c-bbac-382833e94ae0-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-8wgwz\" (UID: \"2362570e-47ce-4c9c-bbac-382833e94ae0\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8wgwz" Sep 16 07:42:20.942599 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:20.942565 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc85s\" (UniqueName: \"kubernetes.io/projected/2362570e-47ce-4c9c-bbac-382833e94ae0-kube-api-access-nc85s\") pod \"kuadrant-console-plugin-fbdd4f445-8wgwz\" (UID: \"2362570e-47ce-4c9c-bbac-382833e94ae0\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8wgwz" Sep 16 07:42:21.330960 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:21.330920 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:42:21.431792 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:21.431747 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:42:21.437707 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:21.437676 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2362570e-47ce-4c9c-bbac-382833e94ae0-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-8wgwz\" (UID: \"2362570e-47ce-4c9c-bbac-382833e94ae0\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8wgwz" Sep 16 07:42:21.437833 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:42:21.437815 2552 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Sep 16 07:42:21.437903 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:42:21.437894 2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2362570e-47ce-4c9c-bbac-382833e94ae0-plugin-serving-cert podName:2362570e-47ce-4c9c-bbac-382833e94ae0 nodeName:}" failed. No retries permitted until 2026-09-16 07:42:22.43787473 +0000 UTC m=+514.021653060 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/2362570e-47ce-4c9c-bbac-382833e94ae0-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-8wgwz" (UID: "2362570e-47ce-4c9c-bbac-382833e94ae0") : secret "plugin-serving-cert" not found Sep 16 07:42:21.627885 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:21.627790 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:42:22.446533 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:22.446475 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2362570e-47ce-4c9c-bbac-382833e94ae0-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-8wgwz\" (UID: \"2362570e-47ce-4c9c-bbac-382833e94ae0\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8wgwz" Sep 16 07:42:22.448950 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:22.448929 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2362570e-47ce-4c9c-bbac-382833e94ae0-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-8wgwz\" (UID: \"2362570e-47ce-4c9c-bbac-382833e94ae0\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8wgwz" Sep 16 07:42:22.629052 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:22.629006 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8wgwz" Sep 16 07:42:22.905687 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:22.905524 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-8wgwz"] Sep 16 07:42:22.908497 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:42:22.908454 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2362570e_47ce_4c9c_bbac_382833e94ae0.slice/crio-f7a4244170fd2df59661753a8b776a09b2e7520972419eae28644ce8f486fa97 WatchSource:0}: Error finding container f7a4244170fd2df59661753a8b776a09b2e7520972419eae28644ce8f486fa97: Status 404 returned error can't find the container with id f7a4244170fd2df59661753a8b776a09b2e7520972419eae28644ce8f486fa97 Sep 16 07:42:23.062585 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:23.062550 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8wgwz" event={"ID":"2362570e-47ce-4c9c-bbac-382833e94ae0","Type":"ContainerStarted","Data":"f7a4244170fd2df59661753a8b776a09b2e7520972419eae28644ce8f486fa97"} Sep 16 07:42:25.148495 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:25.148443 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:42:25.249481 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:25.249442 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:42:25.349552 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:25.349497 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:42:25.410748 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:25.410654 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:42:28.088437 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:28.088399 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8wgwz" event={"ID":"2362570e-47ce-4c9c-bbac-382833e94ae0","Type":"ContainerStarted","Data":"ac0e64393f1ed416ffdacd173ca90126cc2add852f64b12110187c87b26c2ad1"} Sep 16 07:42:28.106964 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:28.106901 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-8wgwz" podStartSLOduration=3.405733717 podStartE2EDuration="8.106885874s" totalImagesPullingTime="4.701152157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:42:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:42:28.105845222 +0000 UTC m=+519.689623559" watchObservedRunningTime="2026-09-16 07:42:28.106885874 +0000 UTC m=+519.690664211" Sep 16 07:42:32.580973 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:32.580940 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s7n7g"] Sep 16 07:42:32.584521 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:32.584496 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s7n7g" Sep 16 07:42:32.589140 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:32.589114 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-xbvtl\"" Sep 16 07:42:32.636455 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:32.636409 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/955c6c96-0681-482f-b967-946e3b2eda9f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-s7n7g\" (UID: \"955c6c96-0681-482f-b967-946e3b2eda9f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s7n7g" Sep 16 07:42:32.636455 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:32.636449 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nbpx\" (UniqueName: \"kubernetes.io/projected/955c6c96-0681-482f-b967-946e3b2eda9f-kube-api-access-6nbpx\") pod \"kuadrant-operator-controller-manager-5769f99d9f-s7n7g\" (UID: \"955c6c96-0681-482f-b967-946e3b2eda9f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s7n7g" Sep 16 07:42:32.656972 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:32.656929 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s7n7g"] Sep 16 07:42:32.737325 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:32.737282 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/955c6c96-0681-482f-b967-946e3b2eda9f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-s7n7g\" (UID: \"955c6c96-0681-482f-b967-946e3b2eda9f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s7n7g" Sep 16 07:42:32.737325 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:32.737325 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nbpx\" (UniqueName: \"kubernetes.io/projected/955c6c96-0681-482f-b967-946e3b2eda9f-kube-api-access-6nbpx\") pod \"kuadrant-operator-controller-manager-5769f99d9f-s7n7g\" (UID: \"955c6c96-0681-482f-b967-946e3b2eda9f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s7n7g" Sep 16 07:42:32.737669 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:32.737651 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/955c6c96-0681-482f-b967-946e3b2eda9f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-s7n7g\" (UID: \"955c6c96-0681-482f-b967-946e3b2eda9f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s7n7g" Sep 16 07:42:32.763848 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:32.763816 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nbpx\" (UniqueName: \"kubernetes.io/projected/955c6c96-0681-482f-b967-946e3b2eda9f-kube-api-access-6nbpx\") pod \"kuadrant-operator-controller-manager-5769f99d9f-s7n7g\" (UID: \"955c6c96-0681-482f-b967-946e3b2eda9f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s7n7g" Sep 16 07:42:32.894591 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:32.894492 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s7n7g" Sep 16 07:42:33.190559 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:33.190526 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s7n7g"] Sep 16 07:42:33.193251 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:42:33.193224 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod955c6c96_0681_482f_b967_946e3b2eda9f.slice/crio-fd082e052e9a5b32742ea230434a19c121dfb10357127daec5a5b031565de84d WatchSource:0}: Error finding container fd082e052e9a5b32742ea230434a19c121dfb10357127daec5a5b031565de84d: Status 404 returned error can't find the container with id fd082e052e9a5b32742ea230434a19c121dfb10357127daec5a5b031565de84d Sep 16 07:42:34.118635 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:34.118549 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s7n7g" event={"ID":"955c6c96-0681-482f-b967-946e3b2eda9f","Type":"ContainerStarted","Data":"fd082e052e9a5b32742ea230434a19c121dfb10357127daec5a5b031565de84d"} Sep 16 07:42:34.146682 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:34.146634 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:42:34.283219 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:34.283142 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:42:36.813741 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:36.813695 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:42:36.923153 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:36.923119 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:42:37.135237 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:37.135152 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s7n7g" event={"ID":"955c6c96-0681-482f-b967-946e3b2eda9f","Type":"ContainerStarted","Data":"dec07385c2e6d7f2a22bfe88fed8e3ec2f4e66935dc3d1fd537a133cee364940"} Sep 16 07:42:37.135385 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:37.135295 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s7n7g" Sep 16 07:42:37.171090 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:37.171030 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s7n7g" podStartSLOduration=1.587900647 podStartE2EDuration="5.171011534s" totalImagesPullingTime="3.583110887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:42:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:42:37.169535542 +0000 UTC m=+528.753313874" watchObservedRunningTime="2026-09-16 07:42:37.171011534 +0000 UTC m=+528.754789887" Sep 16 07:42:40.068741 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:40.068706 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:42:40.199266 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:40.199226 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:42:44.409593 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:44.409505 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:42:44.507040 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:44.507001 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:42:44.606867 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:44.606830 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:42:44.708322 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:44.708286 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:42:45.600329 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:45.600294 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:42:45.666659 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:45.666628 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:42:48.142716 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:48.142685 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s7n7g" Sep 16 07:42:49.440373 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:49.440297 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:42:49.543284 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:49.543245 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:42:53.413930 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:53.413895 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:42:55.398585 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:55.398553 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:42:55.501226 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:55.501184 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:42:56.026237 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:56.026203 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:42:56.106600 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:42:56.106564 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:43:04.455266 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:04.455229 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:43:04.850716 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:04.850617 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:43:06.961633 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:06.961605 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:43:07.126008 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:07.125973 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:43:07.216498 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:07.216406 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:43:07.831132 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:07.831050 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:43:07.920597 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:07.920561 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:43:07.924046 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:07.924004 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-8qjl4"] Sep 16 07:43:07.948072 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:07.948040 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-8qjl4"] Sep 16 07:43:07.948243 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:07.948160 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-8qjl4" Sep 16 07:43:07.950525 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:07.950500 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-j7cgc\"" Sep 16 07:43:08.038482 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:08.038439 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnp6z\" (UniqueName: \"kubernetes.io/projected/68a36295-5914-47f9-9c0f-4b4e1a53da33-kube-api-access-bnp6z\") pod \"authorino-59c974bd48-8qjl4\" (UID: \"68a36295-5914-47f9-9c0f-4b4e1a53da33\") " pod="kuadrant-system/authorino-59c974bd48-8qjl4" Sep 16 07:43:08.139499 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:08.139406 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnp6z\" (UniqueName: \"kubernetes.io/projected/68a36295-5914-47f9-9c0f-4b4e1a53da33-kube-api-access-bnp6z\") pod \"authorino-59c974bd48-8qjl4\" (UID: \"68a36295-5914-47f9-9c0f-4b4e1a53da33\") " pod="kuadrant-system/authorino-59c974bd48-8qjl4" Sep 16 07:43:08.148433 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:08.148399 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnp6z\" (UniqueName: \"kubernetes.io/projected/68a36295-5914-47f9-9c0f-4b4e1a53da33-kube-api-access-bnp6z\") pod \"authorino-59c974bd48-8qjl4\" (UID: \"68a36295-5914-47f9-9c0f-4b4e1a53da33\") " pod="kuadrant-system/authorino-59c974bd48-8qjl4" Sep 16 07:43:08.257197 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:08.257159 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-8qjl4" Sep 16 07:43:08.529167 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:08.528993 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-8qjl4"] Sep 16 07:43:08.532035 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:43:08.531999 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68a36295_5914_47f9_9c0f_4b4e1a53da33.slice/crio-8df3a491be6621a6992724bba9e2dcce93ec09de579bf4a3c13173a5f6065cf8 WatchSource:0}: Error finding container 8df3a491be6621a6992724bba9e2dcce93ec09de579bf4a3c13173a5f6065cf8: Status 404 returned error can't find the container with id 8df3a491be6621a6992724bba9e2dcce93ec09de579bf4a3c13173a5f6065cf8 Sep 16 07:43:09.278250 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:09.278213 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-8qjl4" event={"ID":"68a36295-5914-47f9-9c0f-4b4e1a53da33","Type":"ContainerStarted","Data":"8df3a491be6621a6992724bba9e2dcce93ec09de579bf4a3c13173a5f6065cf8"} Sep 16 07:43:11.289895 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:11.289856 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-8qjl4" event={"ID":"68a36295-5914-47f9-9c0f-4b4e1a53da33","Type":"ContainerStarted","Data":"cbdb630773a1e659e0d079aef6909cf0720924c454b2ccfaffbaf0b9e9239e3e"} Sep 16 07:43:11.304966 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:11.304909 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-8qjl4" podStartSLOduration=2.243830687 podStartE2EDuration="4.30489163s" totalImagesPullingTime="2.061060943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:43:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:43:11.3039234 +0000 UTC m=+562.887701737" watchObservedRunningTime="2026-09-16 07:43:11.30489163 +0000 UTC m=+562.888669967" Sep 16 07:43:23.053904 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:23.053866 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:43:23.147206 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:23.147169 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:43:23.502614 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:23.502583 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:43:35.858258 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:35.858213 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:43:36.064391 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:36.064343 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:43:36.318840 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:36.318804 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-98b8dbd89-cqbzf"] Sep 16 07:43:36.322473 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:36.322450 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-98b8dbd89-cqbzf" Sep 16 07:43:36.324604 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:36.324576 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Sep 16 07:43:36.328337 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:36.328315 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-98b8dbd89-cqbzf"] Sep 16 07:43:36.385394 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:36.385350 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/760137d4-cfff-4b0f-b11a-387bdb7d9fbd-tls-cert\") pod \"authorino-98b8dbd89-cqbzf\" (UID: \"760137d4-cfff-4b0f-b11a-387bdb7d9fbd\") " pod="kuadrant-system/authorino-98b8dbd89-cqbzf" Sep 16 07:43:36.385578 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:36.385409 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnj9d\" (UniqueName: \"kubernetes.io/projected/760137d4-cfff-4b0f-b11a-387bdb7d9fbd-kube-api-access-gnj9d\") pod \"authorino-98b8dbd89-cqbzf\" (UID: \"760137d4-cfff-4b0f-b11a-387bdb7d9fbd\") " pod="kuadrant-system/authorino-98b8dbd89-cqbzf" Sep 16 07:43:36.486079 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:36.486037 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/760137d4-cfff-4b0f-b11a-387bdb7d9fbd-tls-cert\") pod \"authorino-98b8dbd89-cqbzf\" (UID: \"760137d4-cfff-4b0f-b11a-387bdb7d9fbd\") " pod="kuadrant-system/authorino-98b8dbd89-cqbzf" Sep 16 07:43:36.486259 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:36.486124 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnj9d\" (UniqueName: \"kubernetes.io/projected/760137d4-cfff-4b0f-b11a-387bdb7d9fbd-kube-api-access-gnj9d\") pod \"authorino-98b8dbd89-cqbzf\" (UID: \"760137d4-cfff-4b0f-b11a-387bdb7d9fbd\") " pod="kuadrant-system/authorino-98b8dbd89-cqbzf" Sep 16 07:43:36.488486 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:36.488457 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/760137d4-cfff-4b0f-b11a-387bdb7d9fbd-tls-cert\") pod \"authorino-98b8dbd89-cqbzf\" (UID: \"760137d4-cfff-4b0f-b11a-387bdb7d9fbd\") " pod="kuadrant-system/authorino-98b8dbd89-cqbzf" Sep 16 07:43:36.493769 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:36.493744 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnj9d\" (UniqueName: \"kubernetes.io/projected/760137d4-cfff-4b0f-b11a-387bdb7d9fbd-kube-api-access-gnj9d\") pod \"authorino-98b8dbd89-cqbzf\" (UID: \"760137d4-cfff-4b0f-b11a-387bdb7d9fbd\") " pod="kuadrant-system/authorino-98b8dbd89-cqbzf" Sep 16 07:43:36.632987 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:36.632887 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-98b8dbd89-cqbzf" Sep 16 07:43:36.903243 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:36.903120 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-98b8dbd89-cqbzf"] Sep 16 07:43:36.906298 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:43:36.906269 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod760137d4_cfff_4b0f_b11a_387bdb7d9fbd.slice/crio-339aebf5d7a0f7a6bdc5c94818af25946917a043d77b84b2078b5724a380de18 WatchSource:0}: Error finding container 339aebf5d7a0f7a6bdc5c94818af25946917a043d77b84b2078b5724a380de18: Status 404 returned error can't find the container with id 339aebf5d7a0f7a6bdc5c94818af25946917a043d77b84b2078b5724a380de18 Sep 16 07:43:37.405698 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:37.405664 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-98b8dbd89-cqbzf" event={"ID":"760137d4-cfff-4b0f-b11a-387bdb7d9fbd","Type":"ContainerStarted","Data":"339aebf5d7a0f7a6bdc5c94818af25946917a043d77b84b2078b5724a380de18"} Sep 16 07:43:38.416460 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:38.416422 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-98b8dbd89-cqbzf" event={"ID":"760137d4-cfff-4b0f-b11a-387bdb7d9fbd","Type":"ContainerStarted","Data":"f15aa90fd1b8e6d1857823419c86c267a4472dac58b5cd615f0204e2539c9bdd"} Sep 16 07:43:38.432007 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:38.431946 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-98b8dbd89-cqbzf" podStartSLOduration=1.9586131610000002 podStartE2EDuration="2.431928824s" totalImagesPullingTime="473.315663ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:43:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:43:38.429787599 +0000 UTC m=+590.013565940" watchObservedRunningTime="2026-09-16 07:43:38.431928824 +0000 UTC m=+590.015707161" Sep 16 07:43:38.454934 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:38.454896 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-8qjl4"] Sep 16 07:43:38.455112 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:38.455076 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-8qjl4" podUID="68a36295-5914-47f9-9c0f-4b4e1a53da33" containerName="authorino" containerID="cri-o://cbdb630773a1e659e0d079aef6909cf0720924c454b2ccfaffbaf0b9e9239e3e" gracePeriod=30 Sep 16 07:43:38.717590 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:38.717556 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-8qjl4" Sep 16 07:43:38.805628 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:38.805581 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnp6z\" (UniqueName: \"kubernetes.io/projected/68a36295-5914-47f9-9c0f-4b4e1a53da33-kube-api-access-bnp6z\") pod \"68a36295-5914-47f9-9c0f-4b4e1a53da33\" (UID: \"68a36295-5914-47f9-9c0f-4b4e1a53da33\") " Sep 16 07:43:38.807847 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:38.807816 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68a36295-5914-47f9-9c0f-4b4e1a53da33-kube-api-access-bnp6z" (OuterVolumeSpecName: "kube-api-access-bnp6z") pod "68a36295-5914-47f9-9c0f-4b4e1a53da33" (UID: "68a36295-5914-47f9-9c0f-4b4e1a53da33"). InnerVolumeSpecName "kube-api-access-bnp6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:43:38.907111 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:38.907068 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnp6z\" (UniqueName: \"kubernetes.io/projected/68a36295-5914-47f9-9c0f-4b4e1a53da33-kube-api-access-bnp6z\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:43:39.422349 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:39.422313 2552 generic.go:356] "Generic (PLEG): container finished" podID="68a36295-5914-47f9-9c0f-4b4e1a53da33" containerID="cbdb630773a1e659e0d079aef6909cf0720924c454b2ccfaffbaf0b9e9239e3e" exitCode=0 Sep 16 07:43:39.422827 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:39.422361 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-8qjl4" Sep 16 07:43:39.422827 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:39.422400 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-8qjl4" event={"ID":"68a36295-5914-47f9-9c0f-4b4e1a53da33","Type":"ContainerDied","Data":"cbdb630773a1e659e0d079aef6909cf0720924c454b2ccfaffbaf0b9e9239e3e"} Sep 16 07:43:39.422827 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:39.422444 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-8qjl4" event={"ID":"68a36295-5914-47f9-9c0f-4b4e1a53da33","Type":"ContainerDied","Data":"8df3a491be6621a6992724bba9e2dcce93ec09de579bf4a3c13173a5f6065cf8"} Sep 16 07:43:39.422827 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:39.422468 2552 scope.go:117] "RemoveContainer" containerID="cbdb630773a1e659e0d079aef6909cf0720924c454b2ccfaffbaf0b9e9239e3e" Sep 16 07:43:39.430800 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:39.430780 2552 scope.go:117] "RemoveContainer" containerID="cbdb630773a1e659e0d079aef6909cf0720924c454b2ccfaffbaf0b9e9239e3e" Sep 16 07:43:39.431145 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:43:39.431124 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbdb630773a1e659e0d079aef6909cf0720924c454b2ccfaffbaf0b9e9239e3e\": container with ID starting with cbdb630773a1e659e0d079aef6909cf0720924c454b2ccfaffbaf0b9e9239e3e not found: ID does not exist" containerID="cbdb630773a1e659e0d079aef6909cf0720924c454b2ccfaffbaf0b9e9239e3e" Sep 16 07:43:39.431229 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:39.431157 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbdb630773a1e659e0d079aef6909cf0720924c454b2ccfaffbaf0b9e9239e3e"} err="failed to get container status \"cbdb630773a1e659e0d079aef6909cf0720924c454b2ccfaffbaf0b9e9239e3e\": rpc error: code = NotFound desc = could not find container \"cbdb630773a1e659e0d079aef6909cf0720924c454b2ccfaffbaf0b9e9239e3e\": container with ID starting with cbdb630773a1e659e0d079aef6909cf0720924c454b2ccfaffbaf0b9e9239e3e not found: ID does not exist" Sep 16 07:43:39.439214 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:39.439185 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-8qjl4"] Sep 16 07:43:39.441454 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:39.441428 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-8qjl4"] Sep 16 07:43:40.936520 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:40.936484 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68a36295-5914-47f9-9c0f-4b4e1a53da33" path="/var/lib/kubelet/pods/68a36295-5914-47f9-9c0f-4b4e1a53da33/volumes" Sep 16 07:43:42.990815 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:42.990777 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:43:43.085570 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:43.085534 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:43:43.142868 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:43.142830 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:43:43.213758 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:43.213716 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:43:46.085701 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:46.085659 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:43:46.230581 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:43:46.230552 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:44:01.459881 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:01.459842 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:44:01.515618 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:01.515583 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:44:04.899579 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:04.899546 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:44:06.503251 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:06.503213 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:44:10.389161 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:10.389119 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:44:10.489597 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:10.489561 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:44:13.545526 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:13.545430 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:44:13.595934 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:13.595899 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:44:17.669718 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:17.669676 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:44:17.767821 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:17.767782 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:44:19.254910 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:19.254867 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:44:19.356034 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:19.355997 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:44:19.727334 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:19.727298 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:44:19.817804 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:19.817770 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:44:19.921465 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:19.921421 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:44:19.995484 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:19.995390 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:44:29.765175 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:29.765132 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:44:29.884773 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:29.884722 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:44:41.197370 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:41.197317 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:44:46.763525 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:46.763490 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-5bhzn"] Sep 16 07:44:46.764032 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:46.763843 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68a36295-5914-47f9-9c0f-4b4e1a53da33" containerName="authorino" Sep 16 07:44:46.764032 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:46.763860 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="68a36295-5914-47f9-9c0f-4b4e1a53da33" containerName="authorino" Sep 16 07:44:46.764032 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:46.763944 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="68a36295-5914-47f9-9c0f-4b4e1a53da33" containerName="authorino" Sep 16 07:44:46.766968 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:46.766944 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5bhzn" Sep 16 07:44:46.770405 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:46.770374 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 16 07:44:46.770567 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:46.770374 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 16 07:44:46.770567 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:46.770374 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Sep 16 07:44:46.770567 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:46.770389 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-t7r7m\"" Sep 16 07:44:46.779893 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:46.779861 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-5bhzn"] Sep 16 07:44:46.799417 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:46.799387 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:44:46.904562 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:46.904522 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feaebf7f-ae1c-4614-b028-ae3f5b168ecb-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5bhzn\" (UID: \"feaebf7f-ae1c-4614-b028-ae3f5b168ecb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5bhzn" Sep 16 07:44:46.904562 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:46.904562 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feaebf7f-ae1c-4614-b028-ae3f5b168ecb-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5bhzn\" (UID: \"feaebf7f-ae1c-4614-b028-ae3f5b168ecb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5bhzn" Sep 16 07:44:46.904831 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:46.904615 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feaebf7f-ae1c-4614-b028-ae3f5b168ecb-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5bhzn\" (UID: \"feaebf7f-ae1c-4614-b028-ae3f5b168ecb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5bhzn" Sep 16 07:44:46.904831 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:46.904660 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7jbs\" (UniqueName: \"kubernetes.io/projected/feaebf7f-ae1c-4614-b028-ae3f5b168ecb-kube-api-access-w7jbs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5bhzn\" (UID: \"feaebf7f-ae1c-4614-b028-ae3f5b168ecb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5bhzn" Sep 16 07:44:46.904831 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:46.904751 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feaebf7f-ae1c-4614-b028-ae3f5b168ecb-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5bhzn\" (UID: \"feaebf7f-ae1c-4614-b028-ae3f5b168ecb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5bhzn" Sep 16 07:44:46.904831 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:46.904814 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feaebf7f-ae1c-4614-b028-ae3f5b168ecb-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5bhzn\" (UID: \"feaebf7f-ae1c-4614-b028-ae3f5b168ecb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5bhzn" Sep 16 07:44:47.005502 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:47.005463 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feaebf7f-ae1c-4614-b028-ae3f5b168ecb-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5bhzn\" (UID: \"feaebf7f-ae1c-4614-b028-ae3f5b168ecb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5bhzn" Sep 16 07:44:47.005502 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:47.005508 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feaebf7f-ae1c-4614-b028-ae3f5b168ecb-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5bhzn\" (UID: \"feaebf7f-ae1c-4614-b028-ae3f5b168ecb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5bhzn" Sep 16 07:44:47.005732 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:47.005545 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feaebf7f-ae1c-4614-b028-ae3f5b168ecb-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5bhzn\" (UID: \"feaebf7f-ae1c-4614-b028-ae3f5b168ecb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5bhzn" Sep 16 07:44:47.005732 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:47.005562 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feaebf7f-ae1c-4614-b028-ae3f5b168ecb-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5bhzn\" (UID: \"feaebf7f-ae1c-4614-b028-ae3f5b168ecb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5bhzn" Sep 16 07:44:47.005732 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:47.005587 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feaebf7f-ae1c-4614-b028-ae3f5b168ecb-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5bhzn\" (UID: \"feaebf7f-ae1c-4614-b028-ae3f5b168ecb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5bhzn" Sep 16 07:44:47.005732 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:47.005629 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7jbs\" (UniqueName: \"kubernetes.io/projected/feaebf7f-ae1c-4614-b028-ae3f5b168ecb-kube-api-access-w7jbs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5bhzn\" (UID: \"feaebf7f-ae1c-4614-b028-ae3f5b168ecb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5bhzn" Sep 16 07:44:47.006316 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:47.006290 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/feaebf7f-ae1c-4614-b028-ae3f5b168ecb-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5bhzn\" (UID: \"feaebf7f-ae1c-4614-b028-ae3f5b168ecb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5bhzn" Sep 16 07:44:47.006467 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:47.006440 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/feaebf7f-ae1c-4614-b028-ae3f5b168ecb-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5bhzn\" (UID: \"feaebf7f-ae1c-4614-b028-ae3f5b168ecb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5bhzn" Sep 16 07:44:47.006654 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:47.006399 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/feaebf7f-ae1c-4614-b028-ae3f5b168ecb-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5bhzn\" (UID: \"feaebf7f-ae1c-4614-b028-ae3f5b168ecb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5bhzn" Sep 16 07:44:47.011737 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:47.011700 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/feaebf7f-ae1c-4614-b028-ae3f5b168ecb-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5bhzn\" (UID: \"feaebf7f-ae1c-4614-b028-ae3f5b168ecb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5bhzn" Sep 16 07:44:47.012181 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:47.012151 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/feaebf7f-ae1c-4614-b028-ae3f5b168ecb-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5bhzn\" (UID: \"feaebf7f-ae1c-4614-b028-ae3f5b168ecb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5bhzn" Sep 16 07:44:47.014944 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:47.014883 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7jbs\" (UniqueName: \"kubernetes.io/projected/feaebf7f-ae1c-4614-b028-ae3f5b168ecb-kube-api-access-w7jbs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-5bhzn\" (UID: \"feaebf7f-ae1c-4614-b028-ae3f5b168ecb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5bhzn" Sep 16 07:44:47.079395 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:47.079345 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5bhzn" Sep 16 07:44:47.357456 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:47.357429 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-5bhzn"] Sep 16 07:44:47.360037 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:44:47.360006 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeaebf7f_ae1c_4614_b028_ae3f5b168ecb.slice/crio-58967ba7143ba9fc5acc2c5c6901f70210be3be70a9248099fd5e88506f1141c WatchSource:0}: Error finding container 58967ba7143ba9fc5acc2c5c6901f70210be3be70a9248099fd5e88506f1141c: Status 404 returned error can't find the container with id 58967ba7143ba9fc5acc2c5c6901f70210be3be70a9248099fd5e88506f1141c Sep 16 07:44:47.743792 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:47.743747 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5bhzn" event={"ID":"feaebf7f-ae1c-4614-b028-ae3f5b168ecb","Type":"ContainerStarted","Data":"58967ba7143ba9fc5acc2c5c6901f70210be3be70a9248099fd5e88506f1141c"} Sep 16 07:44:48.751223 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:48.751094 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5bhzn" event={"ID":"feaebf7f-ae1c-4614-b028-ae3f5b168ecb","Type":"ContainerStarted","Data":"dbd7435a964acf062a6a53f7317e67229bebbbc932de8d9610c3d5936e14c7e8"} Sep 16 07:44:48.769145 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:48.769069 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5bhzn" podStartSLOduration=1.643125358 podStartE2EDuration="2.769052503s" totalImagesPullingTime="1.125927145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:44:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:44:48.767186503 +0000 UTC m=+660.350964842" watchObservedRunningTime="2026-09-16 07:44:48.769052503 +0000 UTC m=+660.352830841" Sep 16 07:44:51.594322 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:51.594284 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:44:57.080341 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:57.080302 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5bhzn" Sep 16 07:44:57.080802 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:57.080369 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5bhzn" Sep 16 07:44:57.092846 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:57.092819 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5bhzn" Sep 16 07:44:57.803941 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:57.803908 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-5bhzn" Sep 16 07:44:59.592034 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:44:59.591996 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:45:00.132572 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:00.132536 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825745-88658"] Sep 16 07:45:00.136220 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:00.136194 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825745-88658" Sep 16 07:45:00.138825 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:00.138544 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zwk5r\"" Sep 16 07:45:00.141262 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:00.141237 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825745-88658"] Sep 16 07:45:00.147834 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:00.147807 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825745-88658" Sep 16 07:45:00.417327 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:00.417299 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29825745-88658"] Sep 16 07:45:00.419965 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:45:00.419930 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebb41bac_18f5_4c9f_955c_7ab2b7b896eb.slice/crio-f5059575517f2971fb0f7c52b22172302818b7efc2af8f6d3016daa401e39b97 WatchSource:0}: Error finding container f5059575517f2971fb0f7c52b22172302818b7efc2af8f6d3016daa401e39b97: Status 404 returned error can't find the container with id f5059575517f2971fb0f7c52b22172302818b7efc2af8f6d3016daa401e39b97 Sep 16 07:45:00.808173 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:00.808137 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825745-88658" event={"ID":"ebb41bac-18f5-4c9f-955c-7ab2b7b896eb","Type":"ContainerStarted","Data":"f5059575517f2971fb0f7c52b22172302818b7efc2af8f6d3016daa401e39b97"} Sep 16 07:45:02.201860 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:02.201826 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:45:02.301945 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:02.301911 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:45:06.837463 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:06.837427 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825745-88658" event={"ID":"ebb41bac-18f5-4c9f-955c-7ab2b7b896eb","Type":"ContainerStarted","Data":"cf87b66731a489ed213086ce60b113b7b44306c4b086b10685257b4ed342b660"} Sep 16 07:45:06.852402 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:06.852335 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825745-88658" podStartSLOduration=0.894065608 podStartE2EDuration="6.852317697s" totalImagesPullingTime="5.958252089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:45:06.850250735 +0000 UTC m=+678.434029088" watchObservedRunningTime="2026-09-16 07:45:06.852317697 +0000 UTC m=+678.436096034" Sep 16 07:45:07.843930 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:07.843888 2552 generic.go:356] "Generic (PLEG): container finished" podID="ebb41bac-18f5-4c9f-955c-7ab2b7b896eb" containerID="cf87b66731a489ed213086ce60b113b7b44306c4b086b10685257b4ed342b660" exitCode=0 Sep 16 07:45:07.844342 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:07.843978 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825745-88658" event={"ID":"ebb41bac-18f5-4c9f-955c-7ab2b7b896eb","Type":"ContainerDied","Data":"cf87b66731a489ed213086ce60b113b7b44306c4b086b10685257b4ed342b660"} Sep 16 07:45:08.990038 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:08.990004 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825745-88658" Sep 16 07:45:09.854328 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:09.854296 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825745-88658" Sep 16 07:45:09.854529 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:09.854337 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29825745-88658" event={"ID":"ebb41bac-18f5-4c9f-955c-7ab2b7b896eb","Type":"ContainerDied","Data":"f5059575517f2971fb0f7c52b22172302818b7efc2af8f6d3016daa401e39b97"} Sep 16 07:45:09.854529 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:09.854380 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5059575517f2971fb0f7c52b22172302818b7efc2af8f6d3016daa401e39b97" Sep 16 07:45:14.471837 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:14.471795 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftqvhw"] Sep 16 07:45:14.472352 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:14.472173 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebb41bac-18f5-4c9f-955c-7ab2b7b896eb" containerName="cleanup" Sep 16 07:45:14.472352 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:14.472187 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebb41bac-18f5-4c9f-955c-7ab2b7b896eb" containerName="cleanup" Sep 16 07:45:14.472352 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:14.472258 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebb41bac-18f5-4c9f-955c-7ab2b7b896eb" containerName="cleanup" Sep 16 07:45:14.475195 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:14.475174 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftqvhw" Sep 16 07:45:14.477426 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:14.477400 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Sep 16 07:45:14.488051 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:14.488022 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftqvhw"] Sep 16 07:45:14.548093 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:14.548055 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8tr5\" (UniqueName: \"kubernetes.io/projected/23c672c9-120c-4de1-aac1-b99ec52f350f-kube-api-access-w8tr5\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftqvhw\" (UID: \"23c672c9-120c-4de1-aac1-b99ec52f350f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftqvhw" Sep 16 07:45:14.548278 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:14.548152 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23c672c9-120c-4de1-aac1-b99ec52f350f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftqvhw\" (UID: \"23c672c9-120c-4de1-aac1-b99ec52f350f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftqvhw" Sep 16 07:45:14.548278 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:14.548184 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23c672c9-120c-4de1-aac1-b99ec52f350f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftqvhw\" (UID: \"23c672c9-120c-4de1-aac1-b99ec52f350f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftqvhw" Sep 16 07:45:14.548278 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:14.548200 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23c672c9-120c-4de1-aac1-b99ec52f350f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftqvhw\" (UID: \"23c672c9-120c-4de1-aac1-b99ec52f350f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftqvhw" Sep 16 07:45:14.548278 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:14.548218 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23c672c9-120c-4de1-aac1-b99ec52f350f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftqvhw\" (UID: \"23c672c9-120c-4de1-aac1-b99ec52f350f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftqvhw" Sep 16 07:45:14.548278 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:14.548265 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23c672c9-120c-4de1-aac1-b99ec52f350f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftqvhw\" (UID: \"23c672c9-120c-4de1-aac1-b99ec52f350f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftqvhw" Sep 16 07:45:14.587768 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:14.587732 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:45:14.648732 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:14.648690 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23c672c9-120c-4de1-aac1-b99ec52f350f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftqvhw\" (UID: \"23c672c9-120c-4de1-aac1-b99ec52f350f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftqvhw" Sep 16 07:45:14.648952 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:14.648774 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8tr5\" (UniqueName: \"kubernetes.io/projected/23c672c9-120c-4de1-aac1-b99ec52f350f-kube-api-access-w8tr5\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftqvhw\" (UID: \"23c672c9-120c-4de1-aac1-b99ec52f350f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftqvhw" Sep 16 07:45:14.648952 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:14.648820 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23c672c9-120c-4de1-aac1-b99ec52f350f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftqvhw\" (UID: \"23c672c9-120c-4de1-aac1-b99ec52f350f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftqvhw" Sep 16 07:45:14.648952 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:14.648839 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23c672c9-120c-4de1-aac1-b99ec52f350f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftqvhw\" (UID: \"23c672c9-120c-4de1-aac1-b99ec52f350f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftqvhw" Sep 16 07:45:14.648952 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:14.648855 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23c672c9-120c-4de1-aac1-b99ec52f350f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftqvhw\" (UID: \"23c672c9-120c-4de1-aac1-b99ec52f350f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftqvhw" Sep 16 07:45:14.648952 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:14.648875 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23c672c9-120c-4de1-aac1-b99ec52f350f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftqvhw\" (UID: \"23c672c9-120c-4de1-aac1-b99ec52f350f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftqvhw" Sep 16 07:45:14.650518 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:14.650482 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23c672c9-120c-4de1-aac1-b99ec52f350f-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftqvhw\" (UID: \"23c672c9-120c-4de1-aac1-b99ec52f350f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftqvhw" Sep 16 07:45:14.650688 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:14.650522 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23c672c9-120c-4de1-aac1-b99ec52f350f-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftqvhw\" (UID: \"23c672c9-120c-4de1-aac1-b99ec52f350f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftqvhw" Sep 16 07:45:14.650795 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:14.650552 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23c672c9-120c-4de1-aac1-b99ec52f350f-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftqvhw\" (UID: \"23c672c9-120c-4de1-aac1-b99ec52f350f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftqvhw" Sep 16 07:45:14.653141 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:14.653090 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23c672c9-120c-4de1-aac1-b99ec52f350f-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftqvhw\" (UID: \"23c672c9-120c-4de1-aac1-b99ec52f350f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftqvhw" Sep 16 07:45:14.658554 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:14.658524 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23c672c9-120c-4de1-aac1-b99ec52f350f-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftqvhw\" (UID: \"23c672c9-120c-4de1-aac1-b99ec52f350f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftqvhw" Sep 16 07:45:14.659319 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:14.659296 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8tr5\" (UniqueName: \"kubernetes.io/projected/23c672c9-120c-4de1-aac1-b99ec52f350f-kube-api-access-w8tr5\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftqvhw\" (UID: \"23c672c9-120c-4de1-aac1-b99ec52f350f\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftqvhw" Sep 16 07:45:14.785549 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:14.785447 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftqvhw" Sep 16 07:45:15.060614 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:15.060533 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftqvhw"] Sep 16 07:45:15.064168 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:45:15.064130 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23c672c9_120c_4de1_aac1_b99ec52f350f.slice/crio-9f9e0eb15377ed98cbdaa02742848237ccac7a44d1084fe68d44546330788bd3 WatchSource:0}: Error finding container 9f9e0eb15377ed98cbdaa02742848237ccac7a44d1084fe68d44546330788bd3: Status 404 returned error can't find the container with id 9f9e0eb15377ed98cbdaa02742848237ccac7a44d1084fe68d44546330788bd3 Sep 16 07:45:15.065662 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:15.065636 2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 07:45:15.881464 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:15.881427 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftqvhw" event={"ID":"23c672c9-120c-4de1-aac1-b99ec52f350f","Type":"ContainerStarted","Data":"9568e9d68a74340ff97042e6d848fafc956c923221aaf54862596c2a15db2959"} Sep 16 07:45:15.881904 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:15.881482 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftqvhw" event={"ID":"23c672c9-120c-4de1-aac1-b99ec52f350f","Type":"ContainerStarted","Data":"9f9e0eb15377ed98cbdaa02742848237ccac7a44d1084fe68d44546330788bd3"} Sep 16 07:45:15.900131 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:15.900044 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftqvhw" podStartSLOduration=1.6918077280000001 podStartE2EDuration="1.900024896s" totalImagesPullingTime="208.217168ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:45:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:45:15.898085215 +0000 UTC m=+687.481863553" watchObservedRunningTime="2026-09-16 07:45:15.900024896 +0000 UTC m=+687.483803245" Sep 16 07:45:24.786291 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:24.786248 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftqvhw" Sep 16 07:45:24.786776 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:24.786329 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftqvhw" Sep 16 07:45:24.798926 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:24.798892 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftqvhw" Sep 16 07:45:24.937849 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:24.937822 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftqvhw" Sep 16 07:45:31.288977 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:31.288944 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-5csgx"] Sep 16 07:45:31.294893 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:31.294866 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5csgx" Sep 16 07:45:31.297886 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:31.297854 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Sep 16 07:45:31.305638 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:31.305601 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-5csgx"] Sep 16 07:45:31.360970 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:31.360934 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:45:31.399722 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:31.399682 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/154478c5-73f4-4e25-bf30-b75b3d0fe576-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5csgx\" (UID: \"154478c5-73f4-4e25-bf30-b75b3d0fe576\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5csgx" Sep 16 07:45:31.399722 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:31.399721 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/154478c5-73f4-4e25-bf30-b75b3d0fe576-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5csgx\" (UID: \"154478c5-73f4-4e25-bf30-b75b3d0fe576\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5csgx" Sep 16 07:45:31.399924 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:31.399762 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/154478c5-73f4-4e25-bf30-b75b3d0fe576-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5csgx\" (UID: \"154478c5-73f4-4e25-bf30-b75b3d0fe576\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5csgx" Sep 16 07:45:31.399924 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:31.399787 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pknlv\" (UniqueName: \"kubernetes.io/projected/154478c5-73f4-4e25-bf30-b75b3d0fe576-kube-api-access-pknlv\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5csgx\" (UID: \"154478c5-73f4-4e25-bf30-b75b3d0fe576\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5csgx" Sep 16 07:45:31.399924 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:31.399811 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/154478c5-73f4-4e25-bf30-b75b3d0fe576-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5csgx\" (UID: \"154478c5-73f4-4e25-bf30-b75b3d0fe576\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5csgx" Sep 16 07:45:31.399924 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:31.399826 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/154478c5-73f4-4e25-bf30-b75b3d0fe576-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5csgx\" (UID: \"154478c5-73f4-4e25-bf30-b75b3d0fe576\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5csgx" Sep 16 07:45:31.501019 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:31.500970 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/154478c5-73f4-4e25-bf30-b75b3d0fe576-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5csgx\" (UID: \"154478c5-73f4-4e25-bf30-b75b3d0fe576\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5csgx" Sep 16 07:45:31.501019 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:31.501020 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/154478c5-73f4-4e25-bf30-b75b3d0fe576-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5csgx\" (UID: \"154478c5-73f4-4e25-bf30-b75b3d0fe576\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5csgx" Sep 16 07:45:31.501271 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:31.501061 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/154478c5-73f4-4e25-bf30-b75b3d0fe576-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5csgx\" (UID: \"154478c5-73f4-4e25-bf30-b75b3d0fe576\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5csgx" Sep 16 07:45:31.501271 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:31.501087 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pknlv\" (UniqueName: \"kubernetes.io/projected/154478c5-73f4-4e25-bf30-b75b3d0fe576-kube-api-access-pknlv\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5csgx\" (UID: \"154478c5-73f4-4e25-bf30-b75b3d0fe576\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5csgx" Sep 16 07:45:31.501271 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:31.501145 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/154478c5-73f4-4e25-bf30-b75b3d0fe576-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5csgx\" (UID: \"154478c5-73f4-4e25-bf30-b75b3d0fe576\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5csgx" Sep 16 07:45:31.501271 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:31.501169 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/154478c5-73f4-4e25-bf30-b75b3d0fe576-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5csgx\" (UID: \"154478c5-73f4-4e25-bf30-b75b3d0fe576\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5csgx" Sep 16 07:45:31.501497 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:31.501473 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/154478c5-73f4-4e25-bf30-b75b3d0fe576-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5csgx\" (UID: \"154478c5-73f4-4e25-bf30-b75b3d0fe576\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5csgx" Sep 16 07:45:31.501561 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:31.501524 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/154478c5-73f4-4e25-bf30-b75b3d0fe576-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5csgx\" (UID: \"154478c5-73f4-4e25-bf30-b75b3d0fe576\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5csgx" Sep 16 07:45:31.501635 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:31.501615 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/154478c5-73f4-4e25-bf30-b75b3d0fe576-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5csgx\" (UID: \"154478c5-73f4-4e25-bf30-b75b3d0fe576\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5csgx" Sep 16 07:45:31.503455 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:31.503434 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/154478c5-73f4-4e25-bf30-b75b3d0fe576-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5csgx\" (UID: \"154478c5-73f4-4e25-bf30-b75b3d0fe576\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5csgx" Sep 16 07:45:31.503594 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:31.503576 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/154478c5-73f4-4e25-bf30-b75b3d0fe576-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5csgx\" (UID: \"154478c5-73f4-4e25-bf30-b75b3d0fe576\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5csgx" Sep 16 07:45:31.511078 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:31.511054 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pknlv\" (UniqueName: \"kubernetes.io/projected/154478c5-73f4-4e25-bf30-b75b3d0fe576-kube-api-access-pknlv\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-5csgx\" (UID: \"154478c5-73f4-4e25-bf30-b75b3d0fe576\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5csgx" Sep 16 07:45:31.614304 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:31.614204 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5csgx" Sep 16 07:45:31.893965 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:31.893936 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-5csgx"] Sep 16 07:45:31.896479 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:45:31.896450 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod154478c5_73f4_4e25_bf30_b75b3d0fe576.slice/crio-444a77cca2894c9b5e19662d86fabbc2aacaea26791fed36fec9750b8d460698 WatchSource:0}: Error finding container 444a77cca2894c9b5e19662d86fabbc2aacaea26791fed36fec9750b8d460698: Status 404 returned error can't find the container with id 444a77cca2894c9b5e19662d86fabbc2aacaea26791fed36fec9750b8d460698 Sep 16 07:45:31.959376 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:31.959339 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5csgx" event={"ID":"154478c5-73f4-4e25-bf30-b75b3d0fe576","Type":"ContainerStarted","Data":"444a77cca2894c9b5e19662d86fabbc2aacaea26791fed36fec9750b8d460698"} Sep 16 07:45:32.965261 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:32.965220 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5csgx" event={"ID":"154478c5-73f4-4e25-bf30-b75b3d0fe576","Type":"ContainerStarted","Data":"1adc9edf11dc8a10cb0007a7efa6c538a4f07eee1c673f84d9c2b7673035e3b7"} Sep 16 07:45:32.983600 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:32.983537 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5csgx" podStartSLOduration=1.793044656 podStartE2EDuration="1.983520914s" totalImagesPullingTime="190.476258ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:45:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:45:32.982341804 +0000 UTC m=+704.566120145" watchObservedRunningTime="2026-09-16 07:45:32.983520914 +0000 UTC m=+704.567299252" Sep 16 07:45:41.614698 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:41.614606 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5csgx" Sep 16 07:45:41.615078 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:41.614948 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5csgx" Sep 16 07:45:41.627371 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:41.627340 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5csgx" Sep 16 07:45:42.026350 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:42.020734 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-5csgx" Sep 16 07:45:44.472929 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:44.472893 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-5hs67"] Sep 16 07:45:44.476335 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:44.476311 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5hs67" Sep 16 07:45:44.479229 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:44.479203 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Sep 16 07:45:44.486415 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:44.486382 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-5hs67"] Sep 16 07:45:44.511767 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:44.511737 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:45:44.520862 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:44.520830 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6216f4c2-b225-43d5-a1a9-3e42ed6d7809-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5hs67\" (UID: \"6216f4c2-b225-43d5-a1a9-3e42ed6d7809\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5hs67" Sep 16 07:45:44.520990 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:44.520869 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6216f4c2-b225-43d5-a1a9-3e42ed6d7809-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5hs67\" (UID: \"6216f4c2-b225-43d5-a1a9-3e42ed6d7809\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5hs67" Sep 16 07:45:44.520990 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:44.520927 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6216f4c2-b225-43d5-a1a9-3e42ed6d7809-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5hs67\" (UID: \"6216f4c2-b225-43d5-a1a9-3e42ed6d7809\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5hs67" Sep 16 07:45:44.521071 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:44.520991 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vctxt\" (UniqueName: \"kubernetes.io/projected/6216f4c2-b225-43d5-a1a9-3e42ed6d7809-kube-api-access-vctxt\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5hs67\" (UID: \"6216f4c2-b225-43d5-a1a9-3e42ed6d7809\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5hs67" Sep 16 07:45:44.521071 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:44.521019 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6216f4c2-b225-43d5-a1a9-3e42ed6d7809-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5hs67\" (UID: \"6216f4c2-b225-43d5-a1a9-3e42ed6d7809\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5hs67" Sep 16 07:45:44.521169 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:44.521085 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6216f4c2-b225-43d5-a1a9-3e42ed6d7809-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5hs67\" (UID: \"6216f4c2-b225-43d5-a1a9-3e42ed6d7809\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5hs67" Sep 16 07:45:44.622475 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:44.622437 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6216f4c2-b225-43d5-a1a9-3e42ed6d7809-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5hs67\" (UID: \"6216f4c2-b225-43d5-a1a9-3e42ed6d7809\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5hs67" Sep 16 07:45:44.622676 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:44.622501 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vctxt\" (UniqueName: \"kubernetes.io/projected/6216f4c2-b225-43d5-a1a9-3e42ed6d7809-kube-api-access-vctxt\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5hs67\" (UID: \"6216f4c2-b225-43d5-a1a9-3e42ed6d7809\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5hs67" Sep 16 07:45:44.622676 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:44.622526 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6216f4c2-b225-43d5-a1a9-3e42ed6d7809-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5hs67\" (UID: \"6216f4c2-b225-43d5-a1a9-3e42ed6d7809\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5hs67" Sep 16 07:45:44.622676 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:44.622549 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6216f4c2-b225-43d5-a1a9-3e42ed6d7809-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5hs67\" (UID: \"6216f4c2-b225-43d5-a1a9-3e42ed6d7809\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5hs67" Sep 16 07:45:44.622676 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:44.622575 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6216f4c2-b225-43d5-a1a9-3e42ed6d7809-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5hs67\" (UID: \"6216f4c2-b225-43d5-a1a9-3e42ed6d7809\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5hs67" Sep 16 07:45:44.622676 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:44.622594 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6216f4c2-b225-43d5-a1a9-3e42ed6d7809-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5hs67\" (UID: \"6216f4c2-b225-43d5-a1a9-3e42ed6d7809\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5hs67" Sep 16 07:45:44.623002 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:44.622968 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6216f4c2-b225-43d5-a1a9-3e42ed6d7809-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5hs67\" (UID: \"6216f4c2-b225-43d5-a1a9-3e42ed6d7809\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5hs67" Sep 16 07:45:44.623086 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:44.623010 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6216f4c2-b225-43d5-a1a9-3e42ed6d7809-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5hs67\" (UID: \"6216f4c2-b225-43d5-a1a9-3e42ed6d7809\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5hs67" Sep 16 07:45:44.623086 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:44.623026 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6216f4c2-b225-43d5-a1a9-3e42ed6d7809-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5hs67\" (UID: \"6216f4c2-b225-43d5-a1a9-3e42ed6d7809\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5hs67" Sep 16 07:45:44.624868 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:44.624841 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6216f4c2-b225-43d5-a1a9-3e42ed6d7809-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5hs67\" (UID: \"6216f4c2-b225-43d5-a1a9-3e42ed6d7809\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5hs67" Sep 16 07:45:44.625007 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:44.624990 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6216f4c2-b225-43d5-a1a9-3e42ed6d7809-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5hs67\" (UID: \"6216f4c2-b225-43d5-a1a9-3e42ed6d7809\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5hs67" Sep 16 07:45:44.636073 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:44.636045 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vctxt\" (UniqueName: \"kubernetes.io/projected/6216f4c2-b225-43d5-a1a9-3e42ed6d7809-kube-api-access-vctxt\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-5hs67\" (UID: \"6216f4c2-b225-43d5-a1a9-3e42ed6d7809\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5hs67" Sep 16 07:45:44.786967 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:44.786877 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5hs67" Sep 16 07:45:45.065762 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:45.065733 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-5hs67"] Sep 16 07:45:45.067994 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:45:45.067963 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6216f4c2_b225_43d5_a1a9_3e42ed6d7809.slice/crio-59b75ca709ea77c364bbf50be43bdd659460b8eec5abcaefe9fa54fa37c1858a WatchSource:0}: Error finding container 59b75ca709ea77c364bbf50be43bdd659460b8eec5abcaefe9fa54fa37c1858a: Status 404 returned error can't find the container with id 59b75ca709ea77c364bbf50be43bdd659460b8eec5abcaefe9fa54fa37c1858a Sep 16 07:45:46.028505 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:46.028467 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5hs67" event={"ID":"6216f4c2-b225-43d5-a1a9-3e42ed6d7809","Type":"ContainerStarted","Data":"88569fd1ee4487776d3ae7000f75f2bcf9e8023e20d2cec7e716f880a7997bdd"} Sep 16 07:45:46.028936 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:46.028511 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5hs67" event={"ID":"6216f4c2-b225-43d5-a1a9-3e42ed6d7809","Type":"ContainerStarted","Data":"59b75ca709ea77c364bbf50be43bdd659460b8eec5abcaefe9fa54fa37c1858a"} Sep 16 07:45:46.047223 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:46.047168 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5hs67" podStartSLOduration=1.854353668 podStartE2EDuration="2.047149192s" totalImagesPullingTime="192.795524ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:45:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:45:46.046190112 +0000 UTC m=+717.629968451" watchObservedRunningTime="2026-09-16 07:45:46.047149192 +0000 UTC m=+717.630927529" Sep 16 07:45:54.787138 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:54.787076 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5hs67" Sep 16 07:45:54.787570 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:54.787150 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5hs67" Sep 16 07:45:54.800190 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:54.800158 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5hs67" Sep 16 07:45:55.085373 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:45:55.085274 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-5hs67" Sep 16 07:46:40.673172 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:46:40.673130 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-98b8dbd89-cqbzf"] Sep 16 07:46:40.673743 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:46:40.673348 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-98b8dbd89-cqbzf" podUID="760137d4-cfff-4b0f-b11a-387bdb7d9fbd" containerName="authorino" containerID="cri-o://f15aa90fd1b8e6d1857823419c86c267a4472dac58b5cd615f0204e2539c9bdd" gracePeriod=30 Sep 16 07:46:40.916867 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:46:40.916839 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-98b8dbd89-cqbzf" Sep 16 07:46:41.027011 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:46:41.026908 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/760137d4-cfff-4b0f-b11a-387bdb7d9fbd-tls-cert\") pod \"760137d4-cfff-4b0f-b11a-387bdb7d9fbd\" (UID: \"760137d4-cfff-4b0f-b11a-387bdb7d9fbd\") " Sep 16 07:46:41.027011 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:46:41.026992 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnj9d\" (UniqueName: \"kubernetes.io/projected/760137d4-cfff-4b0f-b11a-387bdb7d9fbd-kube-api-access-gnj9d\") pod \"760137d4-cfff-4b0f-b11a-387bdb7d9fbd\" (UID: \"760137d4-cfff-4b0f-b11a-387bdb7d9fbd\") " Sep 16 07:46:41.029106 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:46:41.029067 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/760137d4-cfff-4b0f-b11a-387bdb7d9fbd-kube-api-access-gnj9d" (OuterVolumeSpecName: "kube-api-access-gnj9d") pod "760137d4-cfff-4b0f-b11a-387bdb7d9fbd" (UID: "760137d4-cfff-4b0f-b11a-387bdb7d9fbd"). InnerVolumeSpecName "kube-api-access-gnj9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:46:41.038322 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:46:41.038289 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/760137d4-cfff-4b0f-b11a-387bdb7d9fbd-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "760137d4-cfff-4b0f-b11a-387bdb7d9fbd" (UID: "760137d4-cfff-4b0f-b11a-387bdb7d9fbd"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:46:41.128480 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:46:41.128441 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnj9d\" (UniqueName: \"kubernetes.io/projected/760137d4-cfff-4b0f-b11a-387bdb7d9fbd-kube-api-access-gnj9d\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:46:41.128480 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:46:41.128474 2552 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/760137d4-cfff-4b0f-b11a-387bdb7d9fbd-tls-cert\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:46:41.286158 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:46:41.286047 2552 generic.go:356] "Generic (PLEG): container finished" podID="760137d4-cfff-4b0f-b11a-387bdb7d9fbd" containerID="f15aa90fd1b8e6d1857823419c86c267a4472dac58b5cd615f0204e2539c9bdd" exitCode=0 Sep 16 07:46:41.286158 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:46:41.286087 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-98b8dbd89-cqbzf" Sep 16 07:46:41.286158 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:46:41.286137 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-98b8dbd89-cqbzf" event={"ID":"760137d4-cfff-4b0f-b11a-387bdb7d9fbd","Type":"ContainerDied","Data":"f15aa90fd1b8e6d1857823419c86c267a4472dac58b5cd615f0204e2539c9bdd"} Sep 16 07:46:41.286401 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:46:41.286184 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-98b8dbd89-cqbzf" event={"ID":"760137d4-cfff-4b0f-b11a-387bdb7d9fbd","Type":"ContainerDied","Data":"339aebf5d7a0f7a6bdc5c94818af25946917a043d77b84b2078b5724a380de18"} Sep 16 07:46:41.286401 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:46:41.286205 2552 scope.go:117] "RemoveContainer" containerID="f15aa90fd1b8e6d1857823419c86c267a4472dac58b5cd615f0204e2539c9bdd" Sep 16 07:46:41.295508 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:46:41.295486 2552 scope.go:117] "RemoveContainer" containerID="f15aa90fd1b8e6d1857823419c86c267a4472dac58b5cd615f0204e2539c9bdd" Sep 16 07:46:41.295857 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:46:41.295826 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f15aa90fd1b8e6d1857823419c86c267a4472dac58b5cd615f0204e2539c9bdd\": container with ID starting with f15aa90fd1b8e6d1857823419c86c267a4472dac58b5cd615f0204e2539c9bdd not found: ID does not exist" containerID="f15aa90fd1b8e6d1857823419c86c267a4472dac58b5cd615f0204e2539c9bdd" Sep 16 07:46:41.295966 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:46:41.295867 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f15aa90fd1b8e6d1857823419c86c267a4472dac58b5cd615f0204e2539c9bdd"} err="failed to get container status \"f15aa90fd1b8e6d1857823419c86c267a4472dac58b5cd615f0204e2539c9bdd\": rpc error: code = NotFound desc = could not find container \"f15aa90fd1b8e6d1857823419c86c267a4472dac58b5cd615f0204e2539c9bdd\": container with ID starting with f15aa90fd1b8e6d1857823419c86c267a4472dac58b5cd615f0204e2539c9bdd not found: ID does not exist" Sep 16 07:46:41.296640 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:46:41.296615 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:46:41.309530 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:46:41.309497 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-98b8dbd89-cqbzf"] Sep 16 07:46:41.316761 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:46:41.316730 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-98b8dbd89-cqbzf"] Sep 16 07:46:41.373297 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:46:41.373228 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:46:42.493381 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:46:42.493338 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:46:42.630819 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:46:42.630779 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:46:42.937001 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:46:42.936974 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="760137d4-cfff-4b0f-b11a-387bdb7d9fbd" path="/var/lib/kubelet/pods/760137d4-cfff-4b0f-b11a-387bdb7d9fbd/volumes" Sep 16 07:47:00.235961 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:00.235923 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:00.306056 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:00.306011 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:00.370749 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:00.370713 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:00.585871 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:00.585787 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:00.686520 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:00.686487 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:00.789632 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:00.789589 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:01.045595 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:01.045534 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:01.280466 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:01.280423 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:01.555464 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:01.555419 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:01.650605 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:01.650519 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:02.004401 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:02.004363 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:04.030340 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:04.030303 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:04.136585 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:04.136548 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:04.232607 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:04.232572 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:05.306136 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:05.306087 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:05.416770 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:05.416733 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:05.505773 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:05.505733 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:05.808441 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:05.808408 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:05.900745 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:05.900706 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:05.982757 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:05.982724 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-c9020d-77dd676546-fzccc"] Sep 16 07:47:05.983088 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:05.983076 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="760137d4-cfff-4b0f-b11a-387bdb7d9fbd" containerName="authorino" Sep 16 07:47:05.983154 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:05.983090 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="760137d4-cfff-4b0f-b11a-387bdb7d9fbd" containerName="authorino" Sep 16 07:47:05.983189 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:05.983169 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="760137d4-cfff-4b0f-b11a-387bdb7d9fbd" containerName="authorino" Sep 16 07:47:05.987845 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:05.987820 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c9020d-77dd676546-fzccc" Sep 16 07:47:05.990461 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:05.990434 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-c9020d\"" Sep 16 07:47:05.990713 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:05.990696 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-c9020d-dockercfg-tbl6d\"" Sep 16 07:47:06.003268 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:06.003235 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-c9020d-77dd676546-fzccc"] Sep 16 07:47:06.155461 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:06.155365 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7caee0e-a84b-4c3f-8bf1-3b0ddc2ec2b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-c9020d-77dd676546-fzccc\" (UID: \"f7caee0e-a84b-4c3f-8bf1-3b0ddc2ec2b0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c9020d-77dd676546-fzccc" Sep 16 07:47:06.155461 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:06.155414 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s47w9\" (UniqueName: \"kubernetes.io/projected/f7caee0e-a84b-4c3f-8bf1-3b0ddc2ec2b0-kube-api-access-s47w9\") pod \"payload-pre-processing-e2e-worker-w0-c9020d-77dd676546-fzccc\" (UID: \"f7caee0e-a84b-4c3f-8bf1-3b0ddc2ec2b0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c9020d-77dd676546-fzccc" Sep 16 07:47:06.256894 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:06.256859 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s47w9\" (UniqueName: \"kubernetes.io/projected/f7caee0e-a84b-4c3f-8bf1-3b0ddc2ec2b0-kube-api-access-s47w9\") pod \"payload-pre-processing-e2e-worker-w0-c9020d-77dd676546-fzccc\" (UID: \"f7caee0e-a84b-4c3f-8bf1-3b0ddc2ec2b0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c9020d-77dd676546-fzccc" Sep 16 07:47:06.257061 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:06.256999 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7caee0e-a84b-4c3f-8bf1-3b0ddc2ec2b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-c9020d-77dd676546-fzccc\" (UID: \"f7caee0e-a84b-4c3f-8bf1-3b0ddc2ec2b0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c9020d-77dd676546-fzccc" Sep 16 07:47:06.257794 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:06.257770 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7caee0e-a84b-4c3f-8bf1-3b0ddc2ec2b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-c9020d-77dd676546-fzccc\" (UID: \"f7caee0e-a84b-4c3f-8bf1-3b0ddc2ec2b0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c9020d-77dd676546-fzccc" Sep 16 07:47:06.264840 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:06.264803 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s47w9\" (UniqueName: \"kubernetes.io/projected/f7caee0e-a84b-4c3f-8bf1-3b0ddc2ec2b0-kube-api-access-s47w9\") pod \"payload-pre-processing-e2e-worker-w0-c9020d-77dd676546-fzccc\" (UID: \"f7caee0e-a84b-4c3f-8bf1-3b0ddc2ec2b0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c9020d-77dd676546-fzccc" Sep 16 07:47:06.286023 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:06.285988 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:06.298182 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:06.298151 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c9020d-77dd676546-fzccc" Sep 16 07:47:06.576294 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:06.576251 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-c9020d-77dd676546-fzccc"] Sep 16 07:47:06.579846 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:47:06.579817 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7caee0e_a84b_4c3f_8bf1_3b0ddc2ec2b0.slice/crio-2a1456f1e47669bf82e52f4bc8b7b028315b95b786ad80675f181668b327258a WatchSource:0}: Error finding container 2a1456f1e47669bf82e52f4bc8b7b028315b95b786ad80675f181668b327258a: Status 404 returned error can't find the container with id 2a1456f1e47669bf82e52f4bc8b7b028315b95b786ad80675f181668b327258a Sep 16 07:47:07.392607 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:07.392558 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:07.413847 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:07.413767 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c9020d-77dd676546-fzccc" event={"ID":"f7caee0e-a84b-4c3f-8bf1-3b0ddc2ec2b0","Type":"ContainerStarted","Data":"2a1456f1e47669bf82e52f4bc8b7b028315b95b786ad80675f181668b327258a"} Sep 16 07:47:07.454422 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:07.454370 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:07.690532 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:07.690498 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:08.043692 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:08.043548 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:08.149499 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:08.149462 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:08.501046 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:08.501012 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:08.579082 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:08.579028 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:08.679337 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:08.679295 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:08.885827 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:08.885694 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:08.994366 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:08.994332 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:09.087294 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.087259 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:09.185311 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.185272 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-fc27a5-54dc7c7d8b-5h7m6"] Sep 16 07:47:09.188716 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.188695 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fc27a5-54dc7c7d8b-5h7m6" Sep 16 07:47:09.198749 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.198724 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-fc27a5\"" Sep 16 07:47:09.198937 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.198723 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:09.199898 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.199876 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-fc27a5-dockercfg-tdn5v\"" Sep 16 07:47:09.212263 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.212227 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-fc27a5-54dc7c7d8b-5h7m6"] Sep 16 07:47:09.226464 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.226427 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l"] Sep 16 07:47:09.230155 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.230126 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" Sep 16 07:47:09.233168 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.233143 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-b60434-openshift-default-dockercfg-njft4\"" Sep 16 07:47:09.233334 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.233235 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 16 07:47:09.243288 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.243254 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l"] Sep 16 07:47:09.284698 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.284257 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcb8b77a-fd48-4204-890d-b15f2e4ea7b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-fc27a5-54dc7c7d8b-5h7m6\" (UID: \"dcb8b77a-fd48-4204-890d-b15f2e4ea7b3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fc27a5-54dc7c7d8b-5h7m6" Sep 16 07:47:09.284698 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.284425 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbb7n\" (UniqueName: \"kubernetes.io/projected/dcb8b77a-fd48-4204-890d-b15f2e4ea7b3-kube-api-access-qbb7n\") pod \"payload-pre-processing-e2e-worker-w4-fc27a5-54dc7c7d8b-5h7m6\" (UID: \"dcb8b77a-fd48-4204-890d-b15f2e4ea7b3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fc27a5-54dc7c7d8b-5h7m6" Sep 16 07:47:09.286256 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.286231 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-fc27a5-6bf7fbd5ff-skp6c"] Sep 16 07:47:09.289583 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.289565 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-fc27a5-6bf7fbd5ff-skp6c" Sep 16 07:47:09.294561 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.294530 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:09.302647 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.302616 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-fc27a5-6bf7fbd5ff-skp6c"] Sep 16 07:47:09.385867 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.385824 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56ccdf02-0887-408b-bee7-26f99d1a1c7b-istio-token\") pod \"e2e-autores-b60434-openshift-default-64fbf588d4-s956l\" (UID: \"56ccdf02-0887-408b-bee7-26f99d1a1c7b\") " pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" Sep 16 07:47:09.385867 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.385877 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tdhx\" (UniqueName: \"kubernetes.io/projected/56ccdf02-0887-408b-bee7-26f99d1a1c7b-kube-api-access-6tdhx\") pod \"e2e-autores-b60434-openshift-default-64fbf588d4-s956l\" (UID: \"56ccdf02-0887-408b-bee7-26f99d1a1c7b\") " pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" Sep 16 07:47:09.386090 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.385943 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcb8b77a-fd48-4204-890d-b15f2e4ea7b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-fc27a5-54dc7c7d8b-5h7m6\" (UID: \"dcb8b77a-fd48-4204-890d-b15f2e4ea7b3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fc27a5-54dc7c7d8b-5h7m6" Sep 16 07:47:09.386090 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.385972 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ccdf02-0887-408b-bee7-26f99d1a1c7b-credential-socket\") pod \"e2e-autores-b60434-openshift-default-64fbf588d4-s956l\" (UID: \"56ccdf02-0887-408b-bee7-26f99d1a1c7b\") " pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" Sep 16 07:47:09.386090 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.386002 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ccdf02-0887-408b-bee7-26f99d1a1c7b-workload-socket\") pod \"e2e-autores-b60434-openshift-default-64fbf588d4-s956l\" (UID: \"56ccdf02-0887-408b-bee7-26f99d1a1c7b\") " pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" Sep 16 07:47:09.386090 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.386019 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56ccdf02-0887-408b-bee7-26f99d1a1c7b-istio-data\") pod \"e2e-autores-b60434-openshift-default-64fbf588d4-s956l\" (UID: \"56ccdf02-0887-408b-bee7-26f99d1a1c7b\") " pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" Sep 16 07:47:09.386090 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.386048 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd352ca8-49ca-4482-b54c-499885f04544-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-fc27a5-6bf7fbd5ff-skp6c\" (UID: \"fd352ca8-49ca-4482-b54c-499885f04544\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-fc27a5-6bf7fbd5ff-skp6c" Sep 16 07:47:09.386090 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.386086 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6xqx\" (UniqueName: \"kubernetes.io/projected/fd352ca8-49ca-4482-b54c-499885f04544-kube-api-access-f6xqx\") pod \"payload-processing-e2e-worker-w4-fc27a5-6bf7fbd5ff-skp6c\" (UID: \"fd352ca8-49ca-4482-b54c-499885f04544\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-fc27a5-6bf7fbd5ff-skp6c" Sep 16 07:47:09.386343 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.386144 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbb7n\" (UniqueName: \"kubernetes.io/projected/dcb8b77a-fd48-4204-890d-b15f2e4ea7b3-kube-api-access-qbb7n\") pod \"payload-pre-processing-e2e-worker-w4-fc27a5-54dc7c7d8b-5h7m6\" (UID: \"dcb8b77a-fd48-4204-890d-b15f2e4ea7b3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fc27a5-54dc7c7d8b-5h7m6" Sep 16 07:47:09.386343 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.386256 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56ccdf02-0887-408b-bee7-26f99d1a1c7b-workload-certs\") pod \"e2e-autores-b60434-openshift-default-64fbf588d4-s956l\" (UID: \"56ccdf02-0887-408b-bee7-26f99d1a1c7b\") " pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" Sep 16 07:47:09.386419 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.386351 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56ccdf02-0887-408b-bee7-26f99d1a1c7b-istio-podinfo\") pod \"e2e-autores-b60434-openshift-default-64fbf588d4-s956l\" (UID: \"56ccdf02-0887-408b-bee7-26f99d1a1c7b\") " pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" Sep 16 07:47:09.386419 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.386376 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56ccdf02-0887-408b-bee7-26f99d1a1c7b-istio-envoy\") pod \"e2e-autores-b60434-openshift-default-64fbf588d4-s956l\" (UID: \"56ccdf02-0887-408b-bee7-26f99d1a1c7b\") " pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" Sep 16 07:47:09.386526 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.386460 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56ccdf02-0887-408b-bee7-26f99d1a1c7b-istiod-ca-cert\") pod \"e2e-autores-b60434-openshift-default-64fbf588d4-s956l\" (UID: \"56ccdf02-0887-408b-bee7-26f99d1a1c7b\") " pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" Sep 16 07:47:09.386693 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.386672 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcb8b77a-fd48-4204-890d-b15f2e4ea7b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-fc27a5-54dc7c7d8b-5h7m6\" (UID: \"dcb8b77a-fd48-4204-890d-b15f2e4ea7b3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fc27a5-54dc7c7d8b-5h7m6" Sep 16 07:47:09.398548 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.398515 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbb7n\" (UniqueName: \"kubernetes.io/projected/dcb8b77a-fd48-4204-890d-b15f2e4ea7b3-kube-api-access-qbb7n\") pod \"payload-pre-processing-e2e-worker-w4-fc27a5-54dc7c7d8b-5h7m6\" (UID: \"dcb8b77a-fd48-4204-890d-b15f2e4ea7b3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fc27a5-54dc7c7d8b-5h7m6" Sep 16 07:47:09.428695 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.428656 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c9020d-77dd676546-fzccc" event={"ID":"f7caee0e-a84b-4c3f-8bf1-3b0ddc2ec2b0","Type":"ContainerStarted","Data":"09638339fe0bf373780445cf120880999aad113f52f3eda508a350013e118beb"} Sep 16 07:47:09.428892 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.428819 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c9020d-77dd676546-fzccc" Sep 16 07:47:09.467321 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.467206 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:09.487441 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.487400 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56ccdf02-0887-408b-bee7-26f99d1a1c7b-istio-token\") pod \"e2e-autores-b60434-openshift-default-64fbf588d4-s956l\" (UID: \"56ccdf02-0887-408b-bee7-26f99d1a1c7b\") " pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" Sep 16 07:47:09.487441 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.487444 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tdhx\" (UniqueName: \"kubernetes.io/projected/56ccdf02-0887-408b-bee7-26f99d1a1c7b-kube-api-access-6tdhx\") pod \"e2e-autores-b60434-openshift-default-64fbf588d4-s956l\" (UID: \"56ccdf02-0887-408b-bee7-26f99d1a1c7b\") " pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" Sep 16 07:47:09.487659 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.487563 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ccdf02-0887-408b-bee7-26f99d1a1c7b-credential-socket\") pod \"e2e-autores-b60434-openshift-default-64fbf588d4-s956l\" (UID: \"56ccdf02-0887-408b-bee7-26f99d1a1c7b\") " pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" Sep 16 07:47:09.487659 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.487625 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ccdf02-0887-408b-bee7-26f99d1a1c7b-workload-socket\") pod \"e2e-autores-b60434-openshift-default-64fbf588d4-s956l\" (UID: \"56ccdf02-0887-408b-bee7-26f99d1a1c7b\") " pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" Sep 16 07:47:09.487659 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.487648 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56ccdf02-0887-408b-bee7-26f99d1a1c7b-istio-data\") pod \"e2e-autores-b60434-openshift-default-64fbf588d4-s956l\" (UID: \"56ccdf02-0887-408b-bee7-26f99d1a1c7b\") " pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" Sep 16 07:47:09.487794 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.487668 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd352ca8-49ca-4482-b54c-499885f04544-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-fc27a5-6bf7fbd5ff-skp6c\" (UID: \"fd352ca8-49ca-4482-b54c-499885f04544\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-fc27a5-6bf7fbd5ff-skp6c" Sep 16 07:47:09.487794 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.487707 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6xqx\" (UniqueName: \"kubernetes.io/projected/fd352ca8-49ca-4482-b54c-499885f04544-kube-api-access-f6xqx\") pod \"payload-processing-e2e-worker-w4-fc27a5-6bf7fbd5ff-skp6c\" (UID: \"fd352ca8-49ca-4482-b54c-499885f04544\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-fc27a5-6bf7fbd5ff-skp6c" Sep 16 07:47:09.487794 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.487760 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56ccdf02-0887-408b-bee7-26f99d1a1c7b-workload-certs\") pod \"e2e-autores-b60434-openshift-default-64fbf588d4-s956l\" (UID: \"56ccdf02-0887-408b-bee7-26f99d1a1c7b\") " pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" Sep 16 07:47:09.487794 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.487789 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56ccdf02-0887-408b-bee7-26f99d1a1c7b-istio-podinfo\") pod \"e2e-autores-b60434-openshift-default-64fbf588d4-s956l\" (UID: \"56ccdf02-0887-408b-bee7-26f99d1a1c7b\") " pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" Sep 16 07:47:09.488012 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.487821 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56ccdf02-0887-408b-bee7-26f99d1a1c7b-istio-envoy\") pod \"e2e-autores-b60434-openshift-default-64fbf588d4-s956l\" (UID: \"56ccdf02-0887-408b-bee7-26f99d1a1c7b\") " pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" Sep 16 07:47:09.488012 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.487874 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56ccdf02-0887-408b-bee7-26f99d1a1c7b-istiod-ca-cert\") pod \"e2e-autores-b60434-openshift-default-64fbf588d4-s956l\" (UID: \"56ccdf02-0887-408b-bee7-26f99d1a1c7b\") " pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" Sep 16 07:47:09.488141 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.488018 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ccdf02-0887-408b-bee7-26f99d1a1c7b-credential-socket\") pod \"e2e-autores-b60434-openshift-default-64fbf588d4-s956l\" (UID: \"56ccdf02-0887-408b-bee7-26f99d1a1c7b\") " pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" Sep 16 07:47:09.488141 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.488036 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56ccdf02-0887-408b-bee7-26f99d1a1c7b-istio-data\") pod \"e2e-autores-b60434-openshift-default-64fbf588d4-s956l\" (UID: \"56ccdf02-0887-408b-bee7-26f99d1a1c7b\") " pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" Sep 16 07:47:09.488141 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.488129 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ccdf02-0887-408b-bee7-26f99d1a1c7b-workload-socket\") pod \"e2e-autores-b60434-openshift-default-64fbf588d4-s956l\" (UID: \"56ccdf02-0887-408b-bee7-26f99d1a1c7b\") " pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" Sep 16 07:47:09.488565 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.488542 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56ccdf02-0887-408b-bee7-26f99d1a1c7b-istiod-ca-cert\") pod \"e2e-autores-b60434-openshift-default-64fbf588d4-s956l\" (UID: \"56ccdf02-0887-408b-bee7-26f99d1a1c7b\") " pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" Sep 16 07:47:09.488675 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.488594 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd352ca8-49ca-4482-b54c-499885f04544-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-fc27a5-6bf7fbd5ff-skp6c\" (UID: \"fd352ca8-49ca-4482-b54c-499885f04544\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-fc27a5-6bf7fbd5ff-skp6c" Sep 16 07:47:09.488675 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.488646 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56ccdf02-0887-408b-bee7-26f99d1a1c7b-workload-certs\") pod \"e2e-autores-b60434-openshift-default-64fbf588d4-s956l\" (UID: \"56ccdf02-0887-408b-bee7-26f99d1a1c7b\") " pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" Sep 16 07:47:09.490545 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.490516 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56ccdf02-0887-408b-bee7-26f99d1a1c7b-istio-envoy\") pod \"e2e-autores-b60434-openshift-default-64fbf588d4-s956l\" (UID: \"56ccdf02-0887-408b-bee7-26f99d1a1c7b\") " pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" Sep 16 07:47:09.490726 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.490707 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56ccdf02-0887-408b-bee7-26f99d1a1c7b-istio-podinfo\") pod \"e2e-autores-b60434-openshift-default-64fbf588d4-s956l\" (UID: \"56ccdf02-0887-408b-bee7-26f99d1a1c7b\") " pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" Sep 16 07:47:09.498286 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.498254 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6xqx\" (UniqueName: \"kubernetes.io/projected/fd352ca8-49ca-4482-b54c-499885f04544-kube-api-access-f6xqx\") pod \"payload-processing-e2e-worker-w4-fc27a5-6bf7fbd5ff-skp6c\" (UID: \"fd352ca8-49ca-4482-b54c-499885f04544\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-fc27a5-6bf7fbd5ff-skp6c" Sep 16 07:47:09.498655 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.498630 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56ccdf02-0887-408b-bee7-26f99d1a1c7b-istio-token\") pod \"e2e-autores-b60434-openshift-default-64fbf588d4-s956l\" (UID: \"56ccdf02-0887-408b-bee7-26f99d1a1c7b\") " pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" Sep 16 07:47:09.499217 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.499199 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tdhx\" (UniqueName: \"kubernetes.io/projected/56ccdf02-0887-408b-bee7-26f99d1a1c7b-kube-api-access-6tdhx\") pod \"e2e-autores-b60434-openshift-default-64fbf588d4-s956l\" (UID: \"56ccdf02-0887-408b-bee7-26f99d1a1c7b\") " pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" Sep 16 07:47:09.499929 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.499908 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fc27a5-54dc7c7d8b-5h7m6" Sep 16 07:47:09.541272 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.540866 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" Sep 16 07:47:09.571533 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.571493 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:09.601127 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.600973 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-fc27a5-6bf7fbd5ff-skp6c" Sep 16 07:47:09.668027 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.667685 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:09.785328 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.785196 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c9020d-77dd676546-fzccc" podStartSLOduration=2.413046073 podStartE2EDuration="4.785169962s" totalImagesPullingTime="2.372123889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:47:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:47:09.448385048 +0000 UTC m=+801.032163413" watchObservedRunningTime="2026-09-16 07:47:09.785169962 +0000 UTC m=+801.368948304" Sep 16 07:47:09.786490 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.786466 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-fc27a5-54dc7c7d8b-5h7m6"] Sep 16 07:47:09.788088 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:47:09.788060 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcb8b77a_fd48_4204_890d_b15f2e4ea7b3.slice/crio-3d5aeef0c8e3493546671304de10f4db71f172c6d747581fea67663cd1f6739d WatchSource:0}: Error finding container 3d5aeef0c8e3493546671304de10f4db71f172c6d747581fea67663cd1f6739d: Status 404 returned error can't find the container with id 3d5aeef0c8e3493546671304de10f4db71f172c6d747581fea67663cd1f6739d Sep 16 07:47:09.834612 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.834578 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l"] Sep 16 07:47:09.839320 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:47:09.839284 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56ccdf02_0887_408b_bee7_26f99d1a1c7b.slice/crio-3e34ce9fa28b0c937d21e8a53dd9354f4e40ac52724534bdc911d7a9f8384719 WatchSource:0}: Error finding container 3e34ce9fa28b0c937d21e8a53dd9354f4e40ac52724534bdc911d7a9f8384719: Status 404 returned error can't find the container with id 3e34ce9fa28b0c937d21e8a53dd9354f4e40ac52724534bdc911d7a9f8384719 Sep 16 07:47:09.887978 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.887943 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-fc27a5-6bf7fbd5ff-skp6c"] Sep 16 07:47:09.890801 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:47:09.890770 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd352ca8_49ca_4482_b54c_499885f04544.slice/crio-727d09d6cdcc06c86a9dba0218a425e01bcc057a9c8b822f56aa628fe1e44f09 WatchSource:0}: Error finding container 727d09d6cdcc06c86a9dba0218a425e01bcc057a9c8b822f56aa628fe1e44f09: Status 404 returned error can't find the container with id 727d09d6cdcc06c86a9dba0218a425e01bcc057a9c8b822f56aa628fe1e44f09 Sep 16 07:47:09.921086 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:09.921053 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:10.435055 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:10.435016 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-fc27a5-6bf7fbd5ff-skp6c" event={"ID":"fd352ca8-49ca-4482-b54c-499885f04544","Type":"ContainerStarted","Data":"175a6ab504df43187da9eff00ce7fb4625684a78e0d8fda3b43b8df20907d518"} Sep 16 07:47:10.435270 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:10.435198 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-fc27a5-6bf7fbd5ff-skp6c" Sep 16 07:47:10.435270 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:10.435229 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-fc27a5-6bf7fbd5ff-skp6c" event={"ID":"fd352ca8-49ca-4482-b54c-499885f04544","Type":"ContainerStarted","Data":"727d09d6cdcc06c86a9dba0218a425e01bcc057a9c8b822f56aa628fe1e44f09"} Sep 16 07:47:10.436751 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:10.436721 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fc27a5-54dc7c7d8b-5h7m6" event={"ID":"dcb8b77a-fd48-4204-890d-b15f2e4ea7b3","Type":"ContainerStarted","Data":"5b8459fdfc95a350e62c531a4afea6fbf3eb28112462f988dd4dff70560a9b42"} Sep 16 07:47:10.436885 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:10.436765 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fc27a5-54dc7c7d8b-5h7m6" event={"ID":"dcb8b77a-fd48-4204-890d-b15f2e4ea7b3","Type":"ContainerStarted","Data":"3d5aeef0c8e3493546671304de10f4db71f172c6d747581fea67663cd1f6739d"} Sep 16 07:47:10.436885 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:10.436813 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fc27a5-54dc7c7d8b-5h7m6" Sep 16 07:47:10.437993 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:10.437970 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" event={"ID":"56ccdf02-0887-408b-bee7-26f99d1a1c7b","Type":"ContainerStarted","Data":"3e34ce9fa28b0c937d21e8a53dd9354f4e40ac52724534bdc911d7a9f8384719"} Sep 16 07:47:10.451781 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:10.451707 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-fc27a5-6bf7fbd5ff-skp6c" podStartSLOduration=1.451684365 podStartE2EDuration="1.451684365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:47:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:47:10.450508311 +0000 UTC m=+802.034286651" watchObservedRunningTime="2026-09-16 07:47:10.451684365 +0000 UTC m=+802.035462707" Sep 16 07:47:10.468879 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:10.468793 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fc27a5-54dc7c7d8b-5h7m6" podStartSLOduration=1.46876853 podStartE2EDuration="1.46876853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:47:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:47:10.46662042 +0000 UTC m=+802.050398758" watchObservedRunningTime="2026-09-16 07:47:10.46876853 +0000 UTC m=+802.052546869" Sep 16 07:47:10.491725 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:10.491690 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:11.411735 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:11.411514 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:11.798524 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:11.798458 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:11.894353 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:11.894297 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:12.411165 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:12.411120 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:12.833672 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:12.833621 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 16 07:47:12.834006 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:12.833712 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 16 07:47:12.834006 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:12.833754 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 16 07:47:13.378409 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:13.378370 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:13.460811 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:13.460769 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" event={"ID":"56ccdf02-0887-408b-bee7-26f99d1a1c7b","Type":"ContainerStarted","Data":"6aef8e3374e599eb8898cf1e661cf794731beed0f79011e36481a42a2accbb2b"} Sep 16 07:47:13.483843 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:13.483770 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" podStartSLOduration=1.492158731 podStartE2EDuration="4.483747608s" totalImagesPullingTime="2.991588877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:47:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:47:13.481149317 +0000 UTC m=+805.064927656" watchObservedRunningTime="2026-09-16 07:47:13.483747608 +0000 UTC m=+805.067525951" Sep 16 07:47:13.541986 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:13.541940 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" Sep 16 07:47:13.543685 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:13.543661 2552 patch_prober.go:28] interesting pod/e2e-autores-b60434-openshift-default-64fbf588d4-s956l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Sep 16 07:47:13.543790 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:13.543721 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" podUID="56ccdf02-0887-408b-bee7-26f99d1a1c7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Sep 16 07:47:13.607836 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:13.607802 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:14.026190 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:14.026154 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:14.130452 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:14.130418 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:14.236626 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:14.236591 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:14.546782 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:14.546744 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" Sep 16 07:47:14.970193 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:14.970148 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:15.042632 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:15.042595 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:15.163372 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:15.163342 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:15.472949 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:15.472916 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" Sep 16 07:47:15.473823 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:15.473805 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" Sep 16 07:47:15.615315 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:15.615279 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:15.703015 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:15.702983 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:15.806070 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:15.805964 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:16.209076 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:16.209036 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:16.972859 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:16.972821 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:17.844082 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:17.844042 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:18.540928 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:18.540895 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:18.642492 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:18.642455 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:18.766421 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:18.766383 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:19.029582 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:19.029548 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:19.130906 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:19.130871 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:19.166339 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:19.166302 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-b60434-74756d459-7gzs2"] Sep 16 07:47:19.170396 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:19.170370 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-b60434-74756d459-7gzs2" Sep 16 07:47:19.173607 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:19.173581 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-b60434-dockercfg-srcx6\"" Sep 16 07:47:19.173901 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:19.173879 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-b60434\"" Sep 16 07:47:19.184221 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:19.184190 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-b60434-74756d459-7gzs2"] Sep 16 07:47:19.190446 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:19.190412 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vbm2\" (UniqueName: \"kubernetes.io/projected/1cdff8ad-c269-4f9e-9ddc-31b59467c541-kube-api-access-2vbm2\") pod \"payload-processing-e2e-autores-b60434-74756d459-7gzs2\" (UID: \"1cdff8ad-c269-4f9e-9ddc-31b59467c541\") " pod="openshift-ingress/payload-processing-e2e-autores-b60434-74756d459-7gzs2" Sep 16 07:47:19.190611 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:19.190453 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cdff8ad-c269-4f9e-9ddc-31b59467c541-plugins-config-volume\") pod \"payload-processing-e2e-autores-b60434-74756d459-7gzs2\" (UID: \"1cdff8ad-c269-4f9e-9ddc-31b59467c541\") " pod="openshift-ingress/payload-processing-e2e-autores-b60434-74756d459-7gzs2" Sep 16 07:47:19.234425 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:19.234392 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:19.292010 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:19.291903 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vbm2\" (UniqueName: \"kubernetes.io/projected/1cdff8ad-c269-4f9e-9ddc-31b59467c541-kube-api-access-2vbm2\") pod \"payload-processing-e2e-autores-b60434-74756d459-7gzs2\" (UID: \"1cdff8ad-c269-4f9e-9ddc-31b59467c541\") " pod="openshift-ingress/payload-processing-e2e-autores-b60434-74756d459-7gzs2" Sep 16 07:47:19.292010 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:19.291948 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cdff8ad-c269-4f9e-9ddc-31b59467c541-plugins-config-volume\") pod \"payload-processing-e2e-autores-b60434-74756d459-7gzs2\" (UID: \"1cdff8ad-c269-4f9e-9ddc-31b59467c541\") " pod="openshift-ingress/payload-processing-e2e-autores-b60434-74756d459-7gzs2" Sep 16 07:47:19.292777 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:19.292755 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cdff8ad-c269-4f9e-9ddc-31b59467c541-plugins-config-volume\") pod \"payload-processing-e2e-autores-b60434-74756d459-7gzs2\" (UID: \"1cdff8ad-c269-4f9e-9ddc-31b59467c541\") " pod="openshift-ingress/payload-processing-e2e-autores-b60434-74756d459-7gzs2" Sep 16 07:47:19.318748 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:19.318716 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vbm2\" (UniqueName: \"kubernetes.io/projected/1cdff8ad-c269-4f9e-9ddc-31b59467c541-kube-api-access-2vbm2\") pod \"payload-processing-e2e-autores-b60434-74756d459-7gzs2\" (UID: \"1cdff8ad-c269-4f9e-9ddc-31b59467c541\") " pod="openshift-ingress/payload-processing-e2e-autores-b60434-74756d459-7gzs2" Sep 16 07:47:19.381758 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:19.381718 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:19.482449 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:19.482407 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-b60434-74756d459-7gzs2" Sep 16 07:47:19.552467 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:19.552252 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:19.696636 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:19.696588 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:19.759123 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:19.759073 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-b60434-74756d459-7gzs2"] Sep 16 07:47:19.761212 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:47:19.761177 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cdff8ad_c269_4f9e_9ddc_31b59467c541.slice/crio-3041be73dae4513d6b23f0cf52226382448326fd01c0da8410303018869730a3 WatchSource:0}: Error finding container 3041be73dae4513d6b23f0cf52226382448326fd01c0da8410303018869730a3: Status 404 returned error can't find the container with id 3041be73dae4513d6b23f0cf52226382448326fd01c0da8410303018869730a3 Sep 16 07:47:19.854082 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:19.854050 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:19.968829 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:19.968785 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:20.076456 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:20.076323 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:20.440720 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:20.440688 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c9020d-77dd676546-fzccc" Sep 16 07:47:20.500163 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:20.500131 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-b60434-74756d459-7gzs2" event={"ID":"1cdff8ad-c269-4f9e-9ddc-31b59467c541","Type":"ContainerStarted","Data":"a00c1d1def0d3c16fc07eefd0ac5c63d9332661ce4d6f87eb025b99fcbbbbd78"} Sep 16 07:47:20.500348 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:20.500190 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-b60434-74756d459-7gzs2" event={"ID":"1cdff8ad-c269-4f9e-9ddc-31b59467c541","Type":"ContainerStarted","Data":"3041be73dae4513d6b23f0cf52226382448326fd01c0da8410303018869730a3"} Sep 16 07:47:20.500402 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:20.500348 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-b60434-74756d459-7gzs2" Sep 16 07:47:20.518218 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:20.518154 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-b60434-74756d459-7gzs2" podStartSLOduration=1.518131989 podStartE2EDuration="1.518131989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:47:20.515861655 +0000 UTC m=+812.099639994" watchObservedRunningTime="2026-09-16 07:47:20.518131989 +0000 UTC m=+812.101910329" Sep 16 07:47:20.827603 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:20.827513 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:21.447212 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:21.447179 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fc27a5-54dc7c7d8b-5h7m6" Sep 16 07:47:21.447704 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:21.447259 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-fc27a5-6bf7fbd5ff-skp6c" Sep 16 07:47:21.553243 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:21.553205 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:22.376278 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:22.376232 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:22.604125 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:22.604073 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:23.137170 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:23.137132 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:23.853454 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:23.853414 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:24.045842 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:24.045798 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:24.170640 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:24.170608 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:24.439319 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:24.439234 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc"] Sep 16 07:47:24.443546 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:24.443524 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" Sep 16 07:47:24.445991 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:24.445963 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-dd359c-openshift-default-dockercfg-j85gj\"" Sep 16 07:47:24.455159 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:24.455130 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc"] Sep 16 07:47:24.471328 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:24.471291 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:24.544552 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:24.544504 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-istio-envoy\") pod \"e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc\" (UID: \"d2c050a9-e9fb-48f1-abe1-2249f7276ab1\") " pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" Sep 16 07:47:24.544552 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:24.544549 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-istio-data\") pod \"e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc\" (UID: \"d2c050a9-e9fb-48f1-abe1-2249f7276ab1\") " pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" Sep 16 07:47:24.544822 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:24.544647 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-credential-socket\") pod \"e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc\" (UID: \"d2c050a9-e9fb-48f1-abe1-2249f7276ab1\") " pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" Sep 16 07:47:24.544822 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:24.544713 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5blvn\" (UniqueName: \"kubernetes.io/projected/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-kube-api-access-5blvn\") pod \"e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc\" (UID: \"d2c050a9-e9fb-48f1-abe1-2249f7276ab1\") " pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" Sep 16 07:47:24.544822 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:24.544780 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-istio-token\") pod \"e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc\" (UID: \"d2c050a9-e9fb-48f1-abe1-2249f7276ab1\") " pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" Sep 16 07:47:24.544999 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:24.544827 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-istio-podinfo\") pod \"e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc\" (UID: \"d2c050a9-e9fb-48f1-abe1-2249f7276ab1\") " pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" Sep 16 07:47:24.544999 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:24.544904 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-workload-certs\") pod \"e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc\" (UID: \"d2c050a9-e9fb-48f1-abe1-2249f7276ab1\") " pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" Sep 16 07:47:24.544999 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:24.544952 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-istiod-ca-cert\") pod \"e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc\" (UID: \"d2c050a9-e9fb-48f1-abe1-2249f7276ab1\") " pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" Sep 16 07:47:24.545202 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:24.544997 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-workload-socket\") pod \"e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc\" (UID: \"d2c050a9-e9fb-48f1-abe1-2249f7276ab1\") " pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" Sep 16 07:47:24.606328 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:24.606289 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:24.646591 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:24.646553 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-credential-socket\") pod \"e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc\" (UID: \"d2c050a9-e9fb-48f1-abe1-2249f7276ab1\") " pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" Sep 16 07:47:24.646749 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:24.646597 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5blvn\" (UniqueName: \"kubernetes.io/projected/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-kube-api-access-5blvn\") pod \"e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc\" (UID: \"d2c050a9-e9fb-48f1-abe1-2249f7276ab1\") " pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" Sep 16 07:47:24.646749 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:24.646635 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-istio-token\") pod \"e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc\" (UID: \"d2c050a9-e9fb-48f1-abe1-2249f7276ab1\") " pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" Sep 16 07:47:24.646749 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:24.646672 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-istio-podinfo\") pod \"e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc\" (UID: \"d2c050a9-e9fb-48f1-abe1-2249f7276ab1\") " pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" Sep 16 07:47:24.646749 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:24.646695 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-workload-certs\") pod \"e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc\" (UID: \"d2c050a9-e9fb-48f1-abe1-2249f7276ab1\") " pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" Sep 16 07:47:24.646749 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:24.646714 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-istiod-ca-cert\") pod \"e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc\" (UID: \"d2c050a9-e9fb-48f1-abe1-2249f7276ab1\") " pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" Sep 16 07:47:24.646749 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:24.646733 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-workload-socket\") pod \"e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc\" (UID: \"d2c050a9-e9fb-48f1-abe1-2249f7276ab1\") " pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" Sep 16 07:47:24.647081 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:24.646782 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-istio-envoy\") pod \"e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc\" (UID: \"d2c050a9-e9fb-48f1-abe1-2249f7276ab1\") " pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" Sep 16 07:47:24.647081 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:24.646807 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-istio-data\") pod \"e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc\" (UID: \"d2c050a9-e9fb-48f1-abe1-2249f7276ab1\") " pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" Sep 16 07:47:24.647233 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:24.647142 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-credential-socket\") pod \"e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc\" (UID: \"d2c050a9-e9fb-48f1-abe1-2249f7276ab1\") " pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" Sep 16 07:47:24.647233 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:24.647206 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-workload-socket\") pod \"e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc\" (UID: \"d2c050a9-e9fb-48f1-abe1-2249f7276ab1\") " pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" Sep 16 07:47:24.647333 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:24.647237 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-istio-data\") pod \"e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc\" (UID: \"d2c050a9-e9fb-48f1-abe1-2249f7276ab1\") " pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" Sep 16 07:47:24.647468 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:24.647450 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-workload-certs\") pod \"e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc\" (UID: \"d2c050a9-e9fb-48f1-abe1-2249f7276ab1\") " pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" Sep 16 07:47:24.647876 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:24.647839 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-istiod-ca-cert\") pod \"e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc\" (UID: \"d2c050a9-e9fb-48f1-abe1-2249f7276ab1\") " pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" Sep 16 07:47:24.649159 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:24.649138 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-istio-envoy\") pod \"e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc\" (UID: \"d2c050a9-e9fb-48f1-abe1-2249f7276ab1\") " pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" Sep 16 07:47:24.649290 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:24.649274 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-istio-podinfo\") pod \"e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc\" (UID: \"d2c050a9-e9fb-48f1-abe1-2249f7276ab1\") " pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" Sep 16 07:47:24.655955 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:24.655903 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-istio-token\") pod \"e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc\" (UID: \"d2c050a9-e9fb-48f1-abe1-2249f7276ab1\") " pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" Sep 16 07:47:24.656434 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:24.656411 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5blvn\" (UniqueName: \"kubernetes.io/projected/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-kube-api-access-5blvn\") pod \"e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc\" (UID: \"d2c050a9-e9fb-48f1-abe1-2249f7276ab1\") " pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" Sep 16 07:47:24.756881 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:24.756763 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" Sep 16 07:47:24.769637 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:24.769600 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:25.040059 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:25.040028 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc"] Sep 16 07:47:25.042260 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:47:25.042234 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2c050a9_e9fb_48f1_abe1_2249f7276ab1.slice/crio-d8e744db4abfb879ee848717edfc90ec222af2e25532156472542f899f036434 WatchSource:0}: Error finding container d8e744db4abfb879ee848717edfc90ec222af2e25532156472542f899f036434: Status 404 returned error can't find the container with id d8e744db4abfb879ee848717edfc90ec222af2e25532156472542f899f036434 Sep 16 07:47:25.045265 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:25.045230 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 16 07:47:25.045371 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:25.045301 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 16 07:47:25.045371 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:25.045331 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 16 07:47:25.527692 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:25.527656 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" event={"ID":"d2c050a9-e9fb-48f1-abe1-2249f7276ab1","Type":"ContainerStarted","Data":"fd6773887969b9482d50d364da2251e6eafb40c7af3a57d4ef0b3e5ba544bda9"} Sep 16 07:47:25.527881 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:25.527708 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" event={"ID":"d2c050a9-e9fb-48f1-abe1-2249f7276ab1","Type":"ContainerStarted","Data":"d8e744db4abfb879ee848717edfc90ec222af2e25532156472542f899f036434"} Sep 16 07:47:25.553121 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:25.549461 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" podStartSLOduration=1.549437106 podStartE2EDuration="1.549437106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:47:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:47:25.545901708 +0000 UTC m=+817.129680064" watchObservedRunningTime="2026-09-16 07:47:25.549437106 +0000 UTC m=+817.133215447" Sep 16 07:47:25.562969 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:25.562924 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:25.757015 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:25.756975 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" Sep 16 07:47:25.843031 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:25.842932 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:25.934867 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:25.934830 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:26.050134 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:26.050082 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:26.325768 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:26.325731 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:26.387386 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:26.387352 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:26.763243 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:26.763209 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" Sep 16 07:47:27.195768 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:27.195735 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:27.538229 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:27.538132 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" Sep 16 07:47:27.539073 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:27.539053 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" Sep 16 07:47:27.570921 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:27.570885 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:27.663333 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:27.663299 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:28.039292 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:28.039253 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:28.755949 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:28.755915 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:29.459382 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:29.459346 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:30.101939 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:30.101893 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:30.202511 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:30.202471 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:30.789377 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:30.789338 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:31.509370 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:31.509335 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-b60434-74756d459-7gzs2" Sep 16 07:47:31.677405 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:31.677373 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:32.084872 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:32.084832 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0de2c4d3-csbxf"] Sep 16 07:47:32.088274 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:32.088254 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0de2c4d3-csbxf" Sep 16 07:47:32.090513 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:32.090482 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zwk5r\"" Sep 16 07:47:32.090650 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:32.090538 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 07:47:32.095232 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:32.095198 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0de2c4d3-csbxf"] Sep 16 07:47:32.222119 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:32.222057 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1f20063-166c-4751-a9b8-a0e51cab5e91-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0de2c4d3-csbxf\" (UID: \"e1f20063-166c-4751-a9b8-a0e51cab5e91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0de2c4d3-csbxf" Sep 16 07:47:32.229444 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:32.229414 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:32.322873 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:32.322832 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1f20063-166c-4751-a9b8-a0e51cab5e91-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0de2c4d3-csbxf\" (UID: \"e1f20063-166c-4751-a9b8-a0e51cab5e91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0de2c4d3-csbxf" Sep 16 07:47:32.323518 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:32.323497 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1f20063-166c-4751-a9b8-a0e51cab5e91-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0de2c4d3-csbxf\" (UID: \"e1f20063-166c-4751-a9b8-a0e51cab5e91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0de2c4d3-csbxf" Sep 16 07:47:32.340311 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:32.340219 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:32.400859 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:32.400818 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0de2c4d3-csbxf" Sep 16 07:47:32.435419 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:32.435383 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:32.596630 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:32.596541 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:32.670934 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:32.670896 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:32.709612 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:32.709570 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0de2c4d3-csbxf"] Sep 16 07:47:32.712856 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:47:32.712829 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1f20063_166c_4751_a9b8_a0e51cab5e91.slice/crio-b5f8173ee5c99036484a78d3332a64625c4596708f4cb5ef8ab146544bc89e1a WatchSource:0}: Error finding container b5f8173ee5c99036484a78d3332a64625c4596708f4cb5ef8ab146544bc89e1a: Status 404 returned error can't find the container with id b5f8173ee5c99036484a78d3332a64625c4596708f4cb5ef8ab146544bc89e1a Sep 16 07:47:33.177263 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:33.177215 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:33.280643 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:33.280604 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:33.328174 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:33.328140 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-dd359c-69f4889858-cf67d"] Sep 16 07:47:33.331736 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:33.331715 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd359c-69f4889858-cf67d" Sep 16 07:47:33.337253 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:33.337219 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-dd359c-7cccdb9847-6sb9l"] Sep 16 07:47:33.337504 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:33.337482 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-dd359c-dockercfg-wpls7\"" Sep 16 07:47:33.337611 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:33.337505 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-dd359c\"" Sep 16 07:47:33.341377 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:33.341354 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-dd359c-7cccdb9847-6sb9l" Sep 16 07:47:33.373860 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:33.373809 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-dd359c-69f4889858-cf67d"] Sep 16 07:47:33.388877 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:33.388839 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-dd359c-7cccdb9847-6sb9l"] Sep 16 07:47:33.432159 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:33.432047 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/968b978d-ccf1-4116-876b-357bd1541402-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-dd359c-7cccdb9847-6sb9l\" (UID: \"968b978d-ccf1-4116-876b-357bd1541402\") " pod="openshift-ingress/payload-processing-e2e-inf-b-dd359c-7cccdb9847-6sb9l" Sep 16 07:47:33.432159 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:33.432115 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06c531c4-2b1c-47fd-afa3-6473fc38135f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-dd359c-69f4889858-cf67d\" (UID: \"06c531c4-2b1c-47fd-afa3-6473fc38135f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd359c-69f4889858-cf67d" Sep 16 07:47:33.432370 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:33.432185 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgthw\" (UniqueName: \"kubernetes.io/projected/968b978d-ccf1-4116-876b-357bd1541402-kube-api-access-sgthw\") pod \"payload-processing-e2e-inf-b-dd359c-7cccdb9847-6sb9l\" (UID: \"968b978d-ccf1-4116-876b-357bd1541402\") " pod="openshift-ingress/payload-processing-e2e-inf-b-dd359c-7cccdb9847-6sb9l" Sep 16 07:47:33.432370 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:33.432272 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89l25\" (UniqueName: \"kubernetes.io/projected/06c531c4-2b1c-47fd-afa3-6473fc38135f-kube-api-access-89l25\") pod \"payload-pre-processing-e2e-inf-b-dd359c-69f4889858-cf67d\" (UID: \"06c531c4-2b1c-47fd-afa3-6473fc38135f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd359c-69f4889858-cf67d" Sep 16 07:47:33.533930 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:33.533885 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/968b978d-ccf1-4116-876b-357bd1541402-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-dd359c-7cccdb9847-6sb9l\" (UID: \"968b978d-ccf1-4116-876b-357bd1541402\") " pod="openshift-ingress/payload-processing-e2e-inf-b-dd359c-7cccdb9847-6sb9l" Sep 16 07:47:33.533930 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:33.533933 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06c531c4-2b1c-47fd-afa3-6473fc38135f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-dd359c-69f4889858-cf67d\" (UID: \"06c531c4-2b1c-47fd-afa3-6473fc38135f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd359c-69f4889858-cf67d" Sep 16 07:47:33.534177 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:33.534038 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgthw\" (UniqueName: \"kubernetes.io/projected/968b978d-ccf1-4116-876b-357bd1541402-kube-api-access-sgthw\") pod \"payload-processing-e2e-inf-b-dd359c-7cccdb9847-6sb9l\" (UID: \"968b978d-ccf1-4116-876b-357bd1541402\") " pod="openshift-ingress/payload-processing-e2e-inf-b-dd359c-7cccdb9847-6sb9l" Sep 16 07:47:33.534177 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:33.534078 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89l25\" (UniqueName: \"kubernetes.io/projected/06c531c4-2b1c-47fd-afa3-6473fc38135f-kube-api-access-89l25\") pod \"payload-pre-processing-e2e-inf-b-dd359c-69f4889858-cf67d\" (UID: \"06c531c4-2b1c-47fd-afa3-6473fc38135f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd359c-69f4889858-cf67d" Sep 16 07:47:33.534807 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:33.534788 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/968b978d-ccf1-4116-876b-357bd1541402-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-dd359c-7cccdb9847-6sb9l\" (UID: \"968b978d-ccf1-4116-876b-357bd1541402\") " pod="openshift-ingress/payload-processing-e2e-inf-b-dd359c-7cccdb9847-6sb9l" Sep 16 07:47:33.534989 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:33.534964 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06c531c4-2b1c-47fd-afa3-6473fc38135f-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-dd359c-69f4889858-cf67d\" (UID: \"06c531c4-2b1c-47fd-afa3-6473fc38135f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd359c-69f4889858-cf67d" Sep 16 07:47:33.542301 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:33.542271 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgthw\" (UniqueName: \"kubernetes.io/projected/968b978d-ccf1-4116-876b-357bd1541402-kube-api-access-sgthw\") pod \"payload-processing-e2e-inf-b-dd359c-7cccdb9847-6sb9l\" (UID: \"968b978d-ccf1-4116-876b-357bd1541402\") " pod="openshift-ingress/payload-processing-e2e-inf-b-dd359c-7cccdb9847-6sb9l" Sep 16 07:47:33.542447 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:33.542325 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89l25\" (UniqueName: \"kubernetes.io/projected/06c531c4-2b1c-47fd-afa3-6473fc38135f-kube-api-access-89l25\") pod \"payload-pre-processing-e2e-inf-b-dd359c-69f4889858-cf67d\" (UID: \"06c531c4-2b1c-47fd-afa3-6473fc38135f\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd359c-69f4889858-cf67d" Sep 16 07:47:33.590294 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:33.590260 2552 generic.go:356] "Generic (PLEG): container finished" podID="e1f20063-166c-4751-a9b8-a0e51cab5e91" containerID="008a85a87484c77e6a7f0f338b7e87b49a828b7c8cd7d25007dc167a732e9816" exitCode=0 Sep 16 07:47:33.590294 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:33.590299 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0de2c4d3-csbxf" event={"ID":"e1f20063-166c-4751-a9b8-a0e51cab5e91","Type":"ContainerDied","Data":"008a85a87484c77e6a7f0f338b7e87b49a828b7c8cd7d25007dc167a732e9816"} Sep 16 07:47:33.590494 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:33.590335 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0de2c4d3-csbxf" event={"ID":"e1f20063-166c-4751-a9b8-a0e51cab5e91","Type":"ContainerStarted","Data":"b5f8173ee5c99036484a78d3332a64625c4596708f4cb5ef8ab146544bc89e1a"} Sep 16 07:47:33.643644 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:33.643342 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd359c-69f4889858-cf67d" Sep 16 07:47:33.655368 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:33.655326 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-dd359c-7cccdb9847-6sb9l" Sep 16 07:47:33.693136 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:33.693031 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:33.923651 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:33.923612 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-dd359c-69f4889858-cf67d"] Sep 16 07:47:33.924940 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:47:33.924907 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06c531c4_2b1c_47fd_afa3_6473fc38135f.slice/crio-e671b82421b57e0c11afcb8f466580ccf20c4ea4515adeb13fe6d0a980a880de WatchSource:0}: Error finding container e671b82421b57e0c11afcb8f466580ccf20c4ea4515adeb13fe6d0a980a880de: Status 404 returned error can't find the container with id e671b82421b57e0c11afcb8f466580ccf20c4ea4515adeb13fe6d0a980a880de Sep 16 07:47:33.951544 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:33.951501 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-dd359c-7cccdb9847-6sb9l"] Sep 16 07:47:33.956355 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:47:33.956322 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod968b978d_ccf1_4116_876b_357bd1541402.slice/crio-c7a03e6834f67d60a7c1046138771b9e8e1dc7a2686f2c6f80ef2e7217b047ec WatchSource:0}: Error finding container c7a03e6834f67d60a7c1046138771b9e8e1dc7a2686f2c6f80ef2e7217b047ec: Status 404 returned error can't find the container with id c7a03e6834f67d60a7c1046138771b9e8e1dc7a2686f2c6f80ef2e7217b047ec Sep 16 07:47:34.364409 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:34.364313 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:34.599991 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:34.599950 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-dd359c-7cccdb9847-6sb9l" event={"ID":"968b978d-ccf1-4116-876b-357bd1541402","Type":"ContainerStarted","Data":"c8f94c687d05cc374c73a0b2d9ba495b600b46c15d3f1e69d020a8c2598440bb"} Sep 16 07:47:34.600307 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:34.600276 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-dd359c-7cccdb9847-6sb9l" Sep 16 07:47:34.600429 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:34.600320 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-dd359c-7cccdb9847-6sb9l" event={"ID":"968b978d-ccf1-4116-876b-357bd1541402","Type":"ContainerStarted","Data":"c7a03e6834f67d60a7c1046138771b9e8e1dc7a2686f2c6f80ef2e7217b047ec"} Sep 16 07:47:34.601880 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:34.601855 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd359c-69f4889858-cf67d" event={"ID":"06c531c4-2b1c-47fd-afa3-6473fc38135f","Type":"ContainerStarted","Data":"fb2155bd9b510c702759270b0103ea6b633444d99ab3bad585e36b3c9a20050d"} Sep 16 07:47:34.602003 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:34.601908 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd359c-69f4889858-cf67d" event={"ID":"06c531c4-2b1c-47fd-afa3-6473fc38135f","Type":"ContainerStarted","Data":"e671b82421b57e0c11afcb8f466580ccf20c4ea4515adeb13fe6d0a980a880de"} Sep 16 07:47:34.602059 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:34.602016 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd359c-69f4889858-cf67d" Sep 16 07:47:34.621632 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:34.621503 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-dd359c-7cccdb9847-6sb9l" podStartSLOduration=1.621481426 podStartE2EDuration="1.621481426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:47:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:47:34.618938144 +0000 UTC m=+826.202716505" watchObservedRunningTime="2026-09-16 07:47:34.621481426 +0000 UTC m=+826.205259766" Sep 16 07:47:34.640795 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:34.640739 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd359c-69f4889858-cf67d" podStartSLOduration=1.6407201 podStartE2EDuration="1.6407201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:47:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:47:34.639650454 +0000 UTC m=+826.223428799" watchObservedRunningTime="2026-09-16 07:47:34.6407201 +0000 UTC m=+826.224498438" Sep 16 07:47:34.775003 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:34.774973 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0de2c4d3-csbxf" Sep 16 07:47:34.845871 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:34.845829 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1f20063-166c-4751-a9b8-a0e51cab5e91-maas-api-service-ca\") pod \"e1f20063-166c-4751-a9b8-a0e51cab5e91\" (UID: \"e1f20063-166c-4751-a9b8-a0e51cab5e91\") " Sep 16 07:47:34.846393 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:34.846365 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1f20063-166c-4751-a9b8-a0e51cab5e91-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e1f20063-166c-4751-a9b8-a0e51cab5e91" (UID: "e1f20063-166c-4751-a9b8-a0e51cab5e91"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:47:34.946849 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:34.946822 2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1f20063-166c-4751-a9b8-a0e51cab5e91-maas-api-service-ca\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:47:35.176068 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:35.176026 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:35.608530 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:35.608497 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0de2c4d3-csbxf" Sep 16 07:47:35.608693 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:35.608544 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0de2c4d3-csbxf" event={"ID":"e1f20063-166c-4751-a9b8-a0e51cab5e91","Type":"ContainerDied","Data":"b5f8173ee5c99036484a78d3332a64625c4596708f4cb5ef8ab146544bc89e1a"} Sep 16 07:47:35.608693 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:35.608602 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5f8173ee5c99036484a78d3332a64625c4596708f4cb5ef8ab146544bc89e1a" Sep 16 07:47:35.860010 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:35.859901 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:36.669187 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:36.669152 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:37.290538 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:37.290504 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:37.627318 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:37.627225 2552 generic.go:356] "Generic (PLEG): container finished" podID="56ccdf02-0887-408b-bee7-26f99d1a1c7b" containerID="6aef8e3374e599eb8898cf1e661cf794731beed0f79011e36481a42a2accbb2b" exitCode=0 Sep 16 07:47:37.627318 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:37.627287 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" event={"ID":"56ccdf02-0887-408b-bee7-26f99d1a1c7b","Type":"ContainerDied","Data":"6aef8e3374e599eb8898cf1e661cf794731beed0f79011e36481a42a2accbb2b"} Sep 16 07:47:37.627719 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:37.627703 2552 scope.go:117] "RemoveContainer" containerID="6aef8e3374e599eb8898cf1e661cf794731beed0f79011e36481a42a2accbb2b" Sep 16 07:47:37.628855 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:37.628824 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 16 07:47:37.628904 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:37.628895 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 16 07:47:37.628940 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:37.628925 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 16 07:47:38.168049 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:38.167994 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:38.634649 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:38.634613 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" event={"ID":"56ccdf02-0887-408b-bee7-26f99d1a1c7b","Type":"ContainerStarted","Data":"7a6d86c817d6b88ab7334d516088cdc8f7ad02c6165820956512020ef1d90ceb"} Sep 16 07:47:38.844364 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:38.844323 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:38.950026 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:38.949988 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:39.541354 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:39.541324 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" Sep 16 07:47:39.541721 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:39.541696 2552 patch_prober.go:28] interesting pod/e2e-autores-b60434-openshift-default-64fbf588d4-s956l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Sep 16 07:47:39.541875 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:39.541852 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" podUID="56ccdf02-0887-408b-bee7-26f99d1a1c7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Sep 16 07:47:39.652308 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:39.652272 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:40.541581 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:40.541547 2552 patch_prober.go:28] interesting pod/e2e-autores-b60434-openshift-default-64fbf588d4-s956l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Sep 16 07:47:40.541879 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:40.541855 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" podUID="56ccdf02-0887-408b-bee7-26f99d1a1c7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Sep 16 07:47:41.542398 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:41.542351 2552 patch_prober.go:28] interesting pod/e2e-autores-b60434-openshift-default-64fbf588d4-s956l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Sep 16 07:47:41.542796 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:41.542434 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" podUID="56ccdf02-0887-408b-bee7-26f99d1a1c7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Sep 16 07:47:42.219930 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:42.219889 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:42.542445 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:42.542352 2552 patch_prober.go:28] interesting pod/e2e-autores-b60434-openshift-default-64fbf588d4-s956l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Sep 16 07:47:42.542445 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:42.542418 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" podUID="56ccdf02-0887-408b-bee7-26f99d1a1c7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Sep 16 07:47:42.857058 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:42.856960 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:42.974326 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:42.974291 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:43.037232 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:43.037185 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0de2c4d3-csbxf"] Sep 16 07:47:43.040722 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:43.040681 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0de2c4d3-csbxf"] Sep 16 07:47:43.042252 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:43.042222 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:43.113306 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:43.113019 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:43.197123 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:43.196914 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:43.464056 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:43.464017 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:43.542615 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:43.542578 2552 patch_prober.go:28] interesting pod/e2e-autores-b60434-openshift-default-64fbf588d4-s956l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" start-of-body= Sep 16 07:47:43.543020 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:43.542655 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" podUID="56ccdf02-0887-408b-bee7-26f99d1a1c7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.34:15021/healthz/ready\": dial tcp 10.133.0.34:15021: connect: connection refused" Sep 16 07:47:43.639418 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:43.639376 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:43.735343 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:43.735254 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:43.854554 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:43.854477 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:44.173127 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:44.171166 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:44.306826 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:44.306786 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:44.552827 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:44.551262 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" Sep 16 07:47:44.552827 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:44.552613 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" Sep 16 07:47:44.557638 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:44.557605 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" Sep 16 07:47:44.698677 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:44.698638 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:44.937746 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:44.937708 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1f20063-166c-4751-a9b8-a0e51cab5e91" path="/var/lib/kubelet/pods/e1f20063-166c-4751-a9b8-a0e51cab5e91/volumes" Sep 16 07:47:45.070905 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:45.070869 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:45.610940 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:45.610853 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-dd359c-7cccdb9847-6sb9l" Sep 16 07:47:45.610940 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:45.610927 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd359c-69f4889858-cf67d" Sep 16 07:47:45.800157 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:45.800117 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:46.644084 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:46.644020 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:47.000950 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:47.000909 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:47.101588 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:47.101548 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:47.236200 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:47.236158 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:47.404089 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:47.403990 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:48.197201 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:48.197164 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:48.699398 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:48.699362 2552 generic.go:356] "Generic (PLEG): container finished" podID="d2c050a9-e9fb-48f1-abe1-2249f7276ab1" containerID="fd6773887969b9482d50d364da2251e6eafb40c7af3a57d4ef0b3e5ba544bda9" exitCode=0 Sep 16 07:47:48.699610 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:48.699435 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" event={"ID":"d2c050a9-e9fb-48f1-abe1-2249f7276ab1","Type":"ContainerDied","Data":"fd6773887969b9482d50d364da2251e6eafb40c7af3a57d4ef0b3e5ba544bda9"} Sep 16 07:47:48.699874 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:48.699860 2552 scope.go:117] "RemoveContainer" containerID="fd6773887969b9482d50d364da2251e6eafb40c7af3a57d4ef0b3e5ba544bda9" Sep 16 07:47:48.700951 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:48.700926 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 16 07:47:48.701027 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:48.700992 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 16 07:47:48.701067 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:48.701027 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 16 07:47:48.824310 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:48.824257 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:49.710830 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:49.710792 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" event={"ID":"d2c050a9-e9fb-48f1-abe1-2249f7276ab1","Type":"ContainerStarted","Data":"b6ba6b31d4fe704ae3d6365fa547258dea4ab1d4d2c53de07273ee49086f9752"} Sep 16 07:47:49.726532 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:49.726498 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:49.757415 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:49.757364 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" Sep 16 07:47:49.757709 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:49.757679 2552 patch_prober.go:28] interesting pod/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 16 07:47:49.757784 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:49.757743 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" podUID="d2c050a9-e9fb-48f1-abe1-2249f7276ab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 16 07:47:50.256220 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:50.256174 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-c9020d/simulator-w0-c9020d-kserve-5946f6c555-55w4c"] Sep 16 07:47:50.256841 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:50.256813 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1f20063-166c-4751-a9b8-a0e51cab5e91" containerName="revoke-keys" Sep 16 07:47:50.256841 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:50.256842 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1f20063-166c-4751-a9b8-a0e51cab5e91" containerName="revoke-keys" Sep 16 07:47:50.257005 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:50.256974 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1f20063-166c-4751-a9b8-a0e51cab5e91" containerName="revoke-keys" Sep 16 07:47:50.260885 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:50.260855 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c9020d/simulator-w0-c9020d-kserve-5946f6c555-55w4c" Sep 16 07:47:50.266396 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:50.266365 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c9020d\"/\"default-dockercfg-mcpgb\"" Sep 16 07:47:50.266921 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:50.266901 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c9020d\"/\"simulator-w0-c9020d-kserve-self-signed-certs\"" Sep 16 07:47:50.267276 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:50.267258 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-c9020d\"/\"kube-root-ca.crt\"" Sep 16 07:47:50.267619 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:50.267602 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-c9020d\"/\"openshift-service-ca.crt\"" Sep 16 07:47:50.273616 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:50.272091 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c9020d/simulator-w0-c9020d-kserve-5946f6c555-55w4c"] Sep 16 07:47:50.380019 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:50.379984 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:50.404188 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:50.404147 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x5jz\" (UniqueName: \"kubernetes.io/projected/30c1f9e7-d353-4a41-8336-b87728ac9838-kube-api-access-7x5jz\") pod \"simulator-w0-c9020d-kserve-5946f6c555-55w4c\" (UID: \"30c1f9e7-d353-4a41-8336-b87728ac9838\") " pod="e2e-models-e2e-worker-w0-c9020d/simulator-w0-c9020d-kserve-5946f6c555-55w4c" Sep 16 07:47:50.404380 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:50.404214 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30c1f9e7-d353-4a41-8336-b87728ac9838-model-cache\") pod \"simulator-w0-c9020d-kserve-5946f6c555-55w4c\" (UID: \"30c1f9e7-d353-4a41-8336-b87728ac9838\") " pod="e2e-models-e2e-worker-w0-c9020d/simulator-w0-c9020d-kserve-5946f6c555-55w4c" Sep 16 07:47:50.404380 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:50.404284 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30c1f9e7-d353-4a41-8336-b87728ac9838-tmp-dir\") pod \"simulator-w0-c9020d-kserve-5946f6c555-55w4c\" (UID: \"30c1f9e7-d353-4a41-8336-b87728ac9838\") " pod="e2e-models-e2e-worker-w0-c9020d/simulator-w0-c9020d-kserve-5946f6c555-55w4c" Sep 16 07:47:50.404380 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:50.404315 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30c1f9e7-d353-4a41-8336-b87728ac9838-dshm\") pod \"simulator-w0-c9020d-kserve-5946f6c555-55w4c\" (UID: \"30c1f9e7-d353-4a41-8336-b87728ac9838\") " pod="e2e-models-e2e-worker-w0-c9020d/simulator-w0-c9020d-kserve-5946f6c555-55w4c" Sep 16 07:47:50.404380 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:50.404364 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30c1f9e7-d353-4a41-8336-b87728ac9838-home\") pod \"simulator-w0-c9020d-kserve-5946f6c555-55w4c\" (UID: \"30c1f9e7-d353-4a41-8336-b87728ac9838\") " pod="e2e-models-e2e-worker-w0-c9020d/simulator-w0-c9020d-kserve-5946f6c555-55w4c" Sep 16 07:47:50.404549 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:50.404400 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30c1f9e7-d353-4a41-8336-b87728ac9838-tls-certs\") pod \"simulator-w0-c9020d-kserve-5946f6c555-55w4c\" (UID: \"30c1f9e7-d353-4a41-8336-b87728ac9838\") " pod="e2e-models-e2e-worker-w0-c9020d/simulator-w0-c9020d-kserve-5946f6c555-55w4c" Sep 16 07:47:50.479464 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:50.479422 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:50.505569 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:50.505531 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30c1f9e7-d353-4a41-8336-b87728ac9838-model-cache\") pod \"simulator-w0-c9020d-kserve-5946f6c555-55w4c\" (UID: \"30c1f9e7-d353-4a41-8336-b87728ac9838\") " pod="e2e-models-e2e-worker-w0-c9020d/simulator-w0-c9020d-kserve-5946f6c555-55w4c" Sep 16 07:47:50.505812 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:50.505599 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30c1f9e7-d353-4a41-8336-b87728ac9838-tmp-dir\") pod \"simulator-w0-c9020d-kserve-5946f6c555-55w4c\" (UID: \"30c1f9e7-d353-4a41-8336-b87728ac9838\") " pod="e2e-models-e2e-worker-w0-c9020d/simulator-w0-c9020d-kserve-5946f6c555-55w4c" Sep 16 07:47:50.505812 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:50.505627 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30c1f9e7-d353-4a41-8336-b87728ac9838-dshm\") pod \"simulator-w0-c9020d-kserve-5946f6c555-55w4c\" (UID: \"30c1f9e7-d353-4a41-8336-b87728ac9838\") " pod="e2e-models-e2e-worker-w0-c9020d/simulator-w0-c9020d-kserve-5946f6c555-55w4c" Sep 16 07:47:50.505812 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:50.505696 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30c1f9e7-d353-4a41-8336-b87728ac9838-home\") pod \"simulator-w0-c9020d-kserve-5946f6c555-55w4c\" (UID: \"30c1f9e7-d353-4a41-8336-b87728ac9838\") " pod="e2e-models-e2e-worker-w0-c9020d/simulator-w0-c9020d-kserve-5946f6c555-55w4c" Sep 16 07:47:50.505812 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:50.505736 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30c1f9e7-d353-4a41-8336-b87728ac9838-tls-certs\") pod \"simulator-w0-c9020d-kserve-5946f6c555-55w4c\" (UID: \"30c1f9e7-d353-4a41-8336-b87728ac9838\") " pod="e2e-models-e2e-worker-w0-c9020d/simulator-w0-c9020d-kserve-5946f6c555-55w4c" Sep 16 07:47:50.505812 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:50.505789 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x5jz\" (UniqueName: \"kubernetes.io/projected/30c1f9e7-d353-4a41-8336-b87728ac9838-kube-api-access-7x5jz\") pod \"simulator-w0-c9020d-kserve-5946f6c555-55w4c\" (UID: \"30c1f9e7-d353-4a41-8336-b87728ac9838\") " pod="e2e-models-e2e-worker-w0-c9020d/simulator-w0-c9020d-kserve-5946f6c555-55w4c" Sep 16 07:47:50.506093 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:50.506078 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30c1f9e7-d353-4a41-8336-b87728ac9838-model-cache\") pod \"simulator-w0-c9020d-kserve-5946f6c555-55w4c\" (UID: \"30c1f9e7-d353-4a41-8336-b87728ac9838\") " pod="e2e-models-e2e-worker-w0-c9020d/simulator-w0-c9020d-kserve-5946f6c555-55w4c" Sep 16 07:47:50.506189 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:50.506158 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30c1f9e7-d353-4a41-8336-b87728ac9838-home\") pod \"simulator-w0-c9020d-kserve-5946f6c555-55w4c\" (UID: \"30c1f9e7-d353-4a41-8336-b87728ac9838\") " pod="e2e-models-e2e-worker-w0-c9020d/simulator-w0-c9020d-kserve-5946f6c555-55w4c" Sep 16 07:47:50.506406 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:50.506344 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30c1f9e7-d353-4a41-8336-b87728ac9838-tmp-dir\") pod \"simulator-w0-c9020d-kserve-5946f6c555-55w4c\" (UID: \"30c1f9e7-d353-4a41-8336-b87728ac9838\") " pod="e2e-models-e2e-worker-w0-c9020d/simulator-w0-c9020d-kserve-5946f6c555-55w4c" Sep 16 07:47:50.508333 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:50.508307 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30c1f9e7-d353-4a41-8336-b87728ac9838-dshm\") pod \"simulator-w0-c9020d-kserve-5946f6c555-55w4c\" (UID: \"30c1f9e7-d353-4a41-8336-b87728ac9838\") " pod="e2e-models-e2e-worker-w0-c9020d/simulator-w0-c9020d-kserve-5946f6c555-55w4c" Sep 16 07:47:50.508527 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:50.508511 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30c1f9e7-d353-4a41-8336-b87728ac9838-tls-certs\") pod \"simulator-w0-c9020d-kserve-5946f6c555-55w4c\" (UID: \"30c1f9e7-d353-4a41-8336-b87728ac9838\") " pod="e2e-models-e2e-worker-w0-c9020d/simulator-w0-c9020d-kserve-5946f6c555-55w4c" Sep 16 07:47:50.517185 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:50.517141 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x5jz\" (UniqueName: \"kubernetes.io/projected/30c1f9e7-d353-4a41-8336-b87728ac9838-kube-api-access-7x5jz\") pod \"simulator-w0-c9020d-kserve-5946f6c555-55w4c\" (UID: \"30c1f9e7-d353-4a41-8336-b87728ac9838\") " pod="e2e-models-e2e-worker-w0-c9020d/simulator-w0-c9020d-kserve-5946f6c555-55w4c" Sep 16 07:47:50.580622 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:50.580584 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:50.580868 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:50.580784 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c9020d/simulator-w0-c9020d-kserve-5946f6c555-55w4c" Sep 16 07:47:50.684603 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:50.684563 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:50.760189 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:50.758544 2552 patch_prober.go:28] interesting pod/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 16 07:47:50.760189 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:50.758647 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" podUID="d2c050a9-e9fb-48f1-abe1-2249f7276ab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 16 07:47:50.827502 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:50.827466 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:50.870125 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:50.869936 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c9020d/simulator-w0-c9020d-kserve-5946f6c555-55w4c"] Sep 16 07:47:50.884973 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:47:50.884912 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30c1f9e7_d353_4a41_8336_b87728ac9838.slice/crio-2f914d10412c7e96b30a69bf32123785a2c3efa48d8c98530ca304910673aa20 WatchSource:0}: Error finding container 2f914d10412c7e96b30a69bf32123785a2c3efa48d8c98530ca304910673aa20: Status 404 returned error can't find the container with id 2f914d10412c7e96b30a69bf32123785a2c3efa48d8c98530ca304910673aa20 Sep 16 07:47:50.930984 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:50.930945 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:51.252199 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:51.252159 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:51.724182 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:51.724148 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c9020d/simulator-w0-c9020d-kserve-5946f6c555-55w4c" event={"ID":"30c1f9e7-d353-4a41-8336-b87728ac9838","Type":"ContainerStarted","Data":"fae213868325c5f8c125bd8fe94b9dc7a8b2bbb3150911061db01fcf4ab22128"} Sep 16 07:47:51.724382 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:51.724194 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c9020d/simulator-w0-c9020d-kserve-5946f6c555-55w4c" event={"ID":"30c1f9e7-d353-4a41-8336-b87728ac9838","Type":"ContainerStarted","Data":"2f914d10412c7e96b30a69bf32123785a2c3efa48d8c98530ca304910673aa20"} Sep 16 07:47:51.741854 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:51.741787 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-c9020d/simulator-w0-c9020d-kserve-5946f6c555-55w4c" podStartSLOduration=1.741764635 podStartE2EDuration="1.741764635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:47:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:47:51.739843693 +0000 UTC m=+843.323622030" watchObservedRunningTime="2026-09-16 07:47:51.741764635 +0000 UTC m=+843.325542974" Sep 16 07:47:51.757764 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:51.757282 2552 patch_prober.go:28] interesting pod/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 16 07:47:51.757764 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:51.757361 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" podUID="d2c050a9-e9fb-48f1-abe1-2249f7276ab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 16 07:47:51.988800 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:51.988692 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:52.758233 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:52.758180 2552 patch_prober.go:28] interesting pod/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 16 07:47:52.758454 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:52.758304 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" podUID="d2c050a9-e9fb-48f1-abe1-2249f7276ab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 16 07:47:53.729128 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:53.729074 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:47:53.758256 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:53.758207 2552 patch_prober.go:28] interesting pod/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 16 07:47:53.758467 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:53.758300 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" podUID="d2c050a9-e9fb-48f1-abe1-2249f7276ab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 16 07:47:54.758049 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:54.757992 2552 patch_prober.go:28] interesting pod/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body= Sep 16 07:47:54.758582 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:54.758093 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" podUID="d2c050a9-e9fb-48f1-abe1-2249f7276ab1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" Sep 16 07:47:55.762261 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:55.762227 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" Sep 16 07:47:55.762767 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:55.762500 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" Sep 16 07:47:55.763542 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:47:55.763522 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" Sep 16 07:48:00.581239 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:00.581199 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c9020d/simulator-w0-c9020d-kserve-5946f6c555-55w4c" Sep 16 07:48:00.581647 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:00.581258 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-c9020d/simulator-w0-c9020d-kserve-5946f6c555-55w4c" Sep 16 07:48:00.594043 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:00.594005 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-c9020d/simulator-w0-c9020d-kserve-5946f6c555-55w4c" Sep 16 07:48:00.788611 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:00.788573 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c9020d/simulator-w0-c9020d-kserve-5946f6c555-55w4c" Sep 16 07:48:11.211821 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:11.211765 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:48:12.382363 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:12.382326 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:48:13.170219 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:13.170173 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:48:13.791625 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:13.791588 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:48:14.422789 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:14.422757 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:48:15.177581 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:15.177551 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:48:15.844798 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:15.844760 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:48:16.609024 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:16.608984 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-fc27a5/simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd"] Sep 16 07:48:16.612882 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:16.612858 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fc27a5/simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd" Sep 16 07:48:16.621150 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:16.621121 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-fc27a5\"/\"kube-root-ca.crt\"" Sep 16 07:48:16.621285 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:16.621162 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-fc27a5\"/\"simulator-w4-fc27a5-kserve-self-signed-certs\"" Sep 16 07:48:16.621285 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:16.621181 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-fc27a5\"/\"openshift-service-ca.crt\"" Sep 16 07:48:16.621466 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:16.621450 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-fc27a5\"/\"default-dockercfg-dkpl6\"" Sep 16 07:48:16.640308 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:16.640271 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-fc27a5/simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd"] Sep 16 07:48:16.668490 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:16.668453 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a094926-e874-4ed6-94ef-f5e0616f7026-dshm\") pod \"simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd\" (UID: \"8a094926-e874-4ed6-94ef-f5e0616f7026\") " pod="e2e-models-e2e-worker-w4-fc27a5/simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd" Sep 16 07:48:16.668668 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:16.668525 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp7sj\" (UniqueName: \"kubernetes.io/projected/8a094926-e874-4ed6-94ef-f5e0616f7026-kube-api-access-wp7sj\") pod \"simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd\" (UID: \"8a094926-e874-4ed6-94ef-f5e0616f7026\") " pod="e2e-models-e2e-worker-w4-fc27a5/simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd" Sep 16 07:48:16.668668 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:16.668587 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a094926-e874-4ed6-94ef-f5e0616f7026-tls-certs\") pod \"simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd\" (UID: \"8a094926-e874-4ed6-94ef-f5e0616f7026\") " pod="e2e-models-e2e-worker-w4-fc27a5/simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd" Sep 16 07:48:16.668668 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:16.668625 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a094926-e874-4ed6-94ef-f5e0616f7026-home\") pod \"simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd\" (UID: \"8a094926-e874-4ed6-94ef-f5e0616f7026\") " pod="e2e-models-e2e-worker-w4-fc27a5/simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd" Sep 16 07:48:16.668785 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:16.668741 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a094926-e874-4ed6-94ef-f5e0616f7026-tmp-dir\") pod \"simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd\" (UID: \"8a094926-e874-4ed6-94ef-f5e0616f7026\") " pod="e2e-models-e2e-worker-w4-fc27a5/simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd" Sep 16 07:48:16.668827 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:16.668786 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a094926-e874-4ed6-94ef-f5e0616f7026-model-cache\") pod \"simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd\" (UID: \"8a094926-e874-4ed6-94ef-f5e0616f7026\") " pod="e2e-models-e2e-worker-w4-fc27a5/simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd" Sep 16 07:48:16.691565 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:16.691519 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:48:16.769575 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:16.769534 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a094926-e874-4ed6-94ef-f5e0616f7026-tmp-dir\") pod \"simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd\" (UID: \"8a094926-e874-4ed6-94ef-f5e0616f7026\") " pod="e2e-models-e2e-worker-w4-fc27a5/simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd" Sep 16 07:48:16.769761 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:16.769583 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a094926-e874-4ed6-94ef-f5e0616f7026-model-cache\") pod \"simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd\" (UID: \"8a094926-e874-4ed6-94ef-f5e0616f7026\") " pod="e2e-models-e2e-worker-w4-fc27a5/simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd" Sep 16 07:48:16.769761 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:16.769634 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a094926-e874-4ed6-94ef-f5e0616f7026-dshm\") pod \"simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd\" (UID: \"8a094926-e874-4ed6-94ef-f5e0616f7026\") " pod="e2e-models-e2e-worker-w4-fc27a5/simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd" Sep 16 07:48:16.769761 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:16.769701 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp7sj\" (UniqueName: \"kubernetes.io/projected/8a094926-e874-4ed6-94ef-f5e0616f7026-kube-api-access-wp7sj\") pod \"simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd\" (UID: \"8a094926-e874-4ed6-94ef-f5e0616f7026\") " pod="e2e-models-e2e-worker-w4-fc27a5/simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd" Sep 16 07:48:16.769761 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:16.769726 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a094926-e874-4ed6-94ef-f5e0616f7026-tls-certs\") pod \"simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd\" (UID: \"8a094926-e874-4ed6-94ef-f5e0616f7026\") " pod="e2e-models-e2e-worker-w4-fc27a5/simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd" Sep 16 07:48:16.769761 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:16.769748 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a094926-e874-4ed6-94ef-f5e0616f7026-home\") pod \"simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd\" (UID: \"8a094926-e874-4ed6-94ef-f5e0616f7026\") " pod="e2e-models-e2e-worker-w4-fc27a5/simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd" Sep 16 07:48:16.770051 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:16.770028 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a094926-e874-4ed6-94ef-f5e0616f7026-tmp-dir\") pod \"simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd\" (UID: \"8a094926-e874-4ed6-94ef-f5e0616f7026\") " pod="e2e-models-e2e-worker-w4-fc27a5/simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd" Sep 16 07:48:16.770129 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:16.770064 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a094926-e874-4ed6-94ef-f5e0616f7026-model-cache\") pod \"simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd\" (UID: \"8a094926-e874-4ed6-94ef-f5e0616f7026\") " pod="e2e-models-e2e-worker-w4-fc27a5/simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd" Sep 16 07:48:16.770178 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:16.770133 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a094926-e874-4ed6-94ef-f5e0616f7026-home\") pod \"simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd\" (UID: \"8a094926-e874-4ed6-94ef-f5e0616f7026\") " pod="e2e-models-e2e-worker-w4-fc27a5/simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd" Sep 16 07:48:16.772137 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:16.772080 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a094926-e874-4ed6-94ef-f5e0616f7026-dshm\") pod \"simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd\" (UID: \"8a094926-e874-4ed6-94ef-f5e0616f7026\") " pod="e2e-models-e2e-worker-w4-fc27a5/simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd" Sep 16 07:48:16.772341 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:16.772322 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a094926-e874-4ed6-94ef-f5e0616f7026-tls-certs\") pod \"simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd\" (UID: \"8a094926-e874-4ed6-94ef-f5e0616f7026\") " pod="e2e-models-e2e-worker-w4-fc27a5/simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd" Sep 16 07:48:16.778013 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:16.777961 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp7sj\" (UniqueName: \"kubernetes.io/projected/8a094926-e874-4ed6-94ef-f5e0616f7026-kube-api-access-wp7sj\") pod \"simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd\" (UID: \"8a094926-e874-4ed6-94ef-f5e0616f7026\") " pod="e2e-models-e2e-worker-w4-fc27a5/simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd" Sep 16 07:48:16.779648 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:16.779613 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:48:16.923413 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:16.923377 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fc27a5/simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd" Sep 16 07:48:17.205112 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:17.205071 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-fc27a5/simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd"] Sep 16 07:48:17.206666 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:48:17.206639 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a094926_e874_4ed6_94ef_f5e0616f7026.slice/crio-5a8e5e23fdcb8ef86174aee22b5119e3161b08b8835d931b05f4b60688e64656 WatchSource:0}: Error finding container 5a8e5e23fdcb8ef86174aee22b5119e3161b08b8835d931b05f4b60688e64656: Status 404 returned error can't find the container with id 5a8e5e23fdcb8ef86174aee22b5119e3161b08b8835d931b05f4b60688e64656 Sep 16 07:48:17.685402 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:17.685362 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:48:17.861976 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:17.861938 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fc27a5/simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd" event={"ID":"8a094926-e874-4ed6-94ef-f5e0616f7026","Type":"ContainerStarted","Data":"fae799527d6331b34872566431319d4686f8bd4dfe6baad0adec4b44888b2e35"} Sep 16 07:48:17.862217 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:17.861988 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fc27a5/simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd" event={"ID":"8a094926-e874-4ed6-94ef-f5e0616f7026","Type":"ContainerStarted","Data":"5a8e5e23fdcb8ef86174aee22b5119e3161b08b8835d931b05f4b60688e64656"} Sep 16 07:48:17.879481 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:17.879419 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-fc27a5/simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd" podStartSLOduration=1.879398527 podStartE2EDuration="1.879398527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:48:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:48:17.877412022 +0000 UTC m=+869.461190360" watchObservedRunningTime="2026-09-16 07:48:17.879398527 +0000 UTC m=+869.463176921" Sep 16 07:48:18.337700 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:18.337662 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:48:19.045813 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:19.045775 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:48:19.671316 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:19.671279 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:48:20.410872 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:20.410830 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:48:21.049701 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:21.049663 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:48:21.799303 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:21.799261 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:48:26.924156 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:26.924088 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-fc27a5/simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd" Sep 16 07:48:26.924156 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:26.924159 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-fc27a5/simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd" Sep 16 07:48:26.938328 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:26.938296 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-fc27a5/simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd" Sep 16 07:48:27.926328 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:27.926296 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-fc27a5/simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd" Sep 16 07:48:28.545182 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:28.545139 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:48:28.644863 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:28.644826 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:48:28.745799 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:28.745764 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:48:28.817591 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:28.817503 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:48:29.018423 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:29.018392 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:48:29.192837 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:29.192790 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:48:29.409282 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:29.409243 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:48:29.707774 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:29.707731 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:48:29.895791 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:29.895757 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:48:30.043375 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:30.043285 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:48:30.861547 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:30.861511 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:48:31.735987 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:31.735952 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:48:32.152729 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:32.152640 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:48:32.249851 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:32.249756 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:48:32.559206 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:32.559172 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:48:33.414016 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:33.413978 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:48:34.253038 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:34.252968 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:48:34.354263 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:34.354226 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:48:34.457295 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:34.457260 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:48:34.708919 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:34.708887 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:48:34.813506 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:34.813464 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:48:35.137391 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:35.137284 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:48:35.559374 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:35.559329 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:48:35.901487 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:35.901394 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:48:36.557327 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:36.557287 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:48:37.237847 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:37.237807 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:48:45.981642 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:45.981552 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:48:56.167332 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:56.167286 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:48:56.853033 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:56.852994 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:48:57.556635 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:57.556601 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:48:57.863969 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:57.863877 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4d9f69e4-crjhf"] Sep 16 07:48:57.867996 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:57.867975 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4d9f69e4-crjhf" Sep 16 07:48:57.870326 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:57.870298 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zwk5r\"" Sep 16 07:48:57.870500 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:57.870298 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 07:48:57.876468 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:57.876435 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4d9f69e4-crjhf"] Sep 16 07:48:57.992813 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:57.992770 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e75cf51-1560-4aac-b9fc-9d46d7e36179-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4d9f69e4-crjhf\" (UID: \"0e75cf51-1560-4aac-b9fc-9d46d7e36179\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4d9f69e4-crjhf" Sep 16 07:48:58.093812 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:58.093765 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e75cf51-1560-4aac-b9fc-9d46d7e36179-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4d9f69e4-crjhf\" (UID: \"0e75cf51-1560-4aac-b9fc-9d46d7e36179\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4d9f69e4-crjhf" Sep 16 07:48:58.094455 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:58.094433 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e75cf51-1560-4aac-b9fc-9d46d7e36179-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-4d9f69e4-crjhf\" (UID: \"0e75cf51-1560-4aac-b9fc-9d46d7e36179\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4d9f69e4-crjhf" Sep 16 07:48:58.184973 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:58.184937 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4d9f69e4-crjhf" Sep 16 07:48:58.381478 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:58.381437 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:48:58.462843 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:58.462749 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4d9f69e4-crjhf"] Sep 16 07:48:58.466463 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:48:58.466426 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e75cf51_1560_4aac_b9fc_9d46d7e36179.slice/crio-2d2f842a7aa078ea5716f5d80e93af5a7639e716e54aa9024a4a2b6373d3254e WatchSource:0}: Error finding container 2d2f842a7aa078ea5716f5d80e93af5a7639e716e54aa9024a4a2b6373d3254e: Status 404 returned error can't find the container with id 2d2f842a7aa078ea5716f5d80e93af5a7639e716e54aa9024a4a2b6373d3254e Sep 16 07:48:59.089797 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:59.089755 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4d9f69e4-crjhf" event={"ID":"0e75cf51-1560-4aac-b9fc-9d46d7e36179","Type":"ContainerStarted","Data":"4670774efd0738f98f9517a32b71275f3e79af375210f6833f223dd68008a8ac"} Sep 16 07:48:59.090272 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:59.089820 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4d9f69e4-crjhf" event={"ID":"0e75cf51-1560-4aac-b9fc-9d46d7e36179","Type":"ContainerStarted","Data":"2d2f842a7aa078ea5716f5d80e93af5a7639e716e54aa9024a4a2b6373d3254e"} Sep 16 07:48:59.107326 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:59.107274 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4d9f69e4-crjhf" podStartSLOduration=2.107254444 podStartE2EDuration="2.107254444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:48:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:48:59.104837837 +0000 UTC m=+910.688616174" watchObservedRunningTime="2026-09-16 07:48:59.107254444 +0000 UTC m=+910.691032781" Sep 16 07:48:59.173674 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:59.173639 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:48:59.833712 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:48:59.833668 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:00.097072 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:00.096981 2552 generic.go:356] "Generic (PLEG): container finished" podID="0e75cf51-1560-4aac-b9fc-9d46d7e36179" containerID="4670774efd0738f98f9517a32b71275f3e79af375210f6833f223dd68008a8ac" exitCode=0 Sep 16 07:49:00.097072 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:00.097046 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4d9f69e4-crjhf" event={"ID":"0e75cf51-1560-4aac-b9fc-9d46d7e36179","Type":"ContainerDied","Data":"4670774efd0738f98f9517a32b71275f3e79af375210f6833f223dd68008a8ac"} Sep 16 07:49:00.546959 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:00.546919 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:00.884818 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:00.884728 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:01.252587 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:01.252554 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:01.254054 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:01.254032 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4d9f69e4-crjhf" Sep 16 07:49:01.425033 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:01.424996 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e75cf51-1560-4aac-b9fc-9d46d7e36179-maas-api-service-ca\") pod \"0e75cf51-1560-4aac-b9fc-9d46d7e36179\" (UID: \"0e75cf51-1560-4aac-b9fc-9d46d7e36179\") " Sep 16 07:49:01.425518 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:01.425459 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e75cf51-1560-4aac-b9fc-9d46d7e36179-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0e75cf51-1560-4aac-b9fc-9d46d7e36179" (UID: "0e75cf51-1560-4aac-b9fc-9d46d7e36179"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:49:01.526420 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:01.526371 2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e75cf51-1560-4aac-b9fc-9d46d7e36179-maas-api-service-ca\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:49:01.972904 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:01.972864 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:02.109865 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:02.109825 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4d9f69e4-crjhf" event={"ID":"0e75cf51-1560-4aac-b9fc-9d46d7e36179","Type":"ContainerDied","Data":"2d2f842a7aa078ea5716f5d80e93af5a7639e716e54aa9024a4a2b6373d3254e"} Sep 16 07:49:02.110039 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:02.109891 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d2f842a7aa078ea5716f5d80e93af5a7639e716e54aa9024a4a2b6373d3254e" Sep 16 07:49:02.110039 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:02.109836 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4d9f69e4-crjhf" Sep 16 07:49:02.770431 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:02.770388 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:03.641925 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:03.641886 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:04.219372 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:04.219336 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:04.344904 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:04.344868 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:04.407696 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:04.407645 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:04.474861 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:04.474765 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4d9f69e4-crjhf"] Sep 16 07:49:04.476839 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:04.476808 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4d9f69e4-crjhf"] Sep 16 07:49:04.477614 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:04.477586 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:04.543623 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:04.543584 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:04.890045 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:04.889958 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:04.938828 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:04.938788 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e75cf51-1560-4aac-b9fc-9d46d7e36179" path="/var/lib/kubelet/pods/0e75cf51-1560-4aac-b9fc-9d46d7e36179/volumes" Sep 16 07:49:05.640329 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:05.640298 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:05.779584 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:05.779555 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:06.061005 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:06.060951 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:06.380969 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:06.380870 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:06.682701 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:06.682669 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:07.810446 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:07.810411 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:08.414821 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:08.414783 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:09.254757 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:09.254707 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:09.464094 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:09.464060 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:09.564889 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:09.564792 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:10.010970 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:10.010936 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:10.878942 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:10.878904 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:10.938789 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:10.938745 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:11.011707 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:11.011673 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:11.358071 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:11.358030 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:11.440563 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:11.440524 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:11.816017 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:11.815976 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:12.494274 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:12.494230 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:13.288994 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:13.288960 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:13.910036 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:13.910003 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:14.782467 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:14.782429 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:18.356109 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:18.356068 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:19.039957 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:19.039921 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:19.068576 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:19.068541 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-b60434-74756d459-7gzs2"] Sep 16 07:49:19.068824 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:19.068800 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-b60434-74756d459-7gzs2" podUID="1cdff8ad-c269-4f9e-9ddc-31b59467c541" containerName="payload-processing" containerID="cri-o://a00c1d1def0d3c16fc07eefd0ac5c63d9332661ce4d6f87eb025b99fcbbbbd78" gracePeriod=30 Sep 16 07:49:19.109547 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:19.108179 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:19.175226 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:19.175178 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:19.202491 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:19.202453 2552 generic.go:356] "Generic (PLEG): container finished" podID="1cdff8ad-c269-4f9e-9ddc-31b59467c541" containerID="a00c1d1def0d3c16fc07eefd0ac5c63d9332661ce4d6f87eb025b99fcbbbbd78" exitCode=0 Sep 16 07:49:19.202666 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:19.202531 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-b60434-74756d459-7gzs2" event={"ID":"1cdff8ad-c269-4f9e-9ddc-31b59467c541","Type":"ContainerDied","Data":"a00c1d1def0d3c16fc07eefd0ac5c63d9332661ce4d6f87eb025b99fcbbbbd78"} Sep 16 07:49:19.240224 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:19.240169 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:19.307772 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:19.307524 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:19.333957 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:19.333925 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-b60434-74756d459-7gzs2" Sep 16 07:49:19.392674 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:19.392635 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cdff8ad-c269-4f9e-9ddc-31b59467c541-plugins-config-volume\") pod \"1cdff8ad-c269-4f9e-9ddc-31b59467c541\" (UID: \"1cdff8ad-c269-4f9e-9ddc-31b59467c541\") " Sep 16 07:49:19.393167 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:19.392694 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vbm2\" (UniqueName: \"kubernetes.io/projected/1cdff8ad-c269-4f9e-9ddc-31b59467c541-kube-api-access-2vbm2\") pod \"1cdff8ad-c269-4f9e-9ddc-31b59467c541\" (UID: \"1cdff8ad-c269-4f9e-9ddc-31b59467c541\") " Sep 16 07:49:19.393167 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:19.393148 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cdff8ad-c269-4f9e-9ddc-31b59467c541-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1cdff8ad-c269-4f9e-9ddc-31b59467c541" (UID: "1cdff8ad-c269-4f9e-9ddc-31b59467c541"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:49:19.395007 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:19.394981 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cdff8ad-c269-4f9e-9ddc-31b59467c541-kube-api-access-2vbm2" (OuterVolumeSpecName: "kube-api-access-2vbm2") pod "1cdff8ad-c269-4f9e-9ddc-31b59467c541" (UID: "1cdff8ad-c269-4f9e-9ddc-31b59467c541"). InnerVolumeSpecName "kube-api-access-2vbm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:49:19.493878 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:19.493829 2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1cdff8ad-c269-4f9e-9ddc-31b59467c541-plugins-config-volume\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:49:19.493878 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:19.493870 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vbm2\" (UniqueName: \"kubernetes.io/projected/1cdff8ad-c269-4f9e-9ddc-31b59467c541-kube-api-access-2vbm2\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:49:19.618988 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:19.618889 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:19.680524 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:19.680476 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:20.209614 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:20.209585 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-b60434-74756d459-7gzs2" Sep 16 07:49:20.209614 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:20.209607 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-b60434-74756d459-7gzs2" event={"ID":"1cdff8ad-c269-4f9e-9ddc-31b59467c541","Type":"ContainerDied","Data":"3041be73dae4513d6b23f0cf52226382448326fd01c0da8410303018869730a3"} Sep 16 07:49:20.209864 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:20.209663 2552 scope.go:117] "RemoveContainer" containerID="a00c1d1def0d3c16fc07eefd0ac5c63d9332661ce4d6f87eb025b99fcbbbbd78" Sep 16 07:49:20.231344 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:20.231301 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-b60434-74756d459-7gzs2"] Sep 16 07:49:20.233239 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:20.233217 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-b60434-74756d459-7gzs2"] Sep 16 07:49:20.464479 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:20.464384 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:20.937586 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:20.937553 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1cdff8ad-c269-4f9e-9ddc-31b59467c541" path="/var/lib/kubelet/pods/1cdff8ad-c269-4f9e-9ddc-31b59467c541/volumes" Sep 16 07:49:21.293214 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:21.293129 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:21.979273 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:21.979236 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:22.843409 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:22.843377 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:23.556309 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:23.556259 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:24.576789 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:24.576750 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:24.668216 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:24.668179 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:25.252594 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:25.252541 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:25.906942 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:25.906908 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:26.679112 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:26.679055 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:27.336726 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:27.336685 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:28.165274 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:28.165241 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:31.757837 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:31.757795 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:34.552346 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:34.552302 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:34.627053 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:34.627014 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:34.688606 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:34.688573 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:34.749621 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:34.749580 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:34.814617 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:34.814532 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:34.893496 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:34.893453 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:35.166771 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:35.166679 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:35.551004 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:35.550963 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:35.650814 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:35.650777 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:35.790211 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:35.790178 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:36.220051 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:36.220014 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:37.126928 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:37.126897 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:37.226795 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:37.226752 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:37.921985 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:37.921951 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:38.585272 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:38.585242 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:41.786875 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:41.786829 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:43.041691 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:43.041655 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:43.107483 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:43.107449 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:43.197274 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:43.197239 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:43.295574 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:43.295488 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:43.395491 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:43.395458 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:43.723047 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:43.723010 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:49:58.660823 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:49:58.660780 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:50:01.657166 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:01.657115 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:50:01.858109 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:01.858057 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l"] Sep 16 07:50:01.858322 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:01.858298 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" podUID="56ccdf02-0887-408b-bee7-26f99d1a1c7b" containerName="istio-proxy" containerID="cri-o://7a6d86c817d6b88ab7334d516088cdc8f7ad02c6165820956512020ef1d90ceb" gracePeriod=30 Sep 16 07:50:01.879067 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:01.879032 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:50:02.296421 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:02.296382 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:50:02.615257 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:02.615154 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:50:02.716630 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:02.716599 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:50:04.658005 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:04.657969 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:50:05.302060 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:05.302020 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:50:05.394264 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:05.394232 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:50:05.495055 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:05.495017 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:50:06.292441 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:06.292399 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:50:07.057518 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:07.057420 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:50:07.121969 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:07.121898 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" Sep 16 07:50:07.179403 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:07.179359 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56ccdf02-0887-408b-bee7-26f99d1a1c7b-workload-certs\") pod \"56ccdf02-0887-408b-bee7-26f99d1a1c7b\" (UID: \"56ccdf02-0887-408b-bee7-26f99d1a1c7b\") " Sep 16 07:50:07.179403 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:07.179410 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56ccdf02-0887-408b-bee7-26f99d1a1c7b-istio-token\") pod \"56ccdf02-0887-408b-bee7-26f99d1a1c7b\" (UID: \"56ccdf02-0887-408b-bee7-26f99d1a1c7b\") " Sep 16 07:50:07.179665 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:07.179428 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56ccdf02-0887-408b-bee7-26f99d1a1c7b-istio-podinfo\") pod \"56ccdf02-0887-408b-bee7-26f99d1a1c7b\" (UID: \"56ccdf02-0887-408b-bee7-26f99d1a1c7b\") " Sep 16 07:50:07.179665 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:07.179459 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ccdf02-0887-408b-bee7-26f99d1a1c7b-credential-socket\") pod \"56ccdf02-0887-408b-bee7-26f99d1a1c7b\" (UID: \"56ccdf02-0887-408b-bee7-26f99d1a1c7b\") " Sep 16 07:50:07.179665 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:07.179491 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56ccdf02-0887-408b-bee7-26f99d1a1c7b-istio-data\") pod \"56ccdf02-0887-408b-bee7-26f99d1a1c7b\" (UID: \"56ccdf02-0887-408b-bee7-26f99d1a1c7b\") " Sep 16 07:50:07.179665 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:07.179555 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56ccdf02-0887-408b-bee7-26f99d1a1c7b-istio-envoy\") pod \"56ccdf02-0887-408b-bee7-26f99d1a1c7b\" (UID: \"56ccdf02-0887-408b-bee7-26f99d1a1c7b\") " Sep 16 07:50:07.179665 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:07.179634 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56ccdf02-0887-408b-bee7-26f99d1a1c7b-istiod-ca-cert\") pod \"56ccdf02-0887-408b-bee7-26f99d1a1c7b\" (UID: \"56ccdf02-0887-408b-bee7-26f99d1a1c7b\") " Sep 16 07:50:07.179933 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:07.179668 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6tdhx\" (UniqueName: \"kubernetes.io/projected/56ccdf02-0887-408b-bee7-26f99d1a1c7b-kube-api-access-6tdhx\") pod \"56ccdf02-0887-408b-bee7-26f99d1a1c7b\" (UID: \"56ccdf02-0887-408b-bee7-26f99d1a1c7b\") " Sep 16 07:50:07.179933 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:07.179690 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ccdf02-0887-408b-bee7-26f99d1a1c7b-workload-socket\") pod \"56ccdf02-0887-408b-bee7-26f99d1a1c7b\" (UID: \"56ccdf02-0887-408b-bee7-26f99d1a1c7b\") " Sep 16 07:50:07.179933 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:07.179727 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56ccdf02-0887-408b-bee7-26f99d1a1c7b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "56ccdf02-0887-408b-bee7-26f99d1a1c7b" (UID: "56ccdf02-0887-408b-bee7-26f99d1a1c7b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:50:07.180299 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:07.179967 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56ccdf02-0887-408b-bee7-26f99d1a1c7b-istio-data" (OuterVolumeSpecName: "istio-data") pod "56ccdf02-0887-408b-bee7-26f99d1a1c7b" (UID: "56ccdf02-0887-408b-bee7-26f99d1a1c7b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:50:07.180412 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:07.180116 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56ccdf02-0887-408b-bee7-26f99d1a1c7b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "56ccdf02-0887-408b-bee7-26f99d1a1c7b" (UID: "56ccdf02-0887-408b-bee7-26f99d1a1c7b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:50:07.180412 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:07.180175 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56ccdf02-0887-408b-bee7-26f99d1a1c7b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "56ccdf02-0887-408b-bee7-26f99d1a1c7b" (UID: "56ccdf02-0887-408b-bee7-26f99d1a1c7b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:50:07.180412 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:07.179991 2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56ccdf02-0887-408b-bee7-26f99d1a1c7b-workload-certs\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:50:07.180412 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:07.180198 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56ccdf02-0887-408b-bee7-26f99d1a1c7b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "56ccdf02-0887-408b-bee7-26f99d1a1c7b" (UID: "56ccdf02-0887-408b-bee7-26f99d1a1c7b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:50:07.181876 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:07.181845 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56ccdf02-0887-408b-bee7-26f99d1a1c7b-istio-token" (OuterVolumeSpecName: "istio-token") pod "56ccdf02-0887-408b-bee7-26f99d1a1c7b" (UID: "56ccdf02-0887-408b-bee7-26f99d1a1c7b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:50:07.181981 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:07.181925 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56ccdf02-0887-408b-bee7-26f99d1a1c7b-kube-api-access-6tdhx" (OuterVolumeSpecName: "kube-api-access-6tdhx") pod "56ccdf02-0887-408b-bee7-26f99d1a1c7b" (UID: "56ccdf02-0887-408b-bee7-26f99d1a1c7b"). InnerVolumeSpecName "kube-api-access-6tdhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:50:07.182044 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:07.181971 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56ccdf02-0887-408b-bee7-26f99d1a1c7b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "56ccdf02-0887-408b-bee7-26f99d1a1c7b" (UID: "56ccdf02-0887-408b-bee7-26f99d1a1c7b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:50:07.182151 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:07.182135 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56ccdf02-0887-408b-bee7-26f99d1a1c7b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "56ccdf02-0887-408b-bee7-26f99d1a1c7b" (UID: "56ccdf02-0887-408b-bee7-26f99d1a1c7b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 07:50:07.281266 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:07.281214 2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56ccdf02-0887-408b-bee7-26f99d1a1c7b-istiod-ca-cert\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:50:07.281266 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:07.281256 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tdhx\" (UniqueName: \"kubernetes.io/projected/56ccdf02-0887-408b-bee7-26f99d1a1c7b-kube-api-access-6tdhx\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:50:07.281266 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:07.281272 2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ccdf02-0887-408b-bee7-26f99d1a1c7b-workload-socket\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:50:07.281526 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:07.281286 2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56ccdf02-0887-408b-bee7-26f99d1a1c7b-istio-token\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:50:07.281526 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:07.281298 2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56ccdf02-0887-408b-bee7-26f99d1a1c7b-istio-podinfo\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:50:07.281526 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:07.281308 2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56ccdf02-0887-408b-bee7-26f99d1a1c7b-credential-socket\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:50:07.281526 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:07.281320 2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56ccdf02-0887-408b-bee7-26f99d1a1c7b-istio-data\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:50:07.281526 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:07.281331 2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56ccdf02-0887-408b-bee7-26f99d1a1c7b-istio-envoy\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:50:07.460595 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:07.460559 2552 generic.go:356] "Generic (PLEG): container finished" podID="56ccdf02-0887-408b-bee7-26f99d1a1c7b" containerID="7a6d86c817d6b88ab7334d516088cdc8f7ad02c6165820956512020ef1d90ceb" exitCode=0 Sep 16 07:50:07.461042 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:07.460608 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" Sep 16 07:50:07.461042 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:07.460625 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" event={"ID":"56ccdf02-0887-408b-bee7-26f99d1a1c7b","Type":"ContainerDied","Data":"7a6d86c817d6b88ab7334d516088cdc8f7ad02c6165820956512020ef1d90ceb"} Sep 16 07:50:07.461042 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:07.460672 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l" event={"ID":"56ccdf02-0887-408b-bee7-26f99d1a1c7b","Type":"ContainerDied","Data":"3e34ce9fa28b0c937d21e8a53dd9354f4e40ac52724534bdc911d7a9f8384719"} Sep 16 07:50:07.461042 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:07.460699 2552 scope.go:117] "RemoveContainer" containerID="7a6d86c817d6b88ab7334d516088cdc8f7ad02c6165820956512020ef1d90ceb" Sep 16 07:50:07.470949 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:07.470925 2552 scope.go:117] "RemoveContainer" containerID="6aef8e3374e599eb8898cf1e661cf794731beed0f79011e36481a42a2accbb2b" Sep 16 07:50:07.480176 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:07.480151 2552 scope.go:117] "RemoveContainer" containerID="7a6d86c817d6b88ab7334d516088cdc8f7ad02c6165820956512020ef1d90ceb" Sep 16 07:50:07.480539 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:50:07.480516 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a6d86c817d6b88ab7334d516088cdc8f7ad02c6165820956512020ef1d90ceb\": container with ID starting with 7a6d86c817d6b88ab7334d516088cdc8f7ad02c6165820956512020ef1d90ceb not found: ID does not exist" containerID="7a6d86c817d6b88ab7334d516088cdc8f7ad02c6165820956512020ef1d90ceb" Sep 16 07:50:07.480598 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:07.480550 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a6d86c817d6b88ab7334d516088cdc8f7ad02c6165820956512020ef1d90ceb"} err="failed to get container status \"7a6d86c817d6b88ab7334d516088cdc8f7ad02c6165820956512020ef1d90ceb\": rpc error: code = NotFound desc = could not find container \"7a6d86c817d6b88ab7334d516088cdc8f7ad02c6165820956512020ef1d90ceb\": container with ID starting with 7a6d86c817d6b88ab7334d516088cdc8f7ad02c6165820956512020ef1d90ceb not found: ID does not exist" Sep 16 07:50:07.480598 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:07.480570 2552 scope.go:117] "RemoveContainer" containerID="6aef8e3374e599eb8898cf1e661cf794731beed0f79011e36481a42a2accbb2b" Sep 16 07:50:07.480863 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:50:07.480846 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6aef8e3374e599eb8898cf1e661cf794731beed0f79011e36481a42a2accbb2b\": container with ID starting with 6aef8e3374e599eb8898cf1e661cf794731beed0f79011e36481a42a2accbb2b not found: ID does not exist" containerID="6aef8e3374e599eb8898cf1e661cf794731beed0f79011e36481a42a2accbb2b" Sep 16 07:50:07.480903 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:07.480871 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aef8e3374e599eb8898cf1e661cf794731beed0f79011e36481a42a2accbb2b"} err="failed to get container status \"6aef8e3374e599eb8898cf1e661cf794731beed0f79011e36481a42a2accbb2b\": rpc error: code = NotFound desc = could not find container \"6aef8e3374e599eb8898cf1e661cf794731beed0f79011e36481a42a2accbb2b\": container with ID starting with 6aef8e3374e599eb8898cf1e661cf794731beed0f79011e36481a42a2accbb2b not found: ID does not exist" Sep 16 07:50:07.481976 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:07.481958 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l"] Sep 16 07:50:07.485700 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:07.485674 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-b60434-openshift-default-64fbf588d4-s956l"] Sep 16 07:50:07.831420 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:07.831309 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:50:08.577687 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:08.577650 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:50:08.643700 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:08.643661 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:50:08.755815 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:08.755776 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:50:08.937508 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:08.937470 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56ccdf02-0887-408b-bee7-26f99d1a1c7b" path="/var/lib/kubelet/pods/56ccdf02-0887-408b-bee7-26f99d1a1c7b/volumes" Sep 16 07:50:09.035548 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:09.035516 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:50:09.125795 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:09.125758 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:50:09.579252 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:09.579221 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:50:15.537670 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:15.537571 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:50:15.600581 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:15.600545 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:50:15.683742 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:15.683709 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:50:15.784952 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:15.784919 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:50:16.103840 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:16.103795 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:50:28.366432 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:28.366330 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-c9020d/premium-w0-c9020d-kserve-78578c546b-8zspq"] Sep 16 07:50:28.366958 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:28.366935 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1cdff8ad-c269-4f9e-9ddc-31b59467c541" containerName="payload-processing" Sep 16 07:50:28.367046 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:28.366961 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="1cdff8ad-c269-4f9e-9ddc-31b59467c541" containerName="payload-processing" Sep 16 07:50:28.367046 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:28.366985 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56ccdf02-0887-408b-bee7-26f99d1a1c7b" containerName="istio-proxy" Sep 16 07:50:28.367046 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:28.366994 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="56ccdf02-0887-408b-bee7-26f99d1a1c7b" containerName="istio-proxy" Sep 16 07:50:28.367046 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:28.367017 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e75cf51-1560-4aac-b9fc-9d46d7e36179" containerName="revoke-keys" Sep 16 07:50:28.367046 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:28.367025 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e75cf51-1560-4aac-b9fc-9d46d7e36179" containerName="revoke-keys" Sep 16 07:50:28.367305 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:28.367054 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56ccdf02-0887-408b-bee7-26f99d1a1c7b" containerName="istio-proxy" Sep 16 07:50:28.367305 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:28.367062 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="56ccdf02-0887-408b-bee7-26f99d1a1c7b" containerName="istio-proxy" Sep 16 07:50:28.367305 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:28.367169 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e75cf51-1560-4aac-b9fc-9d46d7e36179" containerName="revoke-keys" Sep 16 07:50:28.367305 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:28.367189 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="56ccdf02-0887-408b-bee7-26f99d1a1c7b" containerName="istio-proxy" Sep 16 07:50:28.367305 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:28.367199 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="1cdff8ad-c269-4f9e-9ddc-31b59467c541" containerName="payload-processing" Sep 16 07:50:28.370269 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:28.370247 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c9020d/premium-w0-c9020d-kserve-78578c546b-8zspq" Sep 16 07:50:28.372520 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:28.372493 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c9020d\"/\"premium-w0-c9020d-kserve-self-signed-certs\"" Sep 16 07:50:28.381085 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:28.381057 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c9020d/premium-w0-c9020d-kserve-78578c546b-8zspq"] Sep 16 07:50:28.388234 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:28.388195 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4543c78d-4a9b-437c-8146-89e1eed8b7a8-tls-certs\") pod \"premium-w0-c9020d-kserve-78578c546b-8zspq\" (UID: \"4543c78d-4a9b-437c-8146-89e1eed8b7a8\") " pod="e2e-models-e2e-worker-w0-c9020d/premium-w0-c9020d-kserve-78578c546b-8zspq" Sep 16 07:50:28.388379 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:28.388251 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4543c78d-4a9b-437c-8146-89e1eed8b7a8-model-cache\") pod \"premium-w0-c9020d-kserve-78578c546b-8zspq\" (UID: \"4543c78d-4a9b-437c-8146-89e1eed8b7a8\") " pod="e2e-models-e2e-worker-w0-c9020d/premium-w0-c9020d-kserve-78578c546b-8zspq" Sep 16 07:50:28.388379 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:28.388305 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4543c78d-4a9b-437c-8146-89e1eed8b7a8-tmp-dir\") pod \"premium-w0-c9020d-kserve-78578c546b-8zspq\" (UID: \"4543c78d-4a9b-437c-8146-89e1eed8b7a8\") " pod="e2e-models-e2e-worker-w0-c9020d/premium-w0-c9020d-kserve-78578c546b-8zspq" Sep 16 07:50:28.388379 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:28.388345 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l68g7\" (UniqueName: \"kubernetes.io/projected/4543c78d-4a9b-437c-8146-89e1eed8b7a8-kube-api-access-l68g7\") pod \"premium-w0-c9020d-kserve-78578c546b-8zspq\" (UID: \"4543c78d-4a9b-437c-8146-89e1eed8b7a8\") " pod="e2e-models-e2e-worker-w0-c9020d/premium-w0-c9020d-kserve-78578c546b-8zspq" Sep 16 07:50:28.388379 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:28.388370 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4543c78d-4a9b-437c-8146-89e1eed8b7a8-home\") pod \"premium-w0-c9020d-kserve-78578c546b-8zspq\" (UID: \"4543c78d-4a9b-437c-8146-89e1eed8b7a8\") " pod="e2e-models-e2e-worker-w0-c9020d/premium-w0-c9020d-kserve-78578c546b-8zspq" Sep 16 07:50:28.388529 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:28.388387 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4543c78d-4a9b-437c-8146-89e1eed8b7a8-dshm\") pod \"premium-w0-c9020d-kserve-78578c546b-8zspq\" (UID: \"4543c78d-4a9b-437c-8146-89e1eed8b7a8\") " pod="e2e-models-e2e-worker-w0-c9020d/premium-w0-c9020d-kserve-78578c546b-8zspq" Sep 16 07:50:28.400343 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:28.400307 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:50:28.488977 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:28.488931 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l68g7\" (UniqueName: \"kubernetes.io/projected/4543c78d-4a9b-437c-8146-89e1eed8b7a8-kube-api-access-l68g7\") pod \"premium-w0-c9020d-kserve-78578c546b-8zspq\" (UID: \"4543c78d-4a9b-437c-8146-89e1eed8b7a8\") " pod="e2e-models-e2e-worker-w0-c9020d/premium-w0-c9020d-kserve-78578c546b-8zspq" Sep 16 07:50:28.488977 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:28.488979 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4543c78d-4a9b-437c-8146-89e1eed8b7a8-home\") pod \"premium-w0-c9020d-kserve-78578c546b-8zspq\" (UID: \"4543c78d-4a9b-437c-8146-89e1eed8b7a8\") " pod="e2e-models-e2e-worker-w0-c9020d/premium-w0-c9020d-kserve-78578c546b-8zspq" Sep 16 07:50:28.489295 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:28.488995 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4543c78d-4a9b-437c-8146-89e1eed8b7a8-dshm\") pod \"premium-w0-c9020d-kserve-78578c546b-8zspq\" (UID: \"4543c78d-4a9b-437c-8146-89e1eed8b7a8\") " pod="e2e-models-e2e-worker-w0-c9020d/premium-w0-c9020d-kserve-78578c546b-8zspq" Sep 16 07:50:28.489295 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:28.489072 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4543c78d-4a9b-437c-8146-89e1eed8b7a8-tls-certs\") pod \"premium-w0-c9020d-kserve-78578c546b-8zspq\" (UID: \"4543c78d-4a9b-437c-8146-89e1eed8b7a8\") " pod="e2e-models-e2e-worker-w0-c9020d/premium-w0-c9020d-kserve-78578c546b-8zspq" Sep 16 07:50:28.489295 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:28.489094 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4543c78d-4a9b-437c-8146-89e1eed8b7a8-model-cache\") pod \"premium-w0-c9020d-kserve-78578c546b-8zspq\" (UID: \"4543c78d-4a9b-437c-8146-89e1eed8b7a8\") " pod="e2e-models-e2e-worker-w0-c9020d/premium-w0-c9020d-kserve-78578c546b-8zspq" Sep 16 07:50:28.489295 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:28.489148 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4543c78d-4a9b-437c-8146-89e1eed8b7a8-tmp-dir\") pod \"premium-w0-c9020d-kserve-78578c546b-8zspq\" (UID: \"4543c78d-4a9b-437c-8146-89e1eed8b7a8\") " pod="e2e-models-e2e-worker-w0-c9020d/premium-w0-c9020d-kserve-78578c546b-8zspq" Sep 16 07:50:28.489501 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:28.489443 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4543c78d-4a9b-437c-8146-89e1eed8b7a8-home\") pod \"premium-w0-c9020d-kserve-78578c546b-8zspq\" (UID: \"4543c78d-4a9b-437c-8146-89e1eed8b7a8\") " pod="e2e-models-e2e-worker-w0-c9020d/premium-w0-c9020d-kserve-78578c546b-8zspq" Sep 16 07:50:28.489542 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:28.489493 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4543c78d-4a9b-437c-8146-89e1eed8b7a8-model-cache\") pod \"premium-w0-c9020d-kserve-78578c546b-8zspq\" (UID: \"4543c78d-4a9b-437c-8146-89e1eed8b7a8\") " pod="e2e-models-e2e-worker-w0-c9020d/premium-w0-c9020d-kserve-78578c546b-8zspq" Sep 16 07:50:28.489542 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:28.489524 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4543c78d-4a9b-437c-8146-89e1eed8b7a8-tmp-dir\") pod \"premium-w0-c9020d-kserve-78578c546b-8zspq\" (UID: \"4543c78d-4a9b-437c-8146-89e1eed8b7a8\") " pod="e2e-models-e2e-worker-w0-c9020d/premium-w0-c9020d-kserve-78578c546b-8zspq" Sep 16 07:50:28.491428 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:28.491410 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4543c78d-4a9b-437c-8146-89e1eed8b7a8-dshm\") pod \"premium-w0-c9020d-kserve-78578c546b-8zspq\" (UID: \"4543c78d-4a9b-437c-8146-89e1eed8b7a8\") " pod="e2e-models-e2e-worker-w0-c9020d/premium-w0-c9020d-kserve-78578c546b-8zspq" Sep 16 07:50:28.491550 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:28.491533 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4543c78d-4a9b-437c-8146-89e1eed8b7a8-tls-certs\") pod \"premium-w0-c9020d-kserve-78578c546b-8zspq\" (UID: \"4543c78d-4a9b-437c-8146-89e1eed8b7a8\") " pod="e2e-models-e2e-worker-w0-c9020d/premium-w0-c9020d-kserve-78578c546b-8zspq" Sep 16 07:50:28.497847 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:28.497817 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l68g7\" (UniqueName: \"kubernetes.io/projected/4543c78d-4a9b-437c-8146-89e1eed8b7a8-kube-api-access-l68g7\") pod \"premium-w0-c9020d-kserve-78578c546b-8zspq\" (UID: \"4543c78d-4a9b-437c-8146-89e1eed8b7a8\") " pod="e2e-models-e2e-worker-w0-c9020d/premium-w0-c9020d-kserve-78578c546b-8zspq" Sep 16 07:50:28.682336 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:28.682296 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c9020d/premium-w0-c9020d-kserve-78578c546b-8zspq" Sep 16 07:50:28.961816 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:28.961785 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c9020d/premium-w0-c9020d-kserve-78578c546b-8zspq"] Sep 16 07:50:28.964082 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:50:28.964052 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4543c78d_4a9b_437c_8146_89e1eed8b7a8.slice/crio-e055ea8afde0b6c9ca3ec6a56f33f0a93b06781f4a5d1969fc1b210ce458656e WatchSource:0}: Error finding container e055ea8afde0b6c9ca3ec6a56f33f0a93b06781f4a5d1969fc1b210ce458656e: Status 404 returned error can't find the container with id e055ea8afde0b6c9ca3ec6a56f33f0a93b06781f4a5d1969fc1b210ce458656e Sep 16 07:50:29.583007 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:29.582967 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c9020d/premium-w0-c9020d-kserve-78578c546b-8zspq" event={"ID":"4543c78d-4a9b-437c-8146-89e1eed8b7a8","Type":"ContainerStarted","Data":"3fddfd579122ffe1f47b71943c34bd8cbc2300b5872d500cc9df0daa204f45b5"} Sep 16 07:50:29.583442 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:29.583020 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c9020d/premium-w0-c9020d-kserve-78578c546b-8zspq" event={"ID":"4543c78d-4a9b-437c-8146-89e1eed8b7a8","Type":"ContainerStarted","Data":"e055ea8afde0b6c9ca3ec6a56f33f0a93b06781f4a5d1969fc1b210ce458656e"} Sep 16 07:50:29.601160 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:29.601082 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-c9020d/premium-w0-c9020d-kserve-78578c546b-8zspq" podStartSLOduration=1.601065922 podStartE2EDuration="1.601065922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:50:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:50:29.600239723 +0000 UTC m=+1001.184018065" watchObservedRunningTime="2026-09-16 07:50:29.601065922 +0000 UTC m=+1001.184844285" Sep 16 07:50:31.229750 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:31.229710 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:50:31.594383 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:31.594288 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:50:31.973963 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:31.973929 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:50:32.601915 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:32.601883 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:50:33.352324 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:33.352288 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:50:34.004291 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:34.004257 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:50:34.626024 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:34.625989 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:50:35.350024 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:35.349983 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:50:37.439337 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:37.439296 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:50:37.551409 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:37.551370 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:50:37.651488 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:37.651432 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:50:38.682636 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:38.682596 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c9020d/premium-w0-c9020d-kserve-78578c546b-8zspq" Sep 16 07:50:38.683068 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:38.682652 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-c9020d/premium-w0-c9020d-kserve-78578c546b-8zspq" Sep 16 07:50:38.695965 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:38.695928 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-c9020d/premium-w0-c9020d-kserve-78578c546b-8zspq" Sep 16 07:50:39.646369 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:39.646332 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c9020d/premium-w0-c9020d-kserve-78578c546b-8zspq" Sep 16 07:50:48.180412 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:48.180332 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:50:48.943502 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:48.943470 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:50:52.579298 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:52.579265 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-fc27a5/premium-w4-fc27a5-kserve-54dbcff578-kzxpf"] Sep 16 07:50:52.579783 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:52.579768 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="56ccdf02-0887-408b-bee7-26f99d1a1c7b" containerName="istio-proxy" Sep 16 07:50:52.584124 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:52.584077 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fc27a5/premium-w4-fc27a5-kserve-54dbcff578-kzxpf" Sep 16 07:50:52.587028 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:52.587002 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-fc27a5\"/\"premium-w4-fc27a5-kserve-self-signed-certs\"" Sep 16 07:50:52.613475 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:52.613437 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-fc27a5/premium-w4-fc27a5-kserve-54dbcff578-kzxpf"] Sep 16 07:50:52.621569 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:52.621529 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:50:52.725253 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:52.725215 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3edcc01e-d0c7-4f87-8a18-eb46302572df-dshm\") pod \"premium-w4-fc27a5-kserve-54dbcff578-kzxpf\" (UID: \"3edcc01e-d0c7-4f87-8a18-eb46302572df\") " pod="e2e-models-e2e-worker-w4-fc27a5/premium-w4-fc27a5-kserve-54dbcff578-kzxpf" Sep 16 07:50:52.725504 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:52.725272 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3edcc01e-d0c7-4f87-8a18-eb46302572df-tls-certs\") pod \"premium-w4-fc27a5-kserve-54dbcff578-kzxpf\" (UID: \"3edcc01e-d0c7-4f87-8a18-eb46302572df\") " pod="e2e-models-e2e-worker-w4-fc27a5/premium-w4-fc27a5-kserve-54dbcff578-kzxpf" Sep 16 07:50:52.725504 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:52.725367 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3edcc01e-d0c7-4f87-8a18-eb46302572df-tmp-dir\") pod \"premium-w4-fc27a5-kserve-54dbcff578-kzxpf\" (UID: \"3edcc01e-d0c7-4f87-8a18-eb46302572df\") " pod="e2e-models-e2e-worker-w4-fc27a5/premium-w4-fc27a5-kserve-54dbcff578-kzxpf" Sep 16 07:50:52.725504 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:52.725451 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3edcc01e-d0c7-4f87-8a18-eb46302572df-home\") pod \"premium-w4-fc27a5-kserve-54dbcff578-kzxpf\" (UID: \"3edcc01e-d0c7-4f87-8a18-eb46302572df\") " pod="e2e-models-e2e-worker-w4-fc27a5/premium-w4-fc27a5-kserve-54dbcff578-kzxpf" Sep 16 07:50:52.725504 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:52.725472 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45k2k\" (UniqueName: \"kubernetes.io/projected/3edcc01e-d0c7-4f87-8a18-eb46302572df-kube-api-access-45k2k\") pod \"premium-w4-fc27a5-kserve-54dbcff578-kzxpf\" (UID: \"3edcc01e-d0c7-4f87-8a18-eb46302572df\") " pod="e2e-models-e2e-worker-w4-fc27a5/premium-w4-fc27a5-kserve-54dbcff578-kzxpf" Sep 16 07:50:52.725686 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:52.725509 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3edcc01e-d0c7-4f87-8a18-eb46302572df-model-cache\") pod \"premium-w4-fc27a5-kserve-54dbcff578-kzxpf\" (UID: \"3edcc01e-d0c7-4f87-8a18-eb46302572df\") " pod="e2e-models-e2e-worker-w4-fc27a5/premium-w4-fc27a5-kserve-54dbcff578-kzxpf" Sep 16 07:50:52.826721 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:52.826671 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3edcc01e-d0c7-4f87-8a18-eb46302572df-home\") pod \"premium-w4-fc27a5-kserve-54dbcff578-kzxpf\" (UID: \"3edcc01e-d0c7-4f87-8a18-eb46302572df\") " pod="e2e-models-e2e-worker-w4-fc27a5/premium-w4-fc27a5-kserve-54dbcff578-kzxpf" Sep 16 07:50:52.826721 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:52.826722 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45k2k\" (UniqueName: \"kubernetes.io/projected/3edcc01e-d0c7-4f87-8a18-eb46302572df-kube-api-access-45k2k\") pod \"premium-w4-fc27a5-kserve-54dbcff578-kzxpf\" (UID: \"3edcc01e-d0c7-4f87-8a18-eb46302572df\") " pod="e2e-models-e2e-worker-w4-fc27a5/premium-w4-fc27a5-kserve-54dbcff578-kzxpf" Sep 16 07:50:52.826983 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:52.826770 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3edcc01e-d0c7-4f87-8a18-eb46302572df-model-cache\") pod \"premium-w4-fc27a5-kserve-54dbcff578-kzxpf\" (UID: \"3edcc01e-d0c7-4f87-8a18-eb46302572df\") " pod="e2e-models-e2e-worker-w4-fc27a5/premium-w4-fc27a5-kserve-54dbcff578-kzxpf" Sep 16 07:50:52.826983 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:52.826806 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3edcc01e-d0c7-4f87-8a18-eb46302572df-dshm\") pod \"premium-w4-fc27a5-kserve-54dbcff578-kzxpf\" (UID: \"3edcc01e-d0c7-4f87-8a18-eb46302572df\") " pod="e2e-models-e2e-worker-w4-fc27a5/premium-w4-fc27a5-kserve-54dbcff578-kzxpf" Sep 16 07:50:52.826983 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:52.826946 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3edcc01e-d0c7-4f87-8a18-eb46302572df-tls-certs\") pod \"premium-w4-fc27a5-kserve-54dbcff578-kzxpf\" (UID: \"3edcc01e-d0c7-4f87-8a18-eb46302572df\") " pod="e2e-models-e2e-worker-w4-fc27a5/premium-w4-fc27a5-kserve-54dbcff578-kzxpf" Sep 16 07:50:52.827188 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:52.827009 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3edcc01e-d0c7-4f87-8a18-eb46302572df-tmp-dir\") pod \"premium-w4-fc27a5-kserve-54dbcff578-kzxpf\" (UID: \"3edcc01e-d0c7-4f87-8a18-eb46302572df\") " pod="e2e-models-e2e-worker-w4-fc27a5/premium-w4-fc27a5-kserve-54dbcff578-kzxpf" Sep 16 07:50:52.827245 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:52.827183 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3edcc01e-d0c7-4f87-8a18-eb46302572df-home\") pod \"premium-w4-fc27a5-kserve-54dbcff578-kzxpf\" (UID: \"3edcc01e-d0c7-4f87-8a18-eb46302572df\") " pod="e2e-models-e2e-worker-w4-fc27a5/premium-w4-fc27a5-kserve-54dbcff578-kzxpf" Sep 16 07:50:52.827245 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:52.827212 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3edcc01e-d0c7-4f87-8a18-eb46302572df-model-cache\") pod \"premium-w4-fc27a5-kserve-54dbcff578-kzxpf\" (UID: \"3edcc01e-d0c7-4f87-8a18-eb46302572df\") " pod="e2e-models-e2e-worker-w4-fc27a5/premium-w4-fc27a5-kserve-54dbcff578-kzxpf" Sep 16 07:50:52.827364 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:52.827347 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3edcc01e-d0c7-4f87-8a18-eb46302572df-tmp-dir\") pod \"premium-w4-fc27a5-kserve-54dbcff578-kzxpf\" (UID: \"3edcc01e-d0c7-4f87-8a18-eb46302572df\") " pod="e2e-models-e2e-worker-w4-fc27a5/premium-w4-fc27a5-kserve-54dbcff578-kzxpf" Sep 16 07:50:52.829250 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:52.829219 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3edcc01e-d0c7-4f87-8a18-eb46302572df-dshm\") pod \"premium-w4-fc27a5-kserve-54dbcff578-kzxpf\" (UID: \"3edcc01e-d0c7-4f87-8a18-eb46302572df\") " pod="e2e-models-e2e-worker-w4-fc27a5/premium-w4-fc27a5-kserve-54dbcff578-kzxpf" Sep 16 07:50:52.829490 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:52.829452 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3edcc01e-d0c7-4f87-8a18-eb46302572df-tls-certs\") pod \"premium-w4-fc27a5-kserve-54dbcff578-kzxpf\" (UID: \"3edcc01e-d0c7-4f87-8a18-eb46302572df\") " pod="e2e-models-e2e-worker-w4-fc27a5/premium-w4-fc27a5-kserve-54dbcff578-kzxpf" Sep 16 07:50:52.838090 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:52.838065 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45k2k\" (UniqueName: \"kubernetes.io/projected/3edcc01e-d0c7-4f87-8a18-eb46302572df-kube-api-access-45k2k\") pod \"premium-w4-fc27a5-kserve-54dbcff578-kzxpf\" (UID: \"3edcc01e-d0c7-4f87-8a18-eb46302572df\") " pod="e2e-models-e2e-worker-w4-fc27a5/premium-w4-fc27a5-kserve-54dbcff578-kzxpf" Sep 16 07:50:52.893973 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:52.893931 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fc27a5/premium-w4-fc27a5-kserve-54dbcff578-kzxpf" Sep 16 07:50:53.170474 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:53.170443 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-fc27a5/premium-w4-fc27a5-kserve-54dbcff578-kzxpf"] Sep 16 07:50:53.172456 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:50:53.172430 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3edcc01e_d0c7_4f87_8a18_eb46302572df.slice/crio-b8db2622f8f43399745bd541c845f3a9ed3daf69b7abdea65dc7c0cce824bc52 WatchSource:0}: Error finding container b8db2622f8f43399745bd541c845f3a9ed3daf69b7abdea65dc7c0cce824bc52: Status 404 returned error can't find the container with id b8db2622f8f43399745bd541c845f3a9ed3daf69b7abdea65dc7c0cce824bc52 Sep 16 07:50:53.709673 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:53.709633 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fc27a5/premium-w4-fc27a5-kserve-54dbcff578-kzxpf" event={"ID":"3edcc01e-d0c7-4f87-8a18-eb46302572df","Type":"ContainerStarted","Data":"5ee0010b9c738fb5eb16240327bad86081d7168a42d9a10c0cf26cc1053b58df"} Sep 16 07:50:53.710035 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:53.709683 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fc27a5/premium-w4-fc27a5-kserve-54dbcff578-kzxpf" event={"ID":"3edcc01e-d0c7-4f87-8a18-eb46302572df","Type":"ContainerStarted","Data":"b8db2622f8f43399745bd541c845f3a9ed3daf69b7abdea65dc7c0cce824bc52"} Sep 16 07:50:53.728533 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:53.728476 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-fc27a5/premium-w4-fc27a5-kserve-54dbcff578-kzxpf" podStartSLOduration=1.728457647 podStartE2EDuration="1.728457647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:50:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:50:53.726390719 +0000 UTC m=+1025.310169069" watchObservedRunningTime="2026-09-16 07:50:53.728457647 +0000 UTC m=+1025.312235984" Sep 16 07:50:54.766532 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:54.766489 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:50:54.867408 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:54.867372 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:50:54.967725 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:54.967681 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:50:55.067243 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:55.067153 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:50:55.705594 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:55.705554 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:50:55.805355 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:55.805320 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:50:55.905543 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:55.905508 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:50:57.409435 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:50:57.409391 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:51:01.995607 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:01.995569 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:51:02.894240 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:02.894203 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-fc27a5/premium-w4-fc27a5-kserve-54dbcff578-kzxpf" Sep 16 07:51:02.894240 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:02.894251 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-fc27a5/premium-w4-fc27a5-kserve-54dbcff578-kzxpf" Sep 16 07:51:02.909973 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:02.909944 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-fc27a5/premium-w4-fc27a5-kserve-54dbcff578-kzxpf" Sep 16 07:51:03.378492 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:03.378459 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:51:03.478582 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:03.478548 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:51:03.776039 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:03.775997 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-fc27a5/premium-w4-fc27a5-kserve-54dbcff578-kzxpf" Sep 16 07:51:04.044508 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:04.044399 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:51:09.757881 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:09.757839 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:51:09.858017 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:09.857981 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:51:09.994392 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:09.994357 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:51:10.276450 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:10.276403 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:51:10.515325 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:10.515288 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:51:15.492758 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:15.492718 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:51:16.281640 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:16.281603 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:51:16.379904 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:16.379866 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:51:17.348539 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:17.348500 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:51:18.337738 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:18.337691 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:51:19.001000 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:19.000959 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:51:19.658256 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:19.658222 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:51:20.485384 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:20.485343 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:51:20.588969 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:20.588921 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:51:20.691693 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:20.691661 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:51:20.926323 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:20.926280 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:51:21.029800 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:21.029763 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:51:21.056890 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:21.056853 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b71faf9e-5f69f8bb5-sstm6"] Sep 16 07:51:21.060661 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:21.060641 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b71faf9e-5f69f8bb5-sstm6" Sep 16 07:51:21.063365 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:21.063339 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-b71faf9e\"" Sep 16 07:51:21.063529 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:21.063419 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-b71faf9e-dockercfg-mnq59\"" Sep 16 07:51:21.073916 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:21.073886 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b71faf9e-5f69f8bb5-sstm6"] Sep 16 07:51:21.100615 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:21.100576 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:51:21.102799 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:21.102779 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23182d3c-1b75-4c1a-80e3-e5a194fcb8b5-plugins-config-volume\") pod \"payload-processing-e2e-mt-b71faf9e-5f69f8bb5-sstm6\" (UID: \"23182d3c-1b75-4c1a-80e3-e5a194fcb8b5\") " pod="openshift-ingress/payload-processing-e2e-mt-b71faf9e-5f69f8bb5-sstm6" Sep 16 07:51:21.102887 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:21.102867 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkp5j\" (UniqueName: \"kubernetes.io/projected/23182d3c-1b75-4c1a-80e3-e5a194fcb8b5-kube-api-access-mkp5j\") pod \"payload-processing-e2e-mt-b71faf9e-5f69f8bb5-sstm6\" (UID: \"23182d3c-1b75-4c1a-80e3-e5a194fcb8b5\") " pod="openshift-ingress/payload-processing-e2e-mt-b71faf9e-5f69f8bb5-sstm6" Sep 16 07:51:21.204215 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:21.204074 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkp5j\" (UniqueName: \"kubernetes.io/projected/23182d3c-1b75-4c1a-80e3-e5a194fcb8b5-kube-api-access-mkp5j\") pod \"payload-processing-e2e-mt-b71faf9e-5f69f8bb5-sstm6\" (UID: \"23182d3c-1b75-4c1a-80e3-e5a194fcb8b5\") " pod="openshift-ingress/payload-processing-e2e-mt-b71faf9e-5f69f8bb5-sstm6" Sep 16 07:51:21.204215 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:21.204211 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23182d3c-1b75-4c1a-80e3-e5a194fcb8b5-plugins-config-volume\") pod \"payload-processing-e2e-mt-b71faf9e-5f69f8bb5-sstm6\" (UID: \"23182d3c-1b75-4c1a-80e3-e5a194fcb8b5\") " pod="openshift-ingress/payload-processing-e2e-mt-b71faf9e-5f69f8bb5-sstm6" Sep 16 07:51:21.204903 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:21.204881 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23182d3c-1b75-4c1a-80e3-e5a194fcb8b5-plugins-config-volume\") pod \"payload-processing-e2e-mt-b71faf9e-5f69f8bb5-sstm6\" (UID: \"23182d3c-1b75-4c1a-80e3-e5a194fcb8b5\") " pod="openshift-ingress/payload-processing-e2e-mt-b71faf9e-5f69f8bb5-sstm6" Sep 16 07:51:21.212660 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:21.212633 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkp5j\" (UniqueName: \"kubernetes.io/projected/23182d3c-1b75-4c1a-80e3-e5a194fcb8b5-kube-api-access-mkp5j\") pod \"payload-processing-e2e-mt-b71faf9e-5f69f8bb5-sstm6\" (UID: \"23182d3c-1b75-4c1a-80e3-e5a194fcb8b5\") " pod="openshift-ingress/payload-processing-e2e-mt-b71faf9e-5f69f8bb5-sstm6" Sep 16 07:51:21.372513 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:21.372470 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b71faf9e-5f69f8bb5-sstm6" Sep 16 07:51:21.649169 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:21.649091 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b71faf9e-5f69f8bb5-sstm6"] Sep 16 07:51:21.651242 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:51:21.651212 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23182d3c_1b75_4c1a_80e3_e5a194fcb8b5.slice/crio-c9921aa407e6d61e372a811ab2447946bf781f832e1084fc926b347a1b37b8b1 WatchSource:0}: Error finding container c9921aa407e6d61e372a811ab2447946bf781f832e1084fc926b347a1b37b8b1: Status 404 returned error can't find the container with id c9921aa407e6d61e372a811ab2447946bf781f832e1084fc926b347a1b37b8b1 Sep 16 07:51:21.865162 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:21.865118 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b71faf9e-5f69f8bb5-sstm6" event={"ID":"23182d3c-1b75-4c1a-80e3-e5a194fcb8b5","Type":"ContainerStarted","Data":"73a91a27f8b14a9e6b6d6abaaac77b36a8cf9a603c6c9207e92ce9b3448a6ae5"} Sep 16 07:51:21.865354 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:21.865177 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b71faf9e-5f69f8bb5-sstm6" event={"ID":"23182d3c-1b75-4c1a-80e3-e5a194fcb8b5","Type":"ContainerStarted","Data":"c9921aa407e6d61e372a811ab2447946bf781f832e1084fc926b347a1b37b8b1"} Sep 16 07:51:21.865354 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:21.865251 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-b71faf9e-5f69f8bb5-sstm6" Sep 16 07:51:21.885308 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:21.885245 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-b71faf9e-5f69f8bb5-sstm6" podStartSLOduration=0.885225229 podStartE2EDuration="885.225229ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:51:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:51:21.882949147 +0000 UTC m=+1053.466727485" watchObservedRunningTime="2026-09-16 07:51:21.885225229 +0000 UTC m=+1053.469003567" Sep 16 07:51:32.872619 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:32.872587 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-b71faf9e-5f69f8bb5-sstm6" Sep 16 07:51:35.411574 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:35.411505 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:51:39.831859 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:39.831826 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:51:39.933586 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:39.933548 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:51:40.787859 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:40.787820 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:51:42.361943 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:42.361904 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:51:43.174659 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:43.174622 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:51:43.858068 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:43.858031 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:51:44.560316 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:44.560277 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:51:45.342670 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:45.342633 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:51:45.443459 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:45.443422 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:51:46.045925 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:46.045885 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:51:46.251381 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:46.251339 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:51:46.819061 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:46.819024 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:51:46.919844 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:46.919807 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:51:47.027561 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:47.027521 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:51:47.349024 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:47.348991 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:51:53.553807 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:53.553766 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:51:53.655937 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:53.655893 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:51:54.856594 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:54.856555 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:51:55.469613 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:55.469571 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:51:56.080870 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:56.080833 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:51:56.950119 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:56.950067 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:51:57.579225 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:57.579188 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:51:58.232661 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:58.232631 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:51:59.101210 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:51:59.101172 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:04.532850 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:04.532810 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:04.557671 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:04.557623 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b71faf9e-5f69f8bb5-sstm6"] Sep 16 07:52:04.557940 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:04.557909 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-b71faf9e-5f69f8bb5-sstm6" podUID="23182d3c-1b75-4c1a-80e3-e5a194fcb8b5" containerName="payload-processing" containerID="cri-o://73a91a27f8b14a9e6b6d6abaaac77b36a8cf9a603c6c9207e92ce9b3448a6ae5" gracePeriod=30 Sep 16 07:52:04.600939 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:04.600901 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:04.701358 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:04.701322 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:04.767282 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:04.767236 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:04.834221 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:04.834196 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b71faf9e-5f69f8bb5-sstm6" Sep 16 07:52:04.961211 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:04.961161 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23182d3c-1b75-4c1a-80e3-e5a194fcb8b5-plugins-config-volume\") pod \"23182d3c-1b75-4c1a-80e3-e5a194fcb8b5\" (UID: \"23182d3c-1b75-4c1a-80e3-e5a194fcb8b5\") " Sep 16 07:52:04.961394 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:04.961258 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mkp5j\" (UniqueName: \"kubernetes.io/projected/23182d3c-1b75-4c1a-80e3-e5a194fcb8b5-kube-api-access-mkp5j\") pod \"23182d3c-1b75-4c1a-80e3-e5a194fcb8b5\" (UID: \"23182d3c-1b75-4c1a-80e3-e5a194fcb8b5\") " Sep 16 07:52:04.961574 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:04.961546 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23182d3c-1b75-4c1a-80e3-e5a194fcb8b5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "23182d3c-1b75-4c1a-80e3-e5a194fcb8b5" (UID: "23182d3c-1b75-4c1a-80e3-e5a194fcb8b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:52:04.963478 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:04.963447 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23182d3c-1b75-4c1a-80e3-e5a194fcb8b5-kube-api-access-mkp5j" (OuterVolumeSpecName: "kube-api-access-mkp5j") pod "23182d3c-1b75-4c1a-80e3-e5a194fcb8b5" (UID: "23182d3c-1b75-4c1a-80e3-e5a194fcb8b5"). InnerVolumeSpecName "kube-api-access-mkp5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:52:04.986500 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:04.986466 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:05.062921 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:05.062816 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkp5j\" (UniqueName: \"kubernetes.io/projected/23182d3c-1b75-4c1a-80e3-e5a194fcb8b5-kube-api-access-mkp5j\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:52:05.062921 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:05.062860 2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23182d3c-1b75-4c1a-80e3-e5a194fcb8b5-plugins-config-volume\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:52:05.087150 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:05.087112 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:05.098715 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:05.098677 2552 generic.go:356] "Generic (PLEG): container finished" podID="23182d3c-1b75-4c1a-80e3-e5a194fcb8b5" containerID="73a91a27f8b14a9e6b6d6abaaac77b36a8cf9a603c6c9207e92ce9b3448a6ae5" exitCode=0 Sep 16 07:52:05.098916 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:05.098728 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b71faf9e-5f69f8bb5-sstm6" Sep 16 07:52:05.098916 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:05.098770 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b71faf9e-5f69f8bb5-sstm6" event={"ID":"23182d3c-1b75-4c1a-80e3-e5a194fcb8b5","Type":"ContainerDied","Data":"73a91a27f8b14a9e6b6d6abaaac77b36a8cf9a603c6c9207e92ce9b3448a6ae5"} Sep 16 07:52:05.098916 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:05.098827 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b71faf9e-5f69f8bb5-sstm6" event={"ID":"23182d3c-1b75-4c1a-80e3-e5a194fcb8b5","Type":"ContainerDied","Data":"c9921aa407e6d61e372a811ab2447946bf781f832e1084fc926b347a1b37b8b1"} Sep 16 07:52:05.098916 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:05.098850 2552 scope.go:117] "RemoveContainer" containerID="73a91a27f8b14a9e6b6d6abaaac77b36a8cf9a603c6c9207e92ce9b3448a6ae5" Sep 16 07:52:05.108083 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:05.108063 2552 scope.go:117] "RemoveContainer" containerID="73a91a27f8b14a9e6b6d6abaaac77b36a8cf9a603c6c9207e92ce9b3448a6ae5" Sep 16 07:52:05.108455 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:52:05.108431 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73a91a27f8b14a9e6b6d6abaaac77b36a8cf9a603c6c9207e92ce9b3448a6ae5\": container with ID starting with 73a91a27f8b14a9e6b6d6abaaac77b36a8cf9a603c6c9207e92ce9b3448a6ae5 not found: ID does not exist" containerID="73a91a27f8b14a9e6b6d6abaaac77b36a8cf9a603c6c9207e92ce9b3448a6ae5" Sep 16 07:52:05.108506 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:05.108464 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73a91a27f8b14a9e6b6d6abaaac77b36a8cf9a603c6c9207e92ce9b3448a6ae5"} err="failed to get container status \"73a91a27f8b14a9e6b6d6abaaac77b36a8cf9a603c6c9207e92ce9b3448a6ae5\": rpc error: code = NotFound desc = could not find container \"73a91a27f8b14a9e6b6d6abaaac77b36a8cf9a603c6c9207e92ce9b3448a6ae5\": container with ID starting with 73a91a27f8b14a9e6b6d6abaaac77b36a8cf9a603c6c9207e92ce9b3448a6ae5 not found: ID does not exist" Sep 16 07:52:05.120155 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:05.120121 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b71faf9e-5f69f8bb5-sstm6"] Sep 16 07:52:05.122094 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:05.122069 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b71faf9e-5f69f8bb5-sstm6"] Sep 16 07:52:05.789857 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:05.789810 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:06.500115 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:06.500080 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:06.939301 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:06.939264 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23182d3c-1b75-4c1a-80e3-e5a194fcb8b5" path="/var/lib/kubelet/pods/23182d3c-1b75-4c1a-80e3-e5a194fcb8b5/volumes" Sep 16 07:52:07.221292 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:07.221178 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:07.875660 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:07.875626 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:08.591974 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:08.591935 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:11.012744 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:11.012698 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:11.112746 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:11.112711 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:11.249538 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:11.249493 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:11.589706 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:11.589665 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:11.679266 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:11.679227 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:11.789358 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:11.789322 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:13.128428 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:13.128389 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-82bbbf-2tt25"] Sep 16 07:52:13.128822 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:13.128801 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23182d3c-1b75-4c1a-80e3-e5a194fcb8b5" containerName="payload-processing" Sep 16 07:52:13.128822 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:13.128813 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="23182d3c-1b75-4c1a-80e3-e5a194fcb8b5" containerName="payload-processing" Sep 16 07:52:13.128900 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:13.128889 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="23182d3c-1b75-4c1a-80e3-e5a194fcb8b5" containerName="payload-processing" Sep 16 07:52:13.133509 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:13.133483 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-82bbbf-2tt25" Sep 16 07:52:13.135974 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:13.135948 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zwk5r\"" Sep 16 07:52:13.136090 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:13.136038 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 07:52:13.141026 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:13.140994 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-82bbbf-2tt25"] Sep 16 07:52:13.146405 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:13.146375 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2e31960-6316-4a0b-b1ca-5b3696f6247d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-82bbbf-2tt25\" (UID: \"f2e31960-6316-4a0b-b1ca-5b3696f6247d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-82bbbf-2tt25" Sep 16 07:52:13.247053 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:13.247007 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2e31960-6316-4a0b-b1ca-5b3696f6247d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-82bbbf-2tt25\" (UID: \"f2e31960-6316-4a0b-b1ca-5b3696f6247d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-82bbbf-2tt25" Sep 16 07:52:13.247687 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:13.247665 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2e31960-6316-4a0b-b1ca-5b3696f6247d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-82bbbf-2tt25\" (UID: \"f2e31960-6316-4a0b-b1ca-5b3696f6247d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-82bbbf-2tt25" Sep 16 07:52:13.448765 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:13.448717 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-82bbbf-2tt25" Sep 16 07:52:13.470649 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:13.470610 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:13.723992 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:13.723941 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-82bbbf-2tt25"] Sep 16 07:52:13.727015 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:52:13.726983 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2e31960_6316_4a0b_b1ca_5b3696f6247d.slice/crio-3485dbb8baa1a4a60fedfaa19a47d7ad9ca9faf0cf7aac9ab2070c999f59d52b WatchSource:0}: Error finding container 3485dbb8baa1a4a60fedfaa19a47d7ad9ca9faf0cf7aac9ab2070c999f59d52b: Status 404 returned error can't find the container with id 3485dbb8baa1a4a60fedfaa19a47d7ad9ca9faf0cf7aac9ab2070c999f59d52b Sep 16 07:52:14.154052 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:14.154011 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-82bbbf-2tt25" event={"ID":"f2e31960-6316-4a0b-b1ca-5b3696f6247d","Type":"ContainerStarted","Data":"f80a6cc7c38b7c3a66fc4ec3f54201316cb84968e607fcb579634d6a44c193a6"} Sep 16 07:52:14.154475 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:14.154069 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-82bbbf-2tt25" event={"ID":"f2e31960-6316-4a0b-b1ca-5b3696f6247d","Type":"ContainerStarted","Data":"3485dbb8baa1a4a60fedfaa19a47d7ad9ca9faf0cf7aac9ab2070c999f59d52b"} Sep 16 07:52:14.171462 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:14.171402 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-82bbbf-2tt25" podStartSLOduration=1.171380646 podStartE2EDuration="1.171380646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:52:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:52:14.170529871 +0000 UTC m=+1105.754308210" watchObservedRunningTime="2026-09-16 07:52:14.171380646 +0000 UTC m=+1105.755158983" Sep 16 07:52:14.254951 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:14.254917 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:14.944930 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:14.944894 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:15.162164 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:15.162127 2552 generic.go:356] "Generic (PLEG): container finished" podID="f2e31960-6316-4a0b-b1ca-5b3696f6247d" containerID="f80a6cc7c38b7c3a66fc4ec3f54201316cb84968e607fcb579634d6a44c193a6" exitCode=0 Sep 16 07:52:15.162548 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:15.162220 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-82bbbf-2tt25" event={"ID":"f2e31960-6316-4a0b-b1ca-5b3696f6247d","Type":"ContainerDied","Data":"f80a6cc7c38b7c3a66fc4ec3f54201316cb84968e607fcb579634d6a44c193a6"} Sep 16 07:52:15.728703 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:15.728673 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:16.312478 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:16.312439 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-82bbbf-2tt25" Sep 16 07:52:16.378486 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:16.378445 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2e31960-6316-4a0b-b1ca-5b3696f6247d-maas-api-service-ca\") pod \"f2e31960-6316-4a0b-b1ca-5b3696f6247d\" (UID: \"f2e31960-6316-4a0b-b1ca-5b3696f6247d\") " Sep 16 07:52:16.378751 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:16.378727 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2e31960-6316-4a0b-b1ca-5b3696f6247d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f2e31960-6316-4a0b-b1ca-5b3696f6247d" (UID: "f2e31960-6316-4a0b-b1ca-5b3696f6247d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:52:16.378830 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:16.378820 2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2e31960-6316-4a0b-b1ca-5b3696f6247d-maas-api-service-ca\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:52:16.456436 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:16.456336 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:17.175625 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:17.175588 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-82bbbf-2tt25" Sep 16 07:52:17.175824 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:17.175682 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-82bbbf-2tt25" event={"ID":"f2e31960-6316-4a0b-b1ca-5b3696f6247d","Type":"ContainerDied","Data":"3485dbb8baa1a4a60fedfaa19a47d7ad9ca9faf0cf7aac9ab2070c999f59d52b"} Sep 16 07:52:17.175824 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:17.175734 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3485dbb8baa1a4a60fedfaa19a47d7ad9ca9faf0cf7aac9ab2070c999f59d52b" Sep 16 07:52:17.175824 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:17.175756 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:17.922280 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:17.922247 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:18.026483 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:18.026448 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:18.128433 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:18.128395 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:18.425416 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:18.425381 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:19.518611 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:19.518570 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:23.616274 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:23.616240 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:23.748632 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:23.748595 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:23.810590 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:23.810553 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-82bbbf-2tt25"] Sep 16 07:52:23.814180 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:23.814144 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-82bbbf-2tt25"] Sep 16 07:52:23.849957 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:23.849921 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:24.075759 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:24.075718 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:24.176415 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:24.176381 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:24.826459 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:24.826419 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:24.926335 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:24.926296 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:24.938234 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:24.938200 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2e31960-6316-4a0b-b1ca-5b3696f6247d" path="/var/lib/kubelet/pods/f2e31960-6316-4a0b-b1ca-5b3696f6247d/volumes" Sep 16 07:52:25.040920 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:25.040887 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:25.663617 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:25.663573 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:25.763660 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:25.763615 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:26.008274 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:26.008217 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:26.288755 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:26.288639 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:26.451861 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:26.451821 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:27.136351 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:27.136312 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:27.781389 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:27.781351 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:28.698432 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:28.698399 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:29.120180 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:29.120073 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:29.214295 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:29.214262 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:29.315735 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:29.315695 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:29.559873 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:29.559840 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:30.306614 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:30.306573 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:31.172557 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:31.172517 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:31.850020 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:31.849973 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:32.499242 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:32.499208 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:32.609318 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:32.609283 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:32.932519 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:32.932489 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:33.033676 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:33.033637 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:33.110780 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:33.110726 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:33.136999 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:33.136964 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4a243c-6d986c5c4c-s42d9"] Sep 16 07:52:33.137402 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:33.137387 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2e31960-6316-4a0b-b1ca-5b3696f6247d" containerName="revoke-keys" Sep 16 07:52:33.137453 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:33.137403 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2e31960-6316-4a0b-b1ca-5b3696f6247d" containerName="revoke-keys" Sep 16 07:52:33.137489 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:33.137477 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2e31960-6316-4a0b-b1ca-5b3696f6247d" containerName="revoke-keys" Sep 16 07:52:33.141073 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:33.141050 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4a243c-6d986c5c4c-s42d9" Sep 16 07:52:33.143933 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:33.143903 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-4a243c\"" Sep 16 07:52:33.144093 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:33.143954 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-4a243c-dockercfg-n7fr9\"" Sep 16 07:52:33.153135 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:33.153083 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4a243c-6d986c5c4c-s42d9"] Sep 16 07:52:33.246089 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:33.245991 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qqbx\" (UniqueName: \"kubernetes.io/projected/a8e51140-8400-4a87-9666-9b3a076dc538-kube-api-access-2qqbx\") pod \"payload-processing-e2e-shared-a-w2-4a243c-6d986c5c4c-s42d9\" (UID: \"a8e51140-8400-4a87-9666-9b3a076dc538\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4a243c-6d986c5c4c-s42d9" Sep 16 07:52:33.246089 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:33.246043 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e51140-8400-4a87-9666-9b3a076dc538-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-4a243c-6d986c5c4c-s42d9\" (UID: \"a8e51140-8400-4a87-9666-9b3a076dc538\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4a243c-6d986c5c4c-s42d9" Sep 16 07:52:33.346607 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:33.346565 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qqbx\" (UniqueName: \"kubernetes.io/projected/a8e51140-8400-4a87-9666-9b3a076dc538-kube-api-access-2qqbx\") pod \"payload-processing-e2e-shared-a-w2-4a243c-6d986c5c4c-s42d9\" (UID: \"a8e51140-8400-4a87-9666-9b3a076dc538\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4a243c-6d986c5c4c-s42d9" Sep 16 07:52:33.346607 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:33.346623 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e51140-8400-4a87-9666-9b3a076dc538-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-4a243c-6d986c5c4c-s42d9\" (UID: \"a8e51140-8400-4a87-9666-9b3a076dc538\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4a243c-6d986c5c4c-s42d9" Sep 16 07:52:33.347587 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:33.347559 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e51140-8400-4a87-9666-9b3a076dc538-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-4a243c-6d986c5c4c-s42d9\" (UID: \"a8e51140-8400-4a87-9666-9b3a076dc538\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4a243c-6d986c5c4c-s42d9" Sep 16 07:52:33.354920 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:33.354878 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qqbx\" (UniqueName: \"kubernetes.io/projected/a8e51140-8400-4a87-9666-9b3a076dc538-kube-api-access-2qqbx\") pod \"payload-processing-e2e-shared-a-w2-4a243c-6d986c5c4c-s42d9\" (UID: \"a8e51140-8400-4a87-9666-9b3a076dc538\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4a243c-6d986c5c4c-s42d9" Sep 16 07:52:33.453966 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:33.453921 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4a243c-6d986c5c4c-s42d9" Sep 16 07:52:33.466352 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:33.466308 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:33.729128 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:33.729067 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4a243c-6d986c5c4c-s42d9"] Sep 16 07:52:33.731499 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:52:33.731469 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8e51140_8400_4a87_9666_9b3a076dc538.slice/crio-fa83ba7c776ed10542b30bcca339110e5151739cf114d190da5c0a6ddcb6687a WatchSource:0}: Error finding container fa83ba7c776ed10542b30bcca339110e5151739cf114d190da5c0a6ddcb6687a: Status 404 returned error can't find the container with id fa83ba7c776ed10542b30bcca339110e5151739cf114d190da5c0a6ddcb6687a Sep 16 07:52:34.268376 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:34.268327 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4a243c-6d986c5c4c-s42d9" event={"ID":"a8e51140-8400-4a87-9666-9b3a076dc538","Type":"ContainerStarted","Data":"5141ec06f9223cb772c918ce2612b29649996474e1fb2b880cf6d046371ff460"} Sep 16 07:52:34.268866 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:34.268405 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4a243c-6d986c5c4c-s42d9" Sep 16 07:52:34.268866 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:34.268426 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4a243c-6d986c5c4c-s42d9" event={"ID":"a8e51140-8400-4a87-9666-9b3a076dc538","Type":"ContainerStarted","Data":"fa83ba7c776ed10542b30bcca339110e5151739cf114d190da5c0a6ddcb6687a"} Sep 16 07:52:34.286528 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:34.286469 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4a243c-6d986c5c4c-s42d9" podStartSLOduration=1.2864533759999999 podStartE2EDuration="1.286453376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:52:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:52:34.284952581 +0000 UTC m=+1125.868730918" watchObservedRunningTime="2026-09-16 07:52:34.286453376 +0000 UTC m=+1125.870231714" Sep 16 07:52:34.289925 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:34.289888 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:35.044414 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:35.044364 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:45.284579 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:45.284541 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4a243c-6d986c5c4c-s42d9" Sep 16 07:52:53.497937 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:53.497890 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:54.687500 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:54.687457 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:55.368494 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:55.368453 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:55.825396 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:55.825357 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:56.201039 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:56.201008 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:56.955243 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:56.955211 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:57.639213 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:57.639175 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:58.298914 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:58.298869 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:59.148094 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:59.148056 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:59.484534 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:59.484500 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:52:59.774476 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:52:59.774358 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:53:00.083255 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:00.083163 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:53:01.696574 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:01.696537 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:53:02.473352 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:02.473307 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:53:02.594144 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:02.594095 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:53:03.481319 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:03.481283 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:53:04.484961 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:04.484921 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:53:05.355329 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:05.355290 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:53:06.046925 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:06.046887 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:53:07.024330 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:07.024284 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:53:07.156353 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:07.156292 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:53:07.525760 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:07.525724 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:53:07.624689 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:07.624653 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:53:07.725504 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:07.725463 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:53:08.004935 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:08.004900 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:53:08.268673 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:08.268566 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:53:08.709027 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:08.708990 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:53:08.989546 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:08.989458 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc"] Sep 16 07:53:08.994788 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:08.994759 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc" Sep 16 07:53:08.997203 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:08.997178 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-3beaf8-openshift-default-dockercfg-gm46m\"" Sep 16 07:53:09.007879 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:09.007849 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc"] Sep 16 07:53:09.036539 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:09.036498 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:53:09.117576 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:09.117536 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-istiod-ca-cert\") pod \"e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc\" (UID: \"bbc18e51-8079-4c30-a3e0-98bd8ddf132d\") " pod="openshift-ingress/e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc" Sep 16 07:53:09.117576 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:09.117590 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhv2b\" (UniqueName: \"kubernetes.io/projected/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-kube-api-access-vhv2b\") pod \"e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc\" (UID: \"bbc18e51-8079-4c30-a3e0-98bd8ddf132d\") " pod="openshift-ingress/e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc" Sep 16 07:53:09.117833 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:09.117628 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-istio-podinfo\") pod \"e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc\" (UID: \"bbc18e51-8079-4c30-a3e0-98bd8ddf132d\") " pod="openshift-ingress/e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc" Sep 16 07:53:09.117833 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:09.117676 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-istio-token\") pod \"e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc\" (UID: \"bbc18e51-8079-4c30-a3e0-98bd8ddf132d\") " pod="openshift-ingress/e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc" Sep 16 07:53:09.117833 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:09.117698 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-istio-data\") pod \"e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc\" (UID: \"bbc18e51-8079-4c30-a3e0-98bd8ddf132d\") " pod="openshift-ingress/e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc" Sep 16 07:53:09.117833 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:09.117721 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-istio-envoy\") pod \"e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc\" (UID: \"bbc18e51-8079-4c30-a3e0-98bd8ddf132d\") " pod="openshift-ingress/e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc" Sep 16 07:53:09.117833 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:09.117741 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-workload-certs\") pod \"e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc\" (UID: \"bbc18e51-8079-4c30-a3e0-98bd8ddf132d\") " pod="openshift-ingress/e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc" Sep 16 07:53:09.117833 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:09.117757 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-credential-socket\") pod \"e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc\" (UID: \"bbc18e51-8079-4c30-a3e0-98bd8ddf132d\") " pod="openshift-ingress/e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc" Sep 16 07:53:09.118068 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:09.117844 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-workload-socket\") pod \"e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc\" (UID: \"bbc18e51-8079-4c30-a3e0-98bd8ddf132d\") " pod="openshift-ingress/e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc" Sep 16 07:53:09.219042 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:09.218993 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-istio-token\") pod \"e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc\" (UID: \"bbc18e51-8079-4c30-a3e0-98bd8ddf132d\") " pod="openshift-ingress/e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc" Sep 16 07:53:09.219042 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:09.219036 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-istio-data\") pod \"e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc\" (UID: \"bbc18e51-8079-4c30-a3e0-98bd8ddf132d\") " pod="openshift-ingress/e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc" Sep 16 07:53:09.219042 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:09.219053 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-istio-envoy\") pod \"e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc\" (UID: \"bbc18e51-8079-4c30-a3e0-98bd8ddf132d\") " pod="openshift-ingress/e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc" Sep 16 07:53:09.219422 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:09.219072 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-workload-certs\") pod \"e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc\" (UID: \"bbc18e51-8079-4c30-a3e0-98bd8ddf132d\") " pod="openshift-ingress/e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc" Sep 16 07:53:09.219422 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:09.219089 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-credential-socket\") pod \"e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc\" (UID: \"bbc18e51-8079-4c30-a3e0-98bd8ddf132d\") " pod="openshift-ingress/e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc" Sep 16 07:53:09.219422 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:09.219149 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-workload-socket\") pod \"e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc\" (UID: \"bbc18e51-8079-4c30-a3e0-98bd8ddf132d\") " pod="openshift-ingress/e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc" Sep 16 07:53:09.219422 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:09.219259 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-istiod-ca-cert\") pod \"e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc\" (UID: \"bbc18e51-8079-4c30-a3e0-98bd8ddf132d\") " pod="openshift-ingress/e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc" Sep 16 07:53:09.219422 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:09.219298 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhv2b\" (UniqueName: \"kubernetes.io/projected/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-kube-api-access-vhv2b\") pod \"e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc\" (UID: \"bbc18e51-8079-4c30-a3e0-98bd8ddf132d\") " pod="openshift-ingress/e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc" Sep 16 07:53:09.219422 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:09.219329 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-istio-podinfo\") pod \"e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc\" (UID: \"bbc18e51-8079-4c30-a3e0-98bd8ddf132d\") " pod="openshift-ingress/e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc" Sep 16 07:53:09.219706 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:09.219543 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-istio-data\") pod \"e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc\" (UID: \"bbc18e51-8079-4c30-a3e0-98bd8ddf132d\") " pod="openshift-ingress/e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc" Sep 16 07:53:09.219767 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:09.219745 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-credential-socket\") pod \"e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc\" (UID: \"bbc18e51-8079-4c30-a3e0-98bd8ddf132d\") " pod="openshift-ingress/e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc" Sep 16 07:53:09.219821 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:09.219762 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-workload-certs\") pod \"e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc\" (UID: \"bbc18e51-8079-4c30-a3e0-98bd8ddf132d\") " pod="openshift-ingress/e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc" Sep 16 07:53:09.219902 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:09.219878 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-workload-socket\") pod \"e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc\" (UID: \"bbc18e51-8079-4c30-a3e0-98bd8ddf132d\") " pod="openshift-ingress/e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc" Sep 16 07:53:09.220080 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:09.220060 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-istiod-ca-cert\") pod \"e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc\" (UID: \"bbc18e51-8079-4c30-a3e0-98bd8ddf132d\") " pod="openshift-ingress/e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc" Sep 16 07:53:09.221537 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:09.221512 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-istio-envoy\") pod \"e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc\" (UID: \"bbc18e51-8079-4c30-a3e0-98bd8ddf132d\") " pod="openshift-ingress/e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc" Sep 16 07:53:09.221762 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:09.221746 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-istio-podinfo\") pod \"e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc\" (UID: \"bbc18e51-8079-4c30-a3e0-98bd8ddf132d\") " pod="openshift-ingress/e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc" Sep 16 07:53:09.234968 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:09.234935 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-istio-token\") pod \"e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc\" (UID: \"bbc18e51-8079-4c30-a3e0-98bd8ddf132d\") " pod="openshift-ingress/e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc" Sep 16 07:53:09.235124 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:09.235056 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhv2b\" (UniqueName: \"kubernetes.io/projected/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-kube-api-access-vhv2b\") pod \"e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc\" (UID: \"bbc18e51-8079-4c30-a3e0-98bd8ddf132d\") " pod="openshift-ingress/e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc" Sep 16 07:53:09.307307 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:09.307204 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc" Sep 16 07:53:09.339850 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:09.339808 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:53:09.589300 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:09.589224 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc"] Sep 16 07:53:09.591893 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:53:09.591861 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbc18e51_8079_4c30_a3e0_98bd8ddf132d.slice/crio-54886cecefe8de17eeb02fe156e5224142d5276522277effde08f01ea0020913 WatchSource:0}: Error finding container 54886cecefe8de17eeb02fe156e5224142d5276522277effde08f01ea0020913: Status 404 returned error can't find the container with id 54886cecefe8de17eeb02fe156e5224142d5276522277effde08f01ea0020913 Sep 16 07:53:09.594304 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:09.594271 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 16 07:53:09.594376 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:09.594338 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 16 07:53:09.594376 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:09.594371 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 16 07:53:10.475311 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:10.475275 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc" event={"ID":"bbc18e51-8079-4c30-a3e0-98bd8ddf132d","Type":"ContainerStarted","Data":"c50615ee17e702b924901336e12551ab3c48fd07530a1c312fcb1287a0da2c71"} Sep 16 07:53:10.475785 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:10.475339 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc" event={"ID":"bbc18e51-8079-4c30-a3e0-98bd8ddf132d","Type":"ContainerStarted","Data":"54886cecefe8de17eeb02fe156e5224142d5276522277effde08f01ea0020913"} Sep 16 07:53:10.498908 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:10.498846 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc" podStartSLOduration=2.498824378 podStartE2EDuration="2.498824378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:53:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:53:10.498816873 +0000 UTC m=+1162.082595224" watchObservedRunningTime="2026-09-16 07:53:10.498824378 +0000 UTC m=+1162.082602718" Sep 16 07:53:10.778655 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:10.778563 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:53:10.871415 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:10.871379 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:53:10.972130 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:10.972070 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:53:11.307413 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:11.307372 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc" Sep 16 07:53:11.313149 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:11.313119 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc" Sep 16 07:53:11.481126 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:11.481061 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc" Sep 16 07:53:11.482290 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:11.482272 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc" Sep 16 07:53:26.059349 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:26.059313 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:53:29.068884 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:29.068845 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:53:29.687531 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:29.687485 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:53:30.472044 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:30.471996 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:53:31.285521 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:31.285478 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:53:31.916618 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:31.916585 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:53:32.385174 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:32.385134 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:53:32.666667 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:32.666582 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:53:33.480879 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:33.480846 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:53:35.234345 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:35.234304 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:53:35.293319 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:35.293278 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:53:41.436880 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:41.436842 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:53:41.807733 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:41.807599 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:53:41.906384 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:41.906342 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:53:44.345397 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:44.345360 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:53:44.646879 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:44.646776 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:53:44.749499 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:44.749462 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:53:45.191404 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:45.191371 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:53:45.950908 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:45.950874 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:53:46.608172 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:46.608074 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:53:47.266351 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:47.266321 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:53:47.971938 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:47.971896 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:53:48.611907 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:48.611872 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:53:48.712713 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:48.712673 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:53:48.957635 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:48.957599 2552 scope.go:117] "RemoveContainer" containerID="008a85a87484c77e6a7f0f338b7e87b49a828b7c8cd7d25007dc167a732e9816" Sep 16 07:53:49.046343 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:49.046308 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:53:49.150005 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:49.149965 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:53:49.240452 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:49.240354 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:53:49.241455 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:49.241430 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fc2a294d-7545bf669d-fsczl"] Sep 16 07:53:49.246546 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:49.246527 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fc2a294d-7545bf669d-fsczl" Sep 16 07:53:49.248874 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:49.248848 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-fc2a294d\"" Sep 16 07:53:49.249002 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:49.248855 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-fc2a294d-dockercfg-ws8s7\"" Sep 16 07:53:49.253945 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:49.253917 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fc2a294d-7545bf669d-fsczl"] Sep 16 07:53:49.347120 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:49.347061 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjt4v\" (UniqueName: \"kubernetes.io/projected/665010a2-d972-420d-bf96-fa90f4da5074-kube-api-access-hjt4v\") pod \"payload-pre-processing-e2e-mt-fc2a294d-7545bf669d-fsczl\" (UID: \"665010a2-d972-420d-bf96-fa90f4da5074\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fc2a294d-7545bf669d-fsczl" Sep 16 07:53:49.347319 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:49.347193 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/665010a2-d972-420d-bf96-fa90f4da5074-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fc2a294d-7545bf669d-fsczl\" (UID: \"665010a2-d972-420d-bf96-fa90f4da5074\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fc2a294d-7545bf669d-fsczl" Sep 16 07:53:49.448375 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:49.448333 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjt4v\" (UniqueName: \"kubernetes.io/projected/665010a2-d972-420d-bf96-fa90f4da5074-kube-api-access-hjt4v\") pod \"payload-pre-processing-e2e-mt-fc2a294d-7545bf669d-fsczl\" (UID: \"665010a2-d972-420d-bf96-fa90f4da5074\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fc2a294d-7545bf669d-fsczl" Sep 16 07:53:49.448566 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:49.448394 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/665010a2-d972-420d-bf96-fa90f4da5074-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fc2a294d-7545bf669d-fsczl\" (UID: \"665010a2-d972-420d-bf96-fa90f4da5074\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fc2a294d-7545bf669d-fsczl" Sep 16 07:53:49.449115 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:49.449081 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/665010a2-d972-420d-bf96-fa90f4da5074-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-fc2a294d-7545bf669d-fsczl\" (UID: \"665010a2-d972-420d-bf96-fa90f4da5074\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fc2a294d-7545bf669d-fsczl" Sep 16 07:53:49.456526 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:49.456494 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjt4v\" (UniqueName: \"kubernetes.io/projected/665010a2-d972-420d-bf96-fa90f4da5074-kube-api-access-hjt4v\") pod \"payload-pre-processing-e2e-mt-fc2a294d-7545bf669d-fsczl\" (UID: \"665010a2-d972-420d-bf96-fa90f4da5074\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-fc2a294d-7545bf669d-fsczl" Sep 16 07:53:49.560631 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:49.560525 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fc2a294d-7545bf669d-fsczl" Sep 16 07:53:49.836971 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:49.836940 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fc2a294d-7545bf669d-fsczl"] Sep 16 07:53:49.838987 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:53:49.838956 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod665010a2_d972_420d_bf96_fa90f4da5074.slice/crio-7b708b8c87805c1f1caadfebed2fcb20501a27f56586bf272a13b6eb024f9cdb WatchSource:0}: Error finding container 7b708b8c87805c1f1caadfebed2fcb20501a27f56586bf272a13b6eb024f9cdb: Status 404 returned error can't find the container with id 7b708b8c87805c1f1caadfebed2fcb20501a27f56586bf272a13b6eb024f9cdb Sep 16 07:53:50.695870 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:50.695826 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fc2a294d-7545bf669d-fsczl" event={"ID":"665010a2-d972-420d-bf96-fa90f4da5074","Type":"ContainerStarted","Data":"d5c478ec4a36f71f055c067dbaf77cf9ea5b4fed3d79ecc64385b2c86b8a7f78"} Sep 16 07:53:50.696074 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:50.695889 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fc2a294d-7545bf669d-fsczl" event={"ID":"665010a2-d972-420d-bf96-fa90f4da5074","Type":"ContainerStarted","Data":"7b708b8c87805c1f1caadfebed2fcb20501a27f56586bf272a13b6eb024f9cdb"} Sep 16 07:53:50.696074 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:50.696042 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-fc2a294d-7545bf669d-fsczl" Sep 16 07:53:50.713658 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:53:50.713601 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-fc2a294d-7545bf669d-fsczl" podStartSLOduration=1.713583686 podStartE2EDuration="1.713583686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:53:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:53:50.71248378 +0000 UTC m=+1202.296262108" watchObservedRunningTime="2026-09-16 07:53:50.713583686 +0000 UTC m=+1202.297362060" Sep 16 07:54:01.704310 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:01.704269 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-fc2a294d-7545bf669d-fsczl" Sep 16 07:54:10.433424 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:10.433390 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:11.792725 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:11.792689 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:11.894167 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:11.894128 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:12.171238 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:12.171203 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5dd30f-9v2dv"] Sep 16 07:54:12.175301 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:12.175273 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5dd30f-9v2dv" Sep 16 07:54:12.177669 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:12.177636 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 07:54:12.177869 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:12.177679 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zwk5r\"" Sep 16 07:54:12.181779 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:12.181750 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5dd30f-9v2dv"] Sep 16 07:54:12.286856 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:12.286816 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea500014-9aa0-4e81-9e5f-cbae84e6a64c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5dd30f-9v2dv\" (UID: \"ea500014-9aa0-4e81-9e5f-cbae84e6a64c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5dd30f-9v2dv" Sep 16 07:54:12.388042 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:12.387993 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea500014-9aa0-4e81-9e5f-cbae84e6a64c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5dd30f-9v2dv\" (UID: \"ea500014-9aa0-4e81-9e5f-cbae84e6a64c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5dd30f-9v2dv" Sep 16 07:54:12.388669 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:12.388647 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea500014-9aa0-4e81-9e5f-cbae84e6a64c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5dd30f-9v2dv\" (UID: \"ea500014-9aa0-4e81-9e5f-cbae84e6a64c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5dd30f-9v2dv" Sep 16 07:54:12.488624 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:12.488524 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5dd30f-9v2dv" Sep 16 07:54:12.597771 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:12.597730 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:12.763691 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:12.763664 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5dd30f-9v2dv"] Sep 16 07:54:12.765613 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:54:12.765574 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea500014_9aa0_4e81_9e5f_cbae84e6a64c.slice/crio-eb28daa9548619df79fa91bade2d870665404380390dd7143867bd4898f994b6 WatchSource:0}: Error finding container eb28daa9548619df79fa91bade2d870665404380390dd7143867bd4898f994b6: Status 404 returned error can't find the container with id eb28daa9548619df79fa91bade2d870665404380390dd7143867bd4898f994b6 Sep 16 07:54:12.828000 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:12.827965 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5dd30f-9v2dv" event={"ID":"ea500014-9aa0-4e81-9e5f-cbae84e6a64c","Type":"ContainerStarted","Data":"eb28daa9548619df79fa91bade2d870665404380390dd7143867bd4898f994b6"} Sep 16 07:54:13.471083 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:13.471048 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:13.834961 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:13.834869 2552 generic.go:356] "Generic (PLEG): container finished" podID="ea500014-9aa0-4e81-9e5f-cbae84e6a64c" containerID="5544bc4b789089294bcd2c5ef3d84f442450c0b952f49e7b340b776ac7956b7e" exitCode=0 Sep 16 07:54:13.834961 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:13.834913 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5dd30f-9v2dv" event={"ID":"ea500014-9aa0-4e81-9e5f-cbae84e6a64c","Type":"ContainerDied","Data":"5544bc4b789089294bcd2c5ef3d84f442450c0b952f49e7b340b776ac7956b7e"} Sep 16 07:54:14.087575 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:14.087487 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:14.694399 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:14.694355 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:14.795715 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:14.795675 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:14.983992 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:14.983959 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5dd30f-9v2dv" Sep 16 07:54:15.119655 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:15.119613 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea500014-9aa0-4e81-9e5f-cbae84e6a64c-maas-api-service-ca\") pod \"ea500014-9aa0-4e81-9e5f-cbae84e6a64c\" (UID: \"ea500014-9aa0-4e81-9e5f-cbae84e6a64c\") " Sep 16 07:54:15.119991 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:15.119967 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea500014-9aa0-4e81-9e5f-cbae84e6a64c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ea500014-9aa0-4e81-9e5f-cbae84e6a64c" (UID: "ea500014-9aa0-4e81-9e5f-cbae84e6a64c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:54:15.220790 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:15.220698 2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea500014-9aa0-4e81-9e5f-cbae84e6a64c-maas-api-service-ca\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:54:15.665874 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:15.665774 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:15.853726 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:15.853692 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5dd30f-9v2dv" Sep 16 07:54:15.853904 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:15.853729 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5dd30f-9v2dv" event={"ID":"ea500014-9aa0-4e81-9e5f-cbae84e6a64c","Type":"ContainerDied","Data":"eb28daa9548619df79fa91bade2d870665404380390dd7143867bd4898f994b6"} Sep 16 07:54:15.853904 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:15.853822 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb28daa9548619df79fa91bade2d870665404380390dd7143867bd4898f994b6" Sep 16 07:54:16.453323 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:16.453283 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:17.269731 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:17.269691 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:17.341578 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:17.341546 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:17.952324 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:17.952289 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:22.301623 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:22.301585 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:22.864475 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:22.864422 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:22.926395 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:22.926360 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:23.028720 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:23.028686 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:23.107355 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:23.107307 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5dd30f-9v2dv"] Sep 16 07:54:23.109371 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:23.109335 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:23.110022 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:23.110001 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5dd30f-9v2dv"] Sep 16 07:54:23.382268 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:23.382232 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:23.483083 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:23.483043 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:24.171689 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:24.171657 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:24.849509 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:24.849472 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:24.938514 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:24.938482 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea500014-9aa0-4e81-9e5f-cbae84e6a64c" path="/var/lib/kubelet/pods/ea500014-9aa0-4e81-9e5f-cbae84e6a64c/volumes" Sep 16 07:54:25.729505 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:25.729471 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:26.365975 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:26.365938 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:26.977153 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:26.977108 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-3beaf8/noten-model-3beaf8-kserve-79d865d78c-ppbh8"] Sep 16 07:54:26.977559 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:26.977543 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea500014-9aa0-4e81-9e5f-cbae84e6a64c" containerName="revoke-keys" Sep 16 07:54:26.977559 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:26.977556 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea500014-9aa0-4e81-9e5f-cbae84e6a64c" containerName="revoke-keys" Sep 16 07:54:26.977743 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:26.977643 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea500014-9aa0-4e81-9e5f-cbae84e6a64c" containerName="revoke-keys" Sep 16 07:54:26.980756 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:26.980728 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-3beaf8/noten-model-3beaf8-kserve-79d865d78c-ppbh8" Sep 16 07:54:26.983145 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:26.983119 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-3beaf8\"/\"kube-root-ca.crt\"" Sep 16 07:54:26.983933 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:26.983906 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-3beaf8\"/\"noten-model-3beaf8-kserve-self-signed-certs\"" Sep 16 07:54:26.984056 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:26.983951 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-3beaf8\"/\"default-dockercfg-lh2gv\"" Sep 16 07:54:26.984056 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:26.983957 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-3beaf8\"/\"openshift-service-ca.crt\"" Sep 16 07:54:26.988889 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:26.988837 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-3beaf8/noten-model-3beaf8-kserve-79d865d78c-ppbh8"] Sep 16 07:54:27.007015 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:27.006974 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:27.042322 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:27.042282 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7124850f-b022-46e4-97b7-ba749a51b48b-home\") pod \"noten-model-3beaf8-kserve-79d865d78c-ppbh8\" (UID: \"7124850f-b022-46e4-97b7-ba749a51b48b\") " pod="ai-tenant-e2e-noten-3beaf8/noten-model-3beaf8-kserve-79d865d78c-ppbh8" Sep 16 07:54:27.042503 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:27.042389 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7124850f-b022-46e4-97b7-ba749a51b48b-dshm\") pod \"noten-model-3beaf8-kserve-79d865d78c-ppbh8\" (UID: \"7124850f-b022-46e4-97b7-ba749a51b48b\") " pod="ai-tenant-e2e-noten-3beaf8/noten-model-3beaf8-kserve-79d865d78c-ppbh8" Sep 16 07:54:27.042503 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:27.042416 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9gfx\" (UniqueName: \"kubernetes.io/projected/7124850f-b022-46e4-97b7-ba749a51b48b-kube-api-access-r9gfx\") pod \"noten-model-3beaf8-kserve-79d865d78c-ppbh8\" (UID: \"7124850f-b022-46e4-97b7-ba749a51b48b\") " pod="ai-tenant-e2e-noten-3beaf8/noten-model-3beaf8-kserve-79d865d78c-ppbh8" Sep 16 07:54:27.042503 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:27.042470 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7124850f-b022-46e4-97b7-ba749a51b48b-model-cache\") pod \"noten-model-3beaf8-kserve-79d865d78c-ppbh8\" (UID: \"7124850f-b022-46e4-97b7-ba749a51b48b\") " pod="ai-tenant-e2e-noten-3beaf8/noten-model-3beaf8-kserve-79d865d78c-ppbh8" Sep 16 07:54:27.042611 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:27.042521 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7124850f-b022-46e4-97b7-ba749a51b48b-tls-certs\") pod \"noten-model-3beaf8-kserve-79d865d78c-ppbh8\" (UID: \"7124850f-b022-46e4-97b7-ba749a51b48b\") " pod="ai-tenant-e2e-noten-3beaf8/noten-model-3beaf8-kserve-79d865d78c-ppbh8" Sep 16 07:54:27.042611 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:27.042550 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7124850f-b022-46e4-97b7-ba749a51b48b-tmp-dir\") pod \"noten-model-3beaf8-kserve-79d865d78c-ppbh8\" (UID: \"7124850f-b022-46e4-97b7-ba749a51b48b\") " pod="ai-tenant-e2e-noten-3beaf8/noten-model-3beaf8-kserve-79d865d78c-ppbh8" Sep 16 07:54:27.108447 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:27.108414 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:27.143394 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:27.143355 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7124850f-b022-46e4-97b7-ba749a51b48b-dshm\") pod \"noten-model-3beaf8-kserve-79d865d78c-ppbh8\" (UID: \"7124850f-b022-46e4-97b7-ba749a51b48b\") " pod="ai-tenant-e2e-noten-3beaf8/noten-model-3beaf8-kserve-79d865d78c-ppbh8" Sep 16 07:54:27.143394 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:27.143392 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9gfx\" (UniqueName: \"kubernetes.io/projected/7124850f-b022-46e4-97b7-ba749a51b48b-kube-api-access-r9gfx\") pod \"noten-model-3beaf8-kserve-79d865d78c-ppbh8\" (UID: \"7124850f-b022-46e4-97b7-ba749a51b48b\") " pod="ai-tenant-e2e-noten-3beaf8/noten-model-3beaf8-kserve-79d865d78c-ppbh8" Sep 16 07:54:27.143652 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:27.143419 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7124850f-b022-46e4-97b7-ba749a51b48b-model-cache\") pod \"noten-model-3beaf8-kserve-79d865d78c-ppbh8\" (UID: \"7124850f-b022-46e4-97b7-ba749a51b48b\") " pod="ai-tenant-e2e-noten-3beaf8/noten-model-3beaf8-kserve-79d865d78c-ppbh8" Sep 16 07:54:27.143652 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:27.143450 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7124850f-b022-46e4-97b7-ba749a51b48b-tls-certs\") pod \"noten-model-3beaf8-kserve-79d865d78c-ppbh8\" (UID: \"7124850f-b022-46e4-97b7-ba749a51b48b\") " pod="ai-tenant-e2e-noten-3beaf8/noten-model-3beaf8-kserve-79d865d78c-ppbh8" Sep 16 07:54:27.143652 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:27.143475 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7124850f-b022-46e4-97b7-ba749a51b48b-tmp-dir\") pod \"noten-model-3beaf8-kserve-79d865d78c-ppbh8\" (UID: \"7124850f-b022-46e4-97b7-ba749a51b48b\") " pod="ai-tenant-e2e-noten-3beaf8/noten-model-3beaf8-kserve-79d865d78c-ppbh8" Sep 16 07:54:27.143652 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:27.143533 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7124850f-b022-46e4-97b7-ba749a51b48b-home\") pod \"noten-model-3beaf8-kserve-79d865d78c-ppbh8\" (UID: \"7124850f-b022-46e4-97b7-ba749a51b48b\") " pod="ai-tenant-e2e-noten-3beaf8/noten-model-3beaf8-kserve-79d865d78c-ppbh8" Sep 16 07:54:27.143924 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:27.143901 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7124850f-b022-46e4-97b7-ba749a51b48b-model-cache\") pod \"noten-model-3beaf8-kserve-79d865d78c-ppbh8\" (UID: \"7124850f-b022-46e4-97b7-ba749a51b48b\") " pod="ai-tenant-e2e-noten-3beaf8/noten-model-3beaf8-kserve-79d865d78c-ppbh8" Sep 16 07:54:27.143986 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:27.143942 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7124850f-b022-46e4-97b7-ba749a51b48b-tmp-dir\") pod \"noten-model-3beaf8-kserve-79d865d78c-ppbh8\" (UID: \"7124850f-b022-46e4-97b7-ba749a51b48b\") " pod="ai-tenant-e2e-noten-3beaf8/noten-model-3beaf8-kserve-79d865d78c-ppbh8" Sep 16 07:54:27.143986 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:27.143979 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7124850f-b022-46e4-97b7-ba749a51b48b-home\") pod \"noten-model-3beaf8-kserve-79d865d78c-ppbh8\" (UID: \"7124850f-b022-46e4-97b7-ba749a51b48b\") " pod="ai-tenant-e2e-noten-3beaf8/noten-model-3beaf8-kserve-79d865d78c-ppbh8" Sep 16 07:54:27.146300 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:27.146264 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7124850f-b022-46e4-97b7-ba749a51b48b-dshm\") pod \"noten-model-3beaf8-kserve-79d865d78c-ppbh8\" (UID: \"7124850f-b022-46e4-97b7-ba749a51b48b\") " pod="ai-tenant-e2e-noten-3beaf8/noten-model-3beaf8-kserve-79d865d78c-ppbh8" Sep 16 07:54:27.146553 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:27.146533 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7124850f-b022-46e4-97b7-ba749a51b48b-tls-certs\") pod \"noten-model-3beaf8-kserve-79d865d78c-ppbh8\" (UID: \"7124850f-b022-46e4-97b7-ba749a51b48b\") " pod="ai-tenant-e2e-noten-3beaf8/noten-model-3beaf8-kserve-79d865d78c-ppbh8" Sep 16 07:54:27.151758 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:27.151729 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9gfx\" (UniqueName: \"kubernetes.io/projected/7124850f-b022-46e4-97b7-ba749a51b48b-kube-api-access-r9gfx\") pod \"noten-model-3beaf8-kserve-79d865d78c-ppbh8\" (UID: \"7124850f-b022-46e4-97b7-ba749a51b48b\") " pod="ai-tenant-e2e-noten-3beaf8/noten-model-3beaf8-kserve-79d865d78c-ppbh8" Sep 16 07:54:27.292766 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:27.292679 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-3beaf8/noten-model-3beaf8-kserve-79d865d78c-ppbh8" Sep 16 07:54:27.572822 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:27.572794 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-3beaf8/noten-model-3beaf8-kserve-79d865d78c-ppbh8"] Sep 16 07:54:27.575172 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:54:27.575147 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7124850f_b022_46e4_97b7_ba749a51b48b.slice/crio-1d3426d1bdbf4a2721843bf061b40215b67c2a8a808fa1c8aa8b5523ff09500e WatchSource:0}: Error finding container 1d3426d1bdbf4a2721843bf061b40215b67c2a8a808fa1c8aa8b5523ff09500e: Status 404 returned error can't find the container with id 1d3426d1bdbf4a2721843bf061b40215b67c2a8a808fa1c8aa8b5523ff09500e Sep 16 07:54:27.718742 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:27.718711 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:27.926609 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:27.926572 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-3beaf8/noten-model-3beaf8-kserve-79d865d78c-ppbh8" event={"ID":"7124850f-b022-46e4-97b7-ba749a51b48b","Type":"ContainerStarted","Data":"1cb6d3a5c6b393dbf50d3e6087a294203f219725e24d069480dee14904420b4f"} Sep 16 07:54:27.926609 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:27.926613 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-3beaf8/noten-model-3beaf8-kserve-79d865d78c-ppbh8" event={"ID":"7124850f-b022-46e4-97b7-ba749a51b48b","Type":"ContainerStarted","Data":"1d3426d1bdbf4a2721843bf061b40215b67c2a8a808fa1c8aa8b5523ff09500e"} Sep 16 07:54:27.945229 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:27.945162 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-3beaf8/noten-model-3beaf8-kserve-79d865d78c-ppbh8" podStartSLOduration=1.945138856 podStartE2EDuration="1.945138856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:54:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:54:27.942692561 +0000 UTC m=+1239.526470911" watchObservedRunningTime="2026-09-16 07:54:27.945138856 +0000 UTC m=+1239.528917209" Sep 16 07:54:28.543288 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:28.543252 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:34.200054 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:34.200006 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:34.451921 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:34.451823 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:35.164241 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:35.164206 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:35.961160 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:35.961126 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:36.591690 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:36.591650 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:37.293495 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:37.293432 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-3beaf8/noten-model-3beaf8-kserve-79d865d78c-ppbh8" Sep 16 07:54:37.293917 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:37.293519 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-3beaf8/noten-model-3beaf8-kserve-79d865d78c-ppbh8" Sep 16 07:54:37.306583 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:37.306545 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-3beaf8/noten-model-3beaf8-kserve-79d865d78c-ppbh8" Sep 16 07:54:37.349818 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:37.349782 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:37.973368 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:37.973332 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:37.995636 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:37.995605 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-3beaf8/noten-model-3beaf8-kserve-79d865d78c-ppbh8" Sep 16 07:54:38.615849 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:38.615808 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:39.373782 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:39.373744 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:39.988131 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:39.988070 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:42.643184 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:42.643075 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:44.960340 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:44.960307 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:45.504974 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:45.504934 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:45.530236 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:45.530197 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-dd359c-7cccdb9847-6sb9l"] Sep 16 07:54:45.530621 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:45.530572 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-dd359c-7cccdb9847-6sb9l" podUID="968b978d-ccf1-4116-876b-357bd1541402" containerName="payload-processing" containerID="cri-o://c8f94c687d05cc374c73a0b2d9ba495b600b46c15d3f1e69d020a8c2598440bb" gracePeriod=30 Sep 16 07:54:45.534772 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:45.534744 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-dd359c-69f4889858-cf67d"] Sep 16 07:54:45.535238 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:45.535203 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd359c-69f4889858-cf67d" podUID="06c531c4-2b1c-47fd-afa3-6473fc38135f" containerName="payload-pre-processing" containerID="cri-o://fb2155bd9b510c702759270b0103ea6b633444d99ab3bad585e36b3c9a20050d" gracePeriod=30 Sep 16 07:54:45.570155 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:45.570119 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:45.609809 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:45.609766 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd359c-69f4889858-cf67d" podUID="06c531c4-2b1c-47fd-afa3-6473fc38135f" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.133.0.39:9004: connect: connection refused" Sep 16 07:54:45.609993 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:45.609771 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-dd359c-7cccdb9847-6sb9l" podUID="968b978d-ccf1-4116-876b-357bd1541402" containerName="payload-processing" probeResult="failure" output="dial tcp 10.133.0.40:9004: connect: connection refused" Sep 16 07:54:45.627169 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:45.627140 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:45.685546 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:45.685499 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:45.741172 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:45.741121 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:45.838072 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:45.838044 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-dd359c-7cccdb9847-6sb9l" Sep 16 07:54:45.841347 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:45.841322 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd359c-69f4889858-cf67d" Sep 16 07:54:45.946787 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:45.946746 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/968b978d-ccf1-4116-876b-357bd1541402-plugins-config-volume\") pod \"968b978d-ccf1-4116-876b-357bd1541402\" (UID: \"968b978d-ccf1-4116-876b-357bd1541402\") " Sep 16 07:54:45.946971 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:45.946797 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06c531c4-2b1c-47fd-afa3-6473fc38135f-plugins-config-volume\") pod \"06c531c4-2b1c-47fd-afa3-6473fc38135f\" (UID: \"06c531c4-2b1c-47fd-afa3-6473fc38135f\") " Sep 16 07:54:45.946971 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:45.946928 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgthw\" (UniqueName: \"kubernetes.io/projected/968b978d-ccf1-4116-876b-357bd1541402-kube-api-access-sgthw\") pod \"968b978d-ccf1-4116-876b-357bd1541402\" (UID: \"968b978d-ccf1-4116-876b-357bd1541402\") " Sep 16 07:54:45.947117 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:45.947009 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-89l25\" (UniqueName: \"kubernetes.io/projected/06c531c4-2b1c-47fd-afa3-6473fc38135f-kube-api-access-89l25\") pod \"06c531c4-2b1c-47fd-afa3-6473fc38135f\" (UID: \"06c531c4-2b1c-47fd-afa3-6473fc38135f\") " Sep 16 07:54:45.947279 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:45.947254 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/968b978d-ccf1-4116-876b-357bd1541402-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "968b978d-ccf1-4116-876b-357bd1541402" (UID: "968b978d-ccf1-4116-876b-357bd1541402"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:54:45.947279 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:45.947259 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06c531c4-2b1c-47fd-afa3-6473fc38135f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "06c531c4-2b1c-47fd-afa3-6473fc38135f" (UID: "06c531c4-2b1c-47fd-afa3-6473fc38135f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:54:45.947530 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:45.947505 2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/968b978d-ccf1-4116-876b-357bd1541402-plugins-config-volume\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:54:45.947657 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:45.947534 2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06c531c4-2b1c-47fd-afa3-6473fc38135f-plugins-config-volume\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:54:45.949233 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:45.949197 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/968b978d-ccf1-4116-876b-357bd1541402-kube-api-access-sgthw" (OuterVolumeSpecName: "kube-api-access-sgthw") pod "968b978d-ccf1-4116-876b-357bd1541402" (UID: "968b978d-ccf1-4116-876b-357bd1541402"). InnerVolumeSpecName "kube-api-access-sgthw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:54:45.949355 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:45.949244 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06c531c4-2b1c-47fd-afa3-6473fc38135f-kube-api-access-89l25" (OuterVolumeSpecName: "kube-api-access-89l25") pod "06c531c4-2b1c-47fd-afa3-6473fc38135f" (UID: "06c531c4-2b1c-47fd-afa3-6473fc38135f"). InnerVolumeSpecName "kube-api-access-89l25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:54:46.026850 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:46.026812 2552 generic.go:356] "Generic (PLEG): container finished" podID="968b978d-ccf1-4116-876b-357bd1541402" containerID="c8f94c687d05cc374c73a0b2d9ba495b600b46c15d3f1e69d020a8c2598440bb" exitCode=0 Sep 16 07:54:46.027303 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:46.026858 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-dd359c-7cccdb9847-6sb9l" Sep 16 07:54:46.027303 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:46.026895 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-dd359c-7cccdb9847-6sb9l" event={"ID":"968b978d-ccf1-4116-876b-357bd1541402","Type":"ContainerDied","Data":"c8f94c687d05cc374c73a0b2d9ba495b600b46c15d3f1e69d020a8c2598440bb"} Sep 16 07:54:46.027303 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:46.026946 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-dd359c-7cccdb9847-6sb9l" event={"ID":"968b978d-ccf1-4116-876b-357bd1541402","Type":"ContainerDied","Data":"c7a03e6834f67d60a7c1046138771b9e8e1dc7a2686f2c6f80ef2e7217b047ec"} Sep 16 07:54:46.027303 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:46.026968 2552 scope.go:117] "RemoveContainer" containerID="c8f94c687d05cc374c73a0b2d9ba495b600b46c15d3f1e69d020a8c2598440bb" Sep 16 07:54:46.028555 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:46.028529 2552 generic.go:356] "Generic (PLEG): container finished" podID="06c531c4-2b1c-47fd-afa3-6473fc38135f" containerID="fb2155bd9b510c702759270b0103ea6b633444d99ab3bad585e36b3c9a20050d" exitCode=0 Sep 16 07:54:46.028687 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:46.028560 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd359c-69f4889858-cf67d" event={"ID":"06c531c4-2b1c-47fd-afa3-6473fc38135f","Type":"ContainerDied","Data":"fb2155bd9b510c702759270b0103ea6b633444d99ab3bad585e36b3c9a20050d"} Sep 16 07:54:46.028687 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:46.028594 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd359c-69f4889858-cf67d" Sep 16 07:54:46.028687 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:46.028609 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-dd359c-69f4889858-cf67d" event={"ID":"06c531c4-2b1c-47fd-afa3-6473fc38135f","Type":"ContainerDied","Data":"e671b82421b57e0c11afcb8f466580ccf20c4ea4515adeb13fe6d0a980a880de"} Sep 16 07:54:46.038085 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:46.037796 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:46.042308 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:46.042285 2552 scope.go:117] "RemoveContainer" containerID="c8f94c687d05cc374c73a0b2d9ba495b600b46c15d3f1e69d020a8c2598440bb" Sep 16 07:54:46.042642 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:54:46.042623 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8f94c687d05cc374c73a0b2d9ba495b600b46c15d3f1e69d020a8c2598440bb\": container with ID starting with c8f94c687d05cc374c73a0b2d9ba495b600b46c15d3f1e69d020a8c2598440bb not found: ID does not exist" containerID="c8f94c687d05cc374c73a0b2d9ba495b600b46c15d3f1e69d020a8c2598440bb" Sep 16 07:54:46.042689 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:46.042653 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8f94c687d05cc374c73a0b2d9ba495b600b46c15d3f1e69d020a8c2598440bb"} err="failed to get container status \"c8f94c687d05cc374c73a0b2d9ba495b600b46c15d3f1e69d020a8c2598440bb\": rpc error: code = NotFound desc = could not find container \"c8f94c687d05cc374c73a0b2d9ba495b600b46c15d3f1e69d020a8c2598440bb\": container with ID starting with c8f94c687d05cc374c73a0b2d9ba495b600b46c15d3f1e69d020a8c2598440bb not found: ID does not exist" Sep 16 07:54:46.042689 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:46.042670 2552 scope.go:117] "RemoveContainer" containerID="fb2155bd9b510c702759270b0103ea6b633444d99ab3bad585e36b3c9a20050d" Sep 16 07:54:46.048413 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:46.048383 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgthw\" (UniqueName: \"kubernetes.io/projected/968b978d-ccf1-4116-876b-357bd1541402-kube-api-access-sgthw\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:54:46.048549 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:46.048416 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89l25\" (UniqueName: \"kubernetes.io/projected/06c531c4-2b1c-47fd-afa3-6473fc38135f-kube-api-access-89l25\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:54:46.051353 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:46.051328 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-dd359c-7cccdb9847-6sb9l"] Sep 16 07:54:46.054310 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:46.054287 2552 scope.go:117] "RemoveContainer" containerID="fb2155bd9b510c702759270b0103ea6b633444d99ab3bad585e36b3c9a20050d" Sep 16 07:54:46.054664 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:54:46.054646 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb2155bd9b510c702759270b0103ea6b633444d99ab3bad585e36b3c9a20050d\": container with ID starting with fb2155bd9b510c702759270b0103ea6b633444d99ab3bad585e36b3c9a20050d not found: ID does not exist" containerID="fb2155bd9b510c702759270b0103ea6b633444d99ab3bad585e36b3c9a20050d" Sep 16 07:54:46.054710 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:46.054672 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb2155bd9b510c702759270b0103ea6b633444d99ab3bad585e36b3c9a20050d"} err="failed to get container status \"fb2155bd9b510c702759270b0103ea6b633444d99ab3bad585e36b3c9a20050d\": rpc error: code = NotFound desc = could not find container \"fb2155bd9b510c702759270b0103ea6b633444d99ab3bad585e36b3c9a20050d\": container with ID starting with fb2155bd9b510c702759270b0103ea6b633444d99ab3bad585e36b3c9a20050d not found: ID does not exist" Sep 16 07:54:46.064182 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:46.064070 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-dd359c-7cccdb9847-6sb9l"] Sep 16 07:54:46.079291 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:46.079253 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-dd359c-69f4889858-cf67d"] Sep 16 07:54:46.081285 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:46.081263 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-dd359c-69f4889858-cf67d"] Sep 16 07:54:46.158689 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:46.158658 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:46.799409 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:46.799376 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:46.937924 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:46.937893 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06c531c4-2b1c-47fd-afa3-6473fc38135f" path="/var/lib/kubelet/pods/06c531c4-2b1c-47fd-afa3-6473fc38135f/volumes" Sep 16 07:54:46.938403 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:46.938383 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="968b978d-ccf1-4116-876b-357bd1541402" path="/var/lib/kubelet/pods/968b978d-ccf1-4116-876b-357bd1541402/volumes" Sep 16 07:54:47.577687 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:47.577649 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:48.226534 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:48.226489 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:48.867182 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:48.867140 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:49.643799 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:49.643764 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:51.072285 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:51.072234 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:59.081815 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:59.081761 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:59.317389 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:59.317347 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc"] Sep 16 07:54:59.317610 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:59.317572 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc" podUID="bbc18e51-8079-4c30-a3e0-98bd8ddf132d" containerName="istio-proxy" containerID="cri-o://c50615ee17e702b924901336e12551ab3c48fd07530a1c312fcb1287a0da2c71" gracePeriod=30 Sep 16 07:54:59.337220 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:59.337075 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:59.540386 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:59.540343 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:54:59.704251 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:54:59.704214 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:55:00.460667 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:00.460631 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:55:00.512369 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:00.512335 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:55:04.583706 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:04.583637 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc" Sep 16 07:55:04.641372 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:04.641337 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-istio-envoy\") pod \"bbc18e51-8079-4c30-a3e0-98bd8ddf132d\" (UID: \"bbc18e51-8079-4c30-a3e0-98bd8ddf132d\") " Sep 16 07:55:04.641556 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:04.641407 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-istiod-ca-cert\") pod \"bbc18e51-8079-4c30-a3e0-98bd8ddf132d\" (UID: \"bbc18e51-8079-4c30-a3e0-98bd8ddf132d\") " Sep 16 07:55:04.641556 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:04.641441 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-credential-socket\") pod \"bbc18e51-8079-4c30-a3e0-98bd8ddf132d\" (UID: \"bbc18e51-8079-4c30-a3e0-98bd8ddf132d\") " Sep 16 07:55:04.641556 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:04.641469 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-istio-podinfo\") pod \"bbc18e51-8079-4c30-a3e0-98bd8ddf132d\" (UID: \"bbc18e51-8079-4c30-a3e0-98bd8ddf132d\") " Sep 16 07:55:04.641690 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:04.641581 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-istio-token\") pod \"bbc18e51-8079-4c30-a3e0-98bd8ddf132d\" (UID: \"bbc18e51-8079-4c30-a3e0-98bd8ddf132d\") " Sep 16 07:55:04.641690 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:04.641643 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-workload-socket\") pod \"bbc18e51-8079-4c30-a3e0-98bd8ddf132d\" (UID: \"bbc18e51-8079-4c30-a3e0-98bd8ddf132d\") " Sep 16 07:55:04.641791 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:04.641694 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-istio-data\") pod \"bbc18e51-8079-4c30-a3e0-98bd8ddf132d\" (UID: \"bbc18e51-8079-4c30-a3e0-98bd8ddf132d\") " Sep 16 07:55:04.641791 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:04.641723 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhv2b\" (UniqueName: \"kubernetes.io/projected/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-kube-api-access-vhv2b\") pod \"bbc18e51-8079-4c30-a3e0-98bd8ddf132d\" (UID: \"bbc18e51-8079-4c30-a3e0-98bd8ddf132d\") " Sep 16 07:55:04.641975 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:04.641781 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bbc18e51-8079-4c30-a3e0-98bd8ddf132d" (UID: "bbc18e51-8079-4c30-a3e0-98bd8ddf132d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:55:04.641975 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:04.641833 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-workload-certs\") pod \"bbc18e51-8079-4c30-a3e0-98bd8ddf132d\" (UID: \"bbc18e51-8079-4c30-a3e0-98bd8ddf132d\") " Sep 16 07:55:04.641975 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:04.641885 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bbc18e51-8079-4c30-a3e0-98bd8ddf132d" (UID: "bbc18e51-8079-4c30-a3e0-98bd8ddf132d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:55:04.642172 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:04.642003 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bbc18e51-8079-4c30-a3e0-98bd8ddf132d" (UID: "bbc18e51-8079-4c30-a3e0-98bd8ddf132d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:55:04.642241 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:04.642185 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-istio-data" (OuterVolumeSpecName: "istio-data") pod "bbc18e51-8079-4c30-a3e0-98bd8ddf132d" (UID: "bbc18e51-8079-4c30-a3e0-98bd8ddf132d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:55:04.642241 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:04.642213 2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-istiod-ca-cert\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:55:04.642241 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:04.642235 2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-credential-socket\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:55:04.642399 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:04.642251 2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-workload-socket\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:55:04.642399 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:04.642366 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bbc18e51-8079-4c30-a3e0-98bd8ddf132d" (UID: "bbc18e51-8079-4c30-a3e0-98bd8ddf132d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:55:04.643458 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:04.643430 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-3beaf8/noten-model-3beaf8-kserve-79d865d78c-ppbh8"] Sep 16 07:55:04.643824 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:04.643765 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bbc18e51-8079-4c30-a3e0-98bd8ddf132d" (UID: "bbc18e51-8079-4c30-a3e0-98bd8ddf132d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:55:04.643982 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:04.643952 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-istio-token" (OuterVolumeSpecName: "istio-token") pod "bbc18e51-8079-4c30-a3e0-98bd8ddf132d" (UID: "bbc18e51-8079-4c30-a3e0-98bd8ddf132d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:55:04.644245 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:04.644127 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-kube-api-access-vhv2b" (OuterVolumeSpecName: "kube-api-access-vhv2b") pod "bbc18e51-8079-4c30-a3e0-98bd8ddf132d" (UID: "bbc18e51-8079-4c30-a3e0-98bd8ddf132d"). InnerVolumeSpecName "kube-api-access-vhv2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:55:04.644245 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:04.644209 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bbc18e51-8079-4c30-a3e0-98bd8ddf132d" (UID: "bbc18e51-8079-4c30-a3e0-98bd8ddf132d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 07:55:04.686203 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:04.686167 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:55:04.743432 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:04.743390 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhv2b\" (UniqueName: \"kubernetes.io/projected/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-kube-api-access-vhv2b\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:55:04.743432 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:04.743426 2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-workload-certs\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:55:04.743432 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:04.743440 2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-istio-envoy\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:55:04.743679 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:04.743452 2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-istio-podinfo\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:55:04.743679 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:04.743461 2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-istio-token\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:55:04.743679 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:04.743469 2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bbc18e51-8079-4c30-a3e0-98bd8ddf132d-istio-data\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:55:05.136321 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:05.136288 2552 generic.go:356] "Generic (PLEG): container finished" podID="bbc18e51-8079-4c30-a3e0-98bd8ddf132d" containerID="c50615ee17e702b924901336e12551ab3c48fd07530a1c312fcb1287a0da2c71" exitCode=0 Sep 16 07:55:05.136475 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:05.136342 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc" Sep 16 07:55:05.136475 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:05.136375 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc" event={"ID":"bbc18e51-8079-4c30-a3e0-98bd8ddf132d","Type":"ContainerDied","Data":"c50615ee17e702b924901336e12551ab3c48fd07530a1c312fcb1287a0da2c71"} Sep 16 07:55:05.136475 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:05.136434 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc" event={"ID":"bbc18e51-8079-4c30-a3e0-98bd8ddf132d","Type":"ContainerDied","Data":"54886cecefe8de17eeb02fe156e5224142d5276522277effde08f01ea0020913"} Sep 16 07:55:05.136475 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:05.136463 2552 scope.go:117] "RemoveContainer" containerID="c50615ee17e702b924901336e12551ab3c48fd07530a1c312fcb1287a0da2c71" Sep 16 07:55:05.145913 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:05.145888 2552 scope.go:117] "RemoveContainer" containerID="c50615ee17e702b924901336e12551ab3c48fd07530a1c312fcb1287a0da2c71" Sep 16 07:55:05.146298 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:55:05.146276 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c50615ee17e702b924901336e12551ab3c48fd07530a1c312fcb1287a0da2c71\": container with ID starting with c50615ee17e702b924901336e12551ab3c48fd07530a1c312fcb1287a0da2c71 not found: ID does not exist" containerID="c50615ee17e702b924901336e12551ab3c48fd07530a1c312fcb1287a0da2c71" Sep 16 07:55:05.146368 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:05.146308 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c50615ee17e702b924901336e12551ab3c48fd07530a1c312fcb1287a0da2c71"} err="failed to get container status \"c50615ee17e702b924901336e12551ab3c48fd07530a1c312fcb1287a0da2c71\": rpc error: code = NotFound desc = could not find container \"c50615ee17e702b924901336e12551ab3c48fd07530a1c312fcb1287a0da2c71\": container with ID starting with c50615ee17e702b924901336e12551ab3c48fd07530a1c312fcb1287a0da2c71 not found: ID does not exist" Sep 16 07:55:05.154327 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:05.154296 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc"] Sep 16 07:55:05.158203 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:05.158172 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-3beaf8-openshift-default-54dc95c84b-kjkgc"] Sep 16 07:55:06.938711 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:06.938675 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbc18e51-8079-4c30-a3e0-98bd8ddf132d" path="/var/lib/kubelet/pods/bbc18e51-8079-4c30-a3e0-98bd8ddf132d/volumes" Sep 16 07:55:11.193081 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:11.193018 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:55:19.663677 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:19.663627 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-3beaf8/noten-model-3beaf8-kserve-79d865d78c-ppbh8" podUID="7124850f-b022-46e4-97b7-ba749a51b48b" containerName="main" containerID="cri-o://1cb6d3a5c6b393dbf50d3e6087a294203f219725e24d069480dee14904420b4f" gracePeriod=45 Sep 16 07:55:19.920996 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:19.920965 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-3beaf8/noten-model-3beaf8-kserve-79d865d78c-ppbh8" Sep 16 07:55:20.005346 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:20.005313 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9gfx\" (UniqueName: \"kubernetes.io/projected/7124850f-b022-46e4-97b7-ba749a51b48b-kube-api-access-r9gfx\") pod \"7124850f-b022-46e4-97b7-ba749a51b48b\" (UID: \"7124850f-b022-46e4-97b7-ba749a51b48b\") " Sep 16 07:55:20.005519 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:20.005367 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7124850f-b022-46e4-97b7-ba749a51b48b-dshm\") pod \"7124850f-b022-46e4-97b7-ba749a51b48b\" (UID: \"7124850f-b022-46e4-97b7-ba749a51b48b\") " Sep 16 07:55:20.005519 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:20.005387 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7124850f-b022-46e4-97b7-ba749a51b48b-model-cache\") pod \"7124850f-b022-46e4-97b7-ba749a51b48b\" (UID: \"7124850f-b022-46e4-97b7-ba749a51b48b\") " Sep 16 07:55:20.005519 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:20.005430 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7124850f-b022-46e4-97b7-ba749a51b48b-tls-certs\") pod \"7124850f-b022-46e4-97b7-ba749a51b48b\" (UID: \"7124850f-b022-46e4-97b7-ba749a51b48b\") " Sep 16 07:55:20.005519 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:20.005451 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7124850f-b022-46e4-97b7-ba749a51b48b-home\") pod \"7124850f-b022-46e4-97b7-ba749a51b48b\" (UID: \"7124850f-b022-46e4-97b7-ba749a51b48b\") " Sep 16 07:55:20.005519 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:20.005504 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7124850f-b022-46e4-97b7-ba749a51b48b-tmp-dir\") pod \"7124850f-b022-46e4-97b7-ba749a51b48b\" (UID: \"7124850f-b022-46e4-97b7-ba749a51b48b\") " Sep 16 07:55:20.005815 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:20.005701 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7124850f-b022-46e4-97b7-ba749a51b48b-model-cache" (OuterVolumeSpecName: "model-cache") pod "7124850f-b022-46e4-97b7-ba749a51b48b" (UID: "7124850f-b022-46e4-97b7-ba749a51b48b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:55:20.005873 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:20.005789 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7124850f-b022-46e4-97b7-ba749a51b48b-home" (OuterVolumeSpecName: "home") pod "7124850f-b022-46e4-97b7-ba749a51b48b" (UID: "7124850f-b022-46e4-97b7-ba749a51b48b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:55:20.006417 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:20.006389 2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7124850f-b022-46e4-97b7-ba749a51b48b-model-cache\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:55:20.006417 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:20.006417 2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7124850f-b022-46e4-97b7-ba749a51b48b-home\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:55:20.006580 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:20.006434 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7124850f-b022-46e4-97b7-ba749a51b48b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7124850f-b022-46e4-97b7-ba749a51b48b" (UID: "7124850f-b022-46e4-97b7-ba749a51b48b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:55:20.007773 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:20.007750 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7124850f-b022-46e4-97b7-ba749a51b48b-dshm" (OuterVolumeSpecName: "dshm") pod "7124850f-b022-46e4-97b7-ba749a51b48b" (UID: "7124850f-b022-46e4-97b7-ba749a51b48b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:55:20.007853 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:20.007784 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7124850f-b022-46e4-97b7-ba749a51b48b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7124850f-b022-46e4-97b7-ba749a51b48b" (UID: "7124850f-b022-46e4-97b7-ba749a51b48b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 07:55:20.007853 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:20.007794 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7124850f-b022-46e4-97b7-ba749a51b48b-kube-api-access-r9gfx" (OuterVolumeSpecName: "kube-api-access-r9gfx") pod "7124850f-b022-46e4-97b7-ba749a51b48b" (UID: "7124850f-b022-46e4-97b7-ba749a51b48b"). InnerVolumeSpecName "kube-api-access-r9gfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:55:20.107563 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:20.107523 2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7124850f-b022-46e4-97b7-ba749a51b48b-tls-certs\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:55:20.107563 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:20.107557 2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7124850f-b022-46e4-97b7-ba749a51b48b-tmp-dir\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:55:20.107563 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:20.107566 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9gfx\" (UniqueName: \"kubernetes.io/projected/7124850f-b022-46e4-97b7-ba749a51b48b-kube-api-access-r9gfx\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:55:20.107802 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:20.107578 2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7124850f-b022-46e4-97b7-ba749a51b48b-dshm\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:55:20.222782 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:20.222685 2552 generic.go:356] "Generic (PLEG): container finished" podID="7124850f-b022-46e4-97b7-ba749a51b48b" containerID="1cb6d3a5c6b393dbf50d3e6087a294203f219725e24d069480dee14904420b4f" exitCode=0 Sep 16 07:55:20.222782 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:20.222734 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-3beaf8/noten-model-3beaf8-kserve-79d865d78c-ppbh8" Sep 16 07:55:20.222782 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:20.222758 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-3beaf8/noten-model-3beaf8-kserve-79d865d78c-ppbh8" event={"ID":"7124850f-b022-46e4-97b7-ba749a51b48b","Type":"ContainerDied","Data":"1cb6d3a5c6b393dbf50d3e6087a294203f219725e24d069480dee14904420b4f"} Sep 16 07:55:20.223076 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:20.222802 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-3beaf8/noten-model-3beaf8-kserve-79d865d78c-ppbh8" event={"ID":"7124850f-b022-46e4-97b7-ba749a51b48b","Type":"ContainerDied","Data":"1d3426d1bdbf4a2721843bf061b40215b67c2a8a808fa1c8aa8b5523ff09500e"} Sep 16 07:55:20.223076 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:20.222825 2552 scope.go:117] "RemoveContainer" containerID="1cb6d3a5c6b393dbf50d3e6087a294203f219725e24d069480dee14904420b4f" Sep 16 07:55:20.232340 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:20.232314 2552 scope.go:117] "RemoveContainer" containerID="1cb6d3a5c6b393dbf50d3e6087a294203f219725e24d069480dee14904420b4f" Sep 16 07:55:20.232679 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:55:20.232657 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cb6d3a5c6b393dbf50d3e6087a294203f219725e24d069480dee14904420b4f\": container with ID starting with 1cb6d3a5c6b393dbf50d3e6087a294203f219725e24d069480dee14904420b4f not found: ID does not exist" containerID="1cb6d3a5c6b393dbf50d3e6087a294203f219725e24d069480dee14904420b4f" Sep 16 07:55:20.232762 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:20.232693 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cb6d3a5c6b393dbf50d3e6087a294203f219725e24d069480dee14904420b4f"} err="failed to get container status \"1cb6d3a5c6b393dbf50d3e6087a294203f219725e24d069480dee14904420b4f\": rpc error: code = NotFound desc = could not find container \"1cb6d3a5c6b393dbf50d3e6087a294203f219725e24d069480dee14904420b4f\": container with ID starting with 1cb6d3a5c6b393dbf50d3e6087a294203f219725e24d069480dee14904420b4f not found: ID does not exist" Sep 16 07:55:20.243492 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:20.243457 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-3beaf8/noten-model-3beaf8-kserve-79d865d78c-ppbh8"] Sep 16 07:55:20.246681 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:20.246647 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-3beaf8/noten-model-3beaf8-kserve-79d865d78c-ppbh8"] Sep 16 07:55:20.937365 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:20.937333 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7124850f-b022-46e4-97b7-ba749a51b48b" path="/var/lib/kubelet/pods/7124850f-b022-46e4-97b7-ba749a51b48b/volumes" Sep 16 07:55:25.268919 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:25.268882 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-c9020d/distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn"] Sep 16 07:55:25.269379 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:25.269364 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7124850f-b022-46e4-97b7-ba749a51b48b" containerName="main" Sep 16 07:55:25.269379 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:25.269380 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="7124850f-b022-46e4-97b7-ba749a51b48b" containerName="main" Sep 16 07:55:25.269467 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:25.269393 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06c531c4-2b1c-47fd-afa3-6473fc38135f" containerName="payload-pre-processing" Sep 16 07:55:25.269467 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:25.269399 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="06c531c4-2b1c-47fd-afa3-6473fc38135f" containerName="payload-pre-processing" Sep 16 07:55:25.269467 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:25.269407 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="968b978d-ccf1-4116-876b-357bd1541402" containerName="payload-processing" Sep 16 07:55:25.269467 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:25.269413 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="968b978d-ccf1-4116-876b-357bd1541402" containerName="payload-processing" Sep 16 07:55:25.269467 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:25.269424 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbc18e51-8079-4c30-a3e0-98bd8ddf132d" containerName="istio-proxy" Sep 16 07:55:25.269467 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:25.269430 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbc18e51-8079-4c30-a3e0-98bd8ddf132d" containerName="istio-proxy" Sep 16 07:55:25.269680 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:25.269497 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="06c531c4-2b1c-47fd-afa3-6473fc38135f" containerName="payload-pre-processing" Sep 16 07:55:25.269680 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:25.269505 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbc18e51-8079-4c30-a3e0-98bd8ddf132d" containerName="istio-proxy" Sep 16 07:55:25.269680 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:25.269511 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="968b978d-ccf1-4116-876b-357bd1541402" containerName="payload-processing" Sep 16 07:55:25.269680 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:25.269516 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="7124850f-b022-46e4-97b7-ba749a51b48b" containerName="main" Sep 16 07:55:25.273751 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:25.273730 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c9020d/distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn" Sep 16 07:55:25.276162 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:25.276134 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c9020d\"/\"distinct-2-w0-c9020d-kserve-self-signed-certs\"" Sep 16 07:55:25.283738 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:25.283702 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c9020d/distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn"] Sep 16 07:55:25.297124 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:25.297063 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:55:25.359973 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:25.359928 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9e4ab9e-482a-459c-b596-f78f156d906c-home\") pod \"distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn\" (UID: \"e9e4ab9e-482a-459c-b596-f78f156d906c\") " pod="e2e-models-e2e-worker-w0-c9020d/distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn" Sep 16 07:55:25.360188 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:25.360008 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9e4ab9e-482a-459c-b596-f78f156d906c-dshm\") pod \"distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn\" (UID: \"e9e4ab9e-482a-459c-b596-f78f156d906c\") " pod="e2e-models-e2e-worker-w0-c9020d/distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn" Sep 16 07:55:25.360188 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:25.360049 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9e4ab9e-482a-459c-b596-f78f156d906c-model-cache\") pod \"distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn\" (UID: \"e9e4ab9e-482a-459c-b596-f78f156d906c\") " pod="e2e-models-e2e-worker-w0-c9020d/distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn" Sep 16 07:55:25.360188 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:25.360080 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9e4ab9e-482a-459c-b596-f78f156d906c-tmp-dir\") pod \"distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn\" (UID: \"e9e4ab9e-482a-459c-b596-f78f156d906c\") " pod="e2e-models-e2e-worker-w0-c9020d/distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn" Sep 16 07:55:25.360188 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:25.360129 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgmjv\" (UniqueName: \"kubernetes.io/projected/e9e4ab9e-482a-459c-b596-f78f156d906c-kube-api-access-jgmjv\") pod \"distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn\" (UID: \"e9e4ab9e-482a-459c-b596-f78f156d906c\") " pod="e2e-models-e2e-worker-w0-c9020d/distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn" Sep 16 07:55:25.360385 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:25.360181 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9e4ab9e-482a-459c-b596-f78f156d906c-tls-certs\") pod \"distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn\" (UID: \"e9e4ab9e-482a-459c-b596-f78f156d906c\") " pod="e2e-models-e2e-worker-w0-c9020d/distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn" Sep 16 07:55:25.460900 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:25.460855 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9e4ab9e-482a-459c-b596-f78f156d906c-tls-certs\") pod \"distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn\" (UID: \"e9e4ab9e-482a-459c-b596-f78f156d906c\") " pod="e2e-models-e2e-worker-w0-c9020d/distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn" Sep 16 07:55:25.461140 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:25.460924 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9e4ab9e-482a-459c-b596-f78f156d906c-home\") pod \"distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn\" (UID: \"e9e4ab9e-482a-459c-b596-f78f156d906c\") " pod="e2e-models-e2e-worker-w0-c9020d/distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn" Sep 16 07:55:25.461140 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:25.460978 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9e4ab9e-482a-459c-b596-f78f156d906c-dshm\") pod \"distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn\" (UID: \"e9e4ab9e-482a-459c-b596-f78f156d906c\") " pod="e2e-models-e2e-worker-w0-c9020d/distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn" Sep 16 07:55:25.461140 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:25.461017 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9e4ab9e-482a-459c-b596-f78f156d906c-model-cache\") pod \"distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn\" (UID: \"e9e4ab9e-482a-459c-b596-f78f156d906c\") " pod="e2e-models-e2e-worker-w0-c9020d/distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn" Sep 16 07:55:25.461140 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:25.461067 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9e4ab9e-482a-459c-b596-f78f156d906c-tmp-dir\") pod \"distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn\" (UID: \"e9e4ab9e-482a-459c-b596-f78f156d906c\") " pod="e2e-models-e2e-worker-w0-c9020d/distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn" Sep 16 07:55:25.461140 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:25.461128 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgmjv\" (UniqueName: \"kubernetes.io/projected/e9e4ab9e-482a-459c-b596-f78f156d906c-kube-api-access-jgmjv\") pod \"distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn\" (UID: \"e9e4ab9e-482a-459c-b596-f78f156d906c\") " pod="e2e-models-e2e-worker-w0-c9020d/distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn" Sep 16 07:55:25.461482 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:25.461451 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9e4ab9e-482a-459c-b596-f78f156d906c-home\") pod \"distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn\" (UID: \"e9e4ab9e-482a-459c-b596-f78f156d906c\") " pod="e2e-models-e2e-worker-w0-c9020d/distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn" Sep 16 07:55:25.461597 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:25.461501 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9e4ab9e-482a-459c-b596-f78f156d906c-tmp-dir\") pod \"distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn\" (UID: \"e9e4ab9e-482a-459c-b596-f78f156d906c\") " pod="e2e-models-e2e-worker-w0-c9020d/distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn" Sep 16 07:55:25.461597 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:25.461551 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9e4ab9e-482a-459c-b596-f78f156d906c-model-cache\") pod \"distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn\" (UID: \"e9e4ab9e-482a-459c-b596-f78f156d906c\") " pod="e2e-models-e2e-worker-w0-c9020d/distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn" Sep 16 07:55:25.463474 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:25.463445 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9e4ab9e-482a-459c-b596-f78f156d906c-dshm\") pod \"distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn\" (UID: \"e9e4ab9e-482a-459c-b596-f78f156d906c\") " pod="e2e-models-e2e-worker-w0-c9020d/distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn" Sep 16 07:55:25.463596 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:25.463542 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9e4ab9e-482a-459c-b596-f78f156d906c-tls-certs\") pod \"distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn\" (UID: \"e9e4ab9e-482a-459c-b596-f78f156d906c\") " pod="e2e-models-e2e-worker-w0-c9020d/distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn" Sep 16 07:55:25.468991 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:25.468966 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgmjv\" (UniqueName: \"kubernetes.io/projected/e9e4ab9e-482a-459c-b596-f78f156d906c-kube-api-access-jgmjv\") pod \"distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn\" (UID: \"e9e4ab9e-482a-459c-b596-f78f156d906c\") " pod="e2e-models-e2e-worker-w0-c9020d/distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn" Sep 16 07:55:25.585475 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:25.585391 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c9020d/distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn" Sep 16 07:55:25.859769 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:25.859737 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c9020d/distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn"] Sep 16 07:55:25.861486 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:55:25.861456 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9e4ab9e_482a_459c_b596_f78f156d906c.slice/crio-58fb6dae1a2357408dfee5fabc6d4d77c8c2bf0ff9d88414163754e1cd8602ab WatchSource:0}: Error finding container 58fb6dae1a2357408dfee5fabc6d4d77c8c2bf0ff9d88414163754e1cd8602ab: Status 404 returned error can't find the container with id 58fb6dae1a2357408dfee5fabc6d4d77c8c2bf0ff9d88414163754e1cd8602ab Sep 16 07:55:26.258971 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:26.258931 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c9020d/distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn" event={"ID":"e9e4ab9e-482a-459c-b596-f78f156d906c","Type":"ContainerStarted","Data":"2bf27c6358eaf2ca4bacc00c9f203c077b8dd7faf0b69627784c254036e78e2b"} Sep 16 07:55:26.258971 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:26.258976 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c9020d/distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn" event={"ID":"e9e4ab9e-482a-459c-b596-f78f156d906c","Type":"ContainerStarted","Data":"58fb6dae1a2357408dfee5fabc6d4d77c8c2bf0ff9d88414163754e1cd8602ab"} Sep 16 07:55:26.275950 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:26.275888 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-c9020d/distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn" podStartSLOduration=1.275867447 podStartE2EDuration="1.275867447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:55:26.274948009 +0000 UTC m=+1297.858726351" watchObservedRunningTime="2026-09-16 07:55:26.275867447 +0000 UTC m=+1297.859645785" Sep 16 07:55:27.864309 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:27.864271 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:55:32.198117 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:32.198066 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:55:33.688245 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:33.688208 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:55:33.916063 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:33.915998 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc"] Sep 16 07:55:33.916349 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:33.916320 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" podUID="d2c050a9-e9fb-48f1-abe1-2249f7276ab1" containerName="istio-proxy" containerID="cri-o://b6ba6b31d4fe704ae3d6365fa547258dea4ab1d4d2c53de07273ee49086f9752" gracePeriod=30 Sep 16 07:55:33.937617 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:33.937579 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:55:34.520842 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:34.520804 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:55:34.844676 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:34.844566 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:55:35.177534 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:35.177501 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:55:35.585870 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:35.585768 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-c9020d/distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn" Sep 16 07:55:35.585870 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:35.585829 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c9020d/distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn" Sep 16 07:55:35.599242 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:35.599208 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-c9020d/distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn" Sep 16 07:55:36.327949 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:36.327918 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c9020d/distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn" Sep 16 07:55:37.565618 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:37.565580 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:55:38.301629 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:38.301590 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:55:38.463417 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:38.463377 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:55:38.559852 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:38.559756 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:55:39.179952 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:39.179835 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:55:39.183669 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:39.183643 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" Sep 16 07:55:39.205137 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:39.205080 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-istio-envoy\") pod \"d2c050a9-e9fb-48f1-abe1-2249f7276ab1\" (UID: \"d2c050a9-e9fb-48f1-abe1-2249f7276ab1\") " Sep 16 07:55:39.205326 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:39.205165 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-istiod-ca-cert\") pod \"d2c050a9-e9fb-48f1-abe1-2249f7276ab1\" (UID: \"d2c050a9-e9fb-48f1-abe1-2249f7276ab1\") " Sep 16 07:55:39.205576 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:39.205546 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d2c050a9-e9fb-48f1-abe1-2249f7276ab1" (UID: "d2c050a9-e9fb-48f1-abe1-2249f7276ab1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:55:39.205642 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:39.205610 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-istio-data\") pod \"d2c050a9-e9fb-48f1-abe1-2249f7276ab1\" (UID: \"d2c050a9-e9fb-48f1-abe1-2249f7276ab1\") " Sep 16 07:55:39.205697 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:39.205648 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-credential-socket\") pod \"d2c050a9-e9fb-48f1-abe1-2249f7276ab1\" (UID: \"d2c050a9-e9fb-48f1-abe1-2249f7276ab1\") " Sep 16 07:55:39.205697 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:39.205668 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-workload-socket\") pod \"d2c050a9-e9fb-48f1-abe1-2249f7276ab1\" (UID: \"d2c050a9-e9fb-48f1-abe1-2249f7276ab1\") " Sep 16 07:55:39.205796 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:39.205720 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-istio-podinfo\") pod \"d2c050a9-e9fb-48f1-abe1-2249f7276ab1\" (UID: \"d2c050a9-e9fb-48f1-abe1-2249f7276ab1\") " Sep 16 07:55:39.205796 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:39.205762 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-workload-certs\") pod \"d2c050a9-e9fb-48f1-abe1-2249f7276ab1\" (UID: \"d2c050a9-e9fb-48f1-abe1-2249f7276ab1\") " Sep 16 07:55:39.205902 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:39.205806 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5blvn\" (UniqueName: \"kubernetes.io/projected/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-kube-api-access-5blvn\") pod \"d2c050a9-e9fb-48f1-abe1-2249f7276ab1\" (UID: \"d2c050a9-e9fb-48f1-abe1-2249f7276ab1\") " Sep 16 07:55:39.205953 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:39.205914 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-istio-data" (OuterVolumeSpecName: "istio-data") pod "d2c050a9-e9fb-48f1-abe1-2249f7276ab1" (UID: "d2c050a9-e9fb-48f1-abe1-2249f7276ab1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:55:39.205953 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:39.205922 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-istio-token\") pod \"d2c050a9-e9fb-48f1-abe1-2249f7276ab1\" (UID: \"d2c050a9-e9fb-48f1-abe1-2249f7276ab1\") " Sep 16 07:55:39.206056 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:39.205994 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d2c050a9-e9fb-48f1-abe1-2249f7276ab1" (UID: "d2c050a9-e9fb-48f1-abe1-2249f7276ab1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:55:39.206056 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:39.206008 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d2c050a9-e9fb-48f1-abe1-2249f7276ab1" (UID: "d2c050a9-e9fb-48f1-abe1-2249f7276ab1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:55:39.206187 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:39.206136 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d2c050a9-e9fb-48f1-abe1-2249f7276ab1" (UID: "d2c050a9-e9fb-48f1-abe1-2249f7276ab1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:55:39.206654 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:39.206630 2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-workload-certs\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:55:39.206885 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:39.206871 2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-istiod-ca-cert\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:55:39.207003 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:39.206991 2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-istio-data\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:55:39.207129 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:39.207087 2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-credential-socket\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:55:39.207239 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:39.207216 2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-workload-socket\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:55:39.208295 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:39.208268 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d2c050a9-e9fb-48f1-abe1-2249f7276ab1" (UID: "d2c050a9-e9fb-48f1-abe1-2249f7276ab1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 07:55:39.208449 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:39.208428 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-kube-api-access-5blvn" (OuterVolumeSpecName: "kube-api-access-5blvn") pod "d2c050a9-e9fb-48f1-abe1-2249f7276ab1" (UID: "d2c050a9-e9fb-48f1-abe1-2249f7276ab1"). InnerVolumeSpecName "kube-api-access-5blvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:55:39.209147 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:39.209121 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d2c050a9-e9fb-48f1-abe1-2249f7276ab1" (UID: "d2c050a9-e9fb-48f1-abe1-2249f7276ab1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 07:55:39.209641 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:39.209619 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-istio-token" (OuterVolumeSpecName: "istio-token") pod "d2c050a9-e9fb-48f1-abe1-2249f7276ab1" (UID: "d2c050a9-e9fb-48f1-abe1-2249f7276ab1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:55:39.308304 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:39.308264 2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-istio-token\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:55:39.308304 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:39.308297 2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-istio-envoy\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:55:39.308304 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:39.308307 2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-istio-podinfo\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:55:39.308555 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:39.308320 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5blvn\" (UniqueName: \"kubernetes.io/projected/d2c050a9-e9fb-48f1-abe1-2249f7276ab1-kube-api-access-5blvn\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:55:39.334535 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:39.334499 2552 generic.go:356] "Generic (PLEG): container finished" podID="d2c050a9-e9fb-48f1-abe1-2249f7276ab1" containerID="b6ba6b31d4fe704ae3d6365fa547258dea4ab1d4d2c53de07273ee49086f9752" exitCode=0 Sep 16 07:55:39.334704 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:39.334550 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" Sep 16 07:55:39.334704 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:39.334593 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" event={"ID":"d2c050a9-e9fb-48f1-abe1-2249f7276ab1","Type":"ContainerDied","Data":"b6ba6b31d4fe704ae3d6365fa547258dea4ab1d4d2c53de07273ee49086f9752"} Sep 16 07:55:39.334704 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:39.334652 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc" event={"ID":"d2c050a9-e9fb-48f1-abe1-2249f7276ab1","Type":"ContainerDied","Data":"d8e744db4abfb879ee848717edfc90ec222af2e25532156472542f899f036434"} Sep 16 07:55:39.334704 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:39.334681 2552 scope.go:117] "RemoveContainer" containerID="b6ba6b31d4fe704ae3d6365fa547258dea4ab1d4d2c53de07273ee49086f9752" Sep 16 07:55:39.344784 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:39.344761 2552 scope.go:117] "RemoveContainer" containerID="fd6773887969b9482d50d364da2251e6eafb40c7af3a57d4ef0b3e5ba544bda9" Sep 16 07:55:39.357267 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:39.357244 2552 scope.go:117] "RemoveContainer" containerID="b6ba6b31d4fe704ae3d6365fa547258dea4ab1d4d2c53de07273ee49086f9752" Sep 16 07:55:39.357523 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:39.357503 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc"] Sep 16 07:55:39.357642 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:55:39.357623 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6ba6b31d4fe704ae3d6365fa547258dea4ab1d4d2c53de07273ee49086f9752\": container with ID starting with b6ba6b31d4fe704ae3d6365fa547258dea4ab1d4d2c53de07273ee49086f9752 not found: ID does not exist" containerID="b6ba6b31d4fe704ae3d6365fa547258dea4ab1d4d2c53de07273ee49086f9752" Sep 16 07:55:39.357692 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:39.357650 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6ba6b31d4fe704ae3d6365fa547258dea4ab1d4d2c53de07273ee49086f9752"} err="failed to get container status \"b6ba6b31d4fe704ae3d6365fa547258dea4ab1d4d2c53de07273ee49086f9752\": rpc error: code = NotFound desc = could not find container \"b6ba6b31d4fe704ae3d6365fa547258dea4ab1d4d2c53de07273ee49086f9752\": container with ID starting with b6ba6b31d4fe704ae3d6365fa547258dea4ab1d4d2c53de07273ee49086f9752 not found: ID does not exist" Sep 16 07:55:39.357692 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:39.357668 2552 scope.go:117] "RemoveContainer" containerID="fd6773887969b9482d50d364da2251e6eafb40c7af3a57d4ef0b3e5ba544bda9" Sep 16 07:55:39.357911 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:55:39.357897 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd6773887969b9482d50d364da2251e6eafb40c7af3a57d4ef0b3e5ba544bda9\": container with ID starting with fd6773887969b9482d50d364da2251e6eafb40c7af3a57d4ef0b3e5ba544bda9 not found: ID does not exist" containerID="fd6773887969b9482d50d364da2251e6eafb40c7af3a57d4ef0b3e5ba544bda9" Sep 16 07:55:39.357956 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:39.357914 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd6773887969b9482d50d364da2251e6eafb40c7af3a57d4ef0b3e5ba544bda9"} err="failed to get container status \"fd6773887969b9482d50d364da2251e6eafb40c7af3a57d4ef0b3e5ba544bda9\": rpc error: code = NotFound desc = could not find container \"fd6773887969b9482d50d364da2251e6eafb40c7af3a57d4ef0b3e5ba544bda9\": container with ID starting with fd6773887969b9482d50d364da2251e6eafb40c7af3a57d4ef0b3e5ba544bda9 not found: ID does not exist" Sep 16 07:55:39.363120 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:39.363073 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-dd359c-openshift-default-56d8c5d78d-jrhdc"] Sep 16 07:55:39.910444 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:39.910407 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:55:40.574926 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:40.574876 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:55:40.937827 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:40.937792 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2c050a9-e9fb-48f1-abe1-2249f7276ab1" path="/var/lib/kubelet/pods/d2c050a9-e9fb-48f1-abe1-2249f7276ab1/volumes" Sep 16 07:55:41.262283 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:41.262187 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:55:41.790832 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:41.790801 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:55:42.069082 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:42.068993 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:55:42.127597 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:42.127560 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:55:42.230830 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:42.230791 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:55:42.599892 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:42.599850 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:55:42.747552 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:42.747517 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:55:42.752893 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:42.752862 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3616cc-55fcf67858-6qb2b"] Sep 16 07:55:42.753323 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:42.753308 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2c050a9-e9fb-48f1-abe1-2249f7276ab1" containerName="istio-proxy" Sep 16 07:55:42.753323 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:42.753324 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2c050a9-e9fb-48f1-abe1-2249f7276ab1" containerName="istio-proxy" Sep 16 07:55:42.753431 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:42.753339 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2c050a9-e9fb-48f1-abe1-2249f7276ab1" containerName="istio-proxy" Sep 16 07:55:42.753431 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:42.753345 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2c050a9-e9fb-48f1-abe1-2249f7276ab1" containerName="istio-proxy" Sep 16 07:55:42.753431 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:42.753425 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2c050a9-e9fb-48f1-abe1-2249f7276ab1" containerName="istio-proxy" Sep 16 07:55:42.753530 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:42.753435 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2c050a9-e9fb-48f1-abe1-2249f7276ab1" containerName="istio-proxy" Sep 16 07:55:42.758267 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:42.758247 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3616cc-55fcf67858-6qb2b" Sep 16 07:55:42.761695 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:42.761672 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-3616cc\"" Sep 16 07:55:42.775877 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:42.775847 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3616cc-55fcf67858-6qb2b"] Sep 16 07:55:42.845654 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:42.845607 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04195d61-0c03-4757-9612-d9e58e5d19e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-3616cc-55fcf67858-6qb2b\" (UID: \"04195d61-0c03-4757-9612-d9e58e5d19e3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3616cc-55fcf67858-6qb2b" Sep 16 07:55:42.845654 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:42.845652 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmwws\" (UniqueName: \"kubernetes.io/projected/04195d61-0c03-4757-9612-d9e58e5d19e3-kube-api-access-fmwws\") pod \"payload-pre-processing-e2e-shared-a-w3-3616cc-55fcf67858-6qb2b\" (UID: \"04195d61-0c03-4757-9612-d9e58e5d19e3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3616cc-55fcf67858-6qb2b" Sep 16 07:55:42.867021 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:42.866924 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-3616cc-6c5d7cd96c-8ht5l"] Sep 16 07:55:42.870807 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:42.870779 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3616cc-6c5d7cd96c-8ht5l" Sep 16 07:55:42.873290 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:42.873263 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-3616cc-dockercfg-88wfv\"" Sep 16 07:55:42.879730 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:42.879682 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-3616cc-6c5d7cd96c-8ht5l"] Sep 16 07:55:42.947232 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:42.947190 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a81a2c2-a21d-47d4-bfb7-0454e8e1f12b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-3616cc-6c5d7cd96c-8ht5l\" (UID: \"4a81a2c2-a21d-47d4-bfb7-0454e8e1f12b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3616cc-6c5d7cd96c-8ht5l" Sep 16 07:55:42.947431 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:42.947261 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04195d61-0c03-4757-9612-d9e58e5d19e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-3616cc-55fcf67858-6qb2b\" (UID: \"04195d61-0c03-4757-9612-d9e58e5d19e3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3616cc-55fcf67858-6qb2b" Sep 16 07:55:42.947431 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:42.947326 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmwws\" (UniqueName: \"kubernetes.io/projected/04195d61-0c03-4757-9612-d9e58e5d19e3-kube-api-access-fmwws\") pod \"payload-pre-processing-e2e-shared-a-w3-3616cc-55fcf67858-6qb2b\" (UID: \"04195d61-0c03-4757-9612-d9e58e5d19e3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3616cc-55fcf67858-6qb2b" Sep 16 07:55:42.947431 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:42.947370 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dg92\" (UniqueName: \"kubernetes.io/projected/4a81a2c2-a21d-47d4-bfb7-0454e8e1f12b-kube-api-access-5dg92\") pod \"payload-processing-e2e-shared-a-w3-3616cc-6c5d7cd96c-8ht5l\" (UID: \"4a81a2c2-a21d-47d4-bfb7-0454e8e1f12b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3616cc-6c5d7cd96c-8ht5l" Sep 16 07:55:42.947986 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:42.947964 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04195d61-0c03-4757-9612-d9e58e5d19e3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-3616cc-55fcf67858-6qb2b\" (UID: \"04195d61-0c03-4757-9612-d9e58e5d19e3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3616cc-55fcf67858-6qb2b" Sep 16 07:55:42.955315 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:42.955284 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmwws\" (UniqueName: \"kubernetes.io/projected/04195d61-0c03-4757-9612-d9e58e5d19e3-kube-api-access-fmwws\") pod \"payload-pre-processing-e2e-shared-a-w3-3616cc-55fcf67858-6qb2b\" (UID: \"04195d61-0c03-4757-9612-d9e58e5d19e3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3616cc-55fcf67858-6qb2b" Sep 16 07:55:43.048604 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:43.048554 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dg92\" (UniqueName: \"kubernetes.io/projected/4a81a2c2-a21d-47d4-bfb7-0454e8e1f12b-kube-api-access-5dg92\") pod \"payload-processing-e2e-shared-a-w3-3616cc-6c5d7cd96c-8ht5l\" (UID: \"4a81a2c2-a21d-47d4-bfb7-0454e8e1f12b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3616cc-6c5d7cd96c-8ht5l" Sep 16 07:55:43.048819 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:43.048796 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a81a2c2-a21d-47d4-bfb7-0454e8e1f12b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-3616cc-6c5d7cd96c-8ht5l\" (UID: \"4a81a2c2-a21d-47d4-bfb7-0454e8e1f12b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3616cc-6c5d7cd96c-8ht5l" Sep 16 07:55:43.049609 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:43.049588 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a81a2c2-a21d-47d4-bfb7-0454e8e1f12b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-3616cc-6c5d7cd96c-8ht5l\" (UID: \"4a81a2c2-a21d-47d4-bfb7-0454e8e1f12b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3616cc-6c5d7cd96c-8ht5l" Sep 16 07:55:43.056739 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:43.056709 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dg92\" (UniqueName: \"kubernetes.io/projected/4a81a2c2-a21d-47d4-bfb7-0454e8e1f12b-kube-api-access-5dg92\") pod \"payload-processing-e2e-shared-a-w3-3616cc-6c5d7cd96c-8ht5l\" (UID: \"4a81a2c2-a21d-47d4-bfb7-0454e8e1f12b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3616cc-6c5d7cd96c-8ht5l" Sep 16 07:55:43.069582 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:43.069541 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3616cc-55fcf67858-6qb2b" Sep 16 07:55:43.183870 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:43.183816 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3616cc-6c5d7cd96c-8ht5l" Sep 16 07:55:43.345318 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:43.345287 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3616cc-55fcf67858-6qb2b"] Sep 16 07:55:43.347282 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:55:43.347252 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04195d61_0c03_4757_9612_d9e58e5d19e3.slice/crio-5f2b9ad3f1fbe562050fbf3e9ba1e1b52502cbaa254d4653eea3d4261161a68c WatchSource:0}: Error finding container 5f2b9ad3f1fbe562050fbf3e9ba1e1b52502cbaa254d4653eea3d4261161a68c: Status 404 returned error can't find the container with id 5f2b9ad3f1fbe562050fbf3e9ba1e1b52502cbaa254d4653eea3d4261161a68c Sep 16 07:55:43.358597 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:43.358566 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3616cc-55fcf67858-6qb2b" event={"ID":"04195d61-0c03-4757-9612-d9e58e5d19e3","Type":"ContainerStarted","Data":"5f2b9ad3f1fbe562050fbf3e9ba1e1b52502cbaa254d4653eea3d4261161a68c"} Sep 16 07:55:43.461789 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:43.461754 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-3616cc-6c5d7cd96c-8ht5l"] Sep 16 07:55:43.463952 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:55:43.463915 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a81a2c2_a21d_47d4_bfb7_0454e8e1f12b.slice/crio-42993ab1678dee94d23330f686ac79d3c08ff0c25b891998ecb30927ae957538 WatchSource:0}: Error finding container 42993ab1678dee94d23330f686ac79d3c08ff0c25b891998ecb30927ae957538: Status 404 returned error can't find the container with id 42993ab1678dee94d23330f686ac79d3c08ff0c25b891998ecb30927ae957538 Sep 16 07:55:44.367621 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:44.367584 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3616cc-6c5d7cd96c-8ht5l" event={"ID":"4a81a2c2-a21d-47d4-bfb7-0454e8e1f12b","Type":"ContainerStarted","Data":"5c0c065a017d9c18ccecc5fb6ae680251ee894ab14f95a229d0b6766d68810f0"} Sep 16 07:55:44.368090 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:44.367657 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3616cc-6c5d7cd96c-8ht5l" Sep 16 07:55:44.368090 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:44.367682 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3616cc-6c5d7cd96c-8ht5l" event={"ID":"4a81a2c2-a21d-47d4-bfb7-0454e8e1f12b","Type":"ContainerStarted","Data":"42993ab1678dee94d23330f686ac79d3c08ff0c25b891998ecb30927ae957538"} Sep 16 07:55:44.369514 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:44.369481 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3616cc-55fcf67858-6qb2b" event={"ID":"04195d61-0c03-4757-9612-d9e58e5d19e3","Type":"ContainerStarted","Data":"20df1296bdba9b56d42450c80b71be0a94176d9306438ec4568d819ce7a881e1"} Sep 16 07:55:44.369662 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:44.369610 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3616cc-55fcf67858-6qb2b" Sep 16 07:55:44.384629 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:44.384568 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3616cc-6c5d7cd96c-8ht5l" podStartSLOduration=2.38454782 podStartE2EDuration="2.38454782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:55:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:55:44.382692625 +0000 UTC m=+1315.966470979" watchObservedRunningTime="2026-09-16 07:55:44.38454782 +0000 UTC m=+1315.968326158" Sep 16 07:55:44.398861 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:44.398797 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3616cc-55fcf67858-6qb2b" podStartSLOduration=2.398778342 podStartE2EDuration="2.398778342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:55:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:55:44.397663425 +0000 UTC m=+1315.981441785" watchObservedRunningTime="2026-09-16 07:55:44.398778342 +0000 UTC m=+1315.982556678" Sep 16 07:55:48.996027 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:48.995983 2552 scope.go:117] "RemoveContainer" containerID="4670774efd0738f98f9517a32b71275f3e79af375210f6833f223dd68008a8ac" Sep 16 07:55:55.378265 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:55.378232 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3616cc-6c5d7cd96c-8ht5l" Sep 16 07:55:55.378795 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:55:55.378307 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3616cc-55fcf67858-6qb2b" Sep 16 07:56:01.008894 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:01.008852 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fc2a294d-lwc7g"] Sep 16 07:56:01.013917 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:01.013894 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fc2a294d-lwc7g" Sep 16 07:56:01.017094 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:01.017065 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zwk5r\"" Sep 16 07:56:01.017369 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:01.017109 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 07:56:01.019165 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:01.019135 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fc2a294d-lwc7g"] Sep 16 07:56:01.141302 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:01.141262 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f195276-8224-4df1-8b70-e98f1b33a49b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fc2a294d-lwc7g\" (UID: \"8f195276-8224-4df1-8b70-e98f1b33a49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fc2a294d-lwc7g" Sep 16 07:56:01.231456 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:01.231418 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:01.242656 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:01.242619 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f195276-8224-4df1-8b70-e98f1b33a49b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fc2a294d-lwc7g\" (UID: \"8f195276-8224-4df1-8b70-e98f1b33a49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fc2a294d-lwc7g" Sep 16 07:56:01.243311 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:01.243290 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f195276-8224-4df1-8b70-e98f1b33a49b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-fc2a294d-lwc7g\" (UID: \"8f195276-8224-4df1-8b70-e98f1b33a49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fc2a294d-lwc7g" Sep 16 07:56:01.327023 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:01.326912 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fc2a294d-lwc7g" Sep 16 07:56:01.600738 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:01.600710 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fc2a294d-lwc7g"] Sep 16 07:56:01.602977 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:56:01.602942 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f195276_8224_4df1_8b70_e98f1b33a49b.slice/crio-63ecd0dd1b64eb718f82757c2fcaf182940b17927cd20f9b78f0ecd351abe38e WatchSource:0}: Error finding container 63ecd0dd1b64eb718f82757c2fcaf182940b17927cd20f9b78f0ecd351abe38e: Status 404 returned error can't find the container with id 63ecd0dd1b64eb718f82757c2fcaf182940b17927cd20f9b78f0ecd351abe38e Sep 16 07:56:01.948045 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:01.947994 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:02.474024 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:02.473987 2552 generic.go:356] "Generic (PLEG): container finished" podID="8f195276-8224-4df1-8b70-e98f1b33a49b" containerID="1cbf22575a6ff24a053a82717554103551ac11aad6a09ca818e0968d0f855e9e" exitCode=0 Sep 16 07:56:02.474437 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:02.474078 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fc2a294d-lwc7g" event={"ID":"8f195276-8224-4df1-8b70-e98f1b33a49b","Type":"ContainerDied","Data":"1cbf22575a6ff24a053a82717554103551ac11aad6a09ca818e0968d0f855e9e"} Sep 16 07:56:02.474437 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:02.474184 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fc2a294d-lwc7g" event={"ID":"8f195276-8224-4df1-8b70-e98f1b33a49b","Type":"ContainerStarted","Data":"63ecd0dd1b64eb718f82757c2fcaf182940b17927cd20f9b78f0ecd351abe38e"} Sep 16 07:56:02.770067 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:02.770024 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:03.403704 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:03.403667 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:03.616342 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:03.616309 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fc2a294d-lwc7g" Sep 16 07:56:03.768980 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:03.768937 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f195276-8224-4df1-8b70-e98f1b33a49b-maas-api-service-ca\") pod \"8f195276-8224-4df1-8b70-e98f1b33a49b\" (UID: \"8f195276-8224-4df1-8b70-e98f1b33a49b\") " Sep 16 07:56:03.769335 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:03.769310 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f195276-8224-4df1-8b70-e98f1b33a49b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8f195276-8224-4df1-8b70-e98f1b33a49b" (UID: "8f195276-8224-4df1-8b70-e98f1b33a49b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:56:03.870464 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:03.870424 2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f195276-8224-4df1-8b70-e98f1b33a49b-maas-api-service-ca\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:56:04.192365 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:04.192316 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:04.485983 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:04.485896 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fc2a294d-lwc7g" Sep 16 07:56:04.485983 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:04.485909 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fc2a294d-lwc7g" event={"ID":"8f195276-8224-4df1-8b70-e98f1b33a49b","Type":"ContainerDied","Data":"63ecd0dd1b64eb718f82757c2fcaf182940b17927cd20f9b78f0ecd351abe38e"} Sep 16 07:56:04.485983 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:04.485951 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63ecd0dd1b64eb718f82757c2fcaf182940b17927cd20f9b78f0ecd351abe38e" Sep 16 07:56:04.971924 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:04.971882 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:05.690963 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:05.690931 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:06.469365 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:06.469322 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:07.344272 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:07.344233 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:07.986786 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:07.986749 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:08.494426 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:08.494385 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:08.657279 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:08.657240 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:08.854549 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:08.854444 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:09.161494 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:09.161400 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz"] Sep 16 07:56:09.161840 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:09.161829 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f195276-8224-4df1-8b70-e98f1b33a49b" containerName="revoke-keys" Sep 16 07:56:09.161880 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:09.161841 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f195276-8224-4df1-8b70-e98f1b33a49b" containerName="revoke-keys" Sep 16 07:56:09.161922 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:09.161911 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f195276-8224-4df1-8b70-e98f1b33a49b" containerName="revoke-keys" Sep 16 07:56:09.165160 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:09.165138 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" Sep 16 07:56:09.167643 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:09.167620 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 16 07:56:09.167856 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:09.167841 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-e9fc8c-openshift-default-dockercfg-jcd9q\"" Sep 16 07:56:09.176777 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:09.176748 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz"] Sep 16 07:56:09.189729 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:09.189693 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:09.324207 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:09.324077 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/859b21e8-3a32-49ae-be7c-a7249d2e46a7-istio-envoy\") pod \"e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz\" (UID: \"859b21e8-3a32-49ae-be7c-a7249d2e46a7\") " pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" Sep 16 07:56:09.324207 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:09.324170 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/859b21e8-3a32-49ae-be7c-a7249d2e46a7-workload-certs\") pod \"e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz\" (UID: \"859b21e8-3a32-49ae-be7c-a7249d2e46a7\") " pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" Sep 16 07:56:09.324466 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:09.324231 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/859b21e8-3a32-49ae-be7c-a7249d2e46a7-istio-podinfo\") pod \"e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz\" (UID: \"859b21e8-3a32-49ae-be7c-a7249d2e46a7\") " pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" Sep 16 07:56:09.324466 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:09.324262 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn4ck\" (UniqueName: \"kubernetes.io/projected/859b21e8-3a32-49ae-be7c-a7249d2e46a7-kube-api-access-rn4ck\") pod \"e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz\" (UID: \"859b21e8-3a32-49ae-be7c-a7249d2e46a7\") " pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" Sep 16 07:56:09.324466 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:09.324290 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/859b21e8-3a32-49ae-be7c-a7249d2e46a7-istiod-ca-cert\") pod \"e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz\" (UID: \"859b21e8-3a32-49ae-be7c-a7249d2e46a7\") " pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" Sep 16 07:56:09.324466 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:09.324359 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/859b21e8-3a32-49ae-be7c-a7249d2e46a7-istio-token\") pod \"e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz\" (UID: \"859b21e8-3a32-49ae-be7c-a7249d2e46a7\") " pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" Sep 16 07:56:09.324466 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:09.324418 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/859b21e8-3a32-49ae-be7c-a7249d2e46a7-credential-socket\") pod \"e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz\" (UID: \"859b21e8-3a32-49ae-be7c-a7249d2e46a7\") " pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" Sep 16 07:56:09.324697 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:09.324488 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/859b21e8-3a32-49ae-be7c-a7249d2e46a7-istio-data\") pod \"e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz\" (UID: \"859b21e8-3a32-49ae-be7c-a7249d2e46a7\") " pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" Sep 16 07:56:09.324697 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:09.324557 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/859b21e8-3a32-49ae-be7c-a7249d2e46a7-workload-socket\") pod \"e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz\" (UID: \"859b21e8-3a32-49ae-be7c-a7249d2e46a7\") " pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" Sep 16 07:56:09.381111 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:09.381061 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:09.425944 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:09.425904 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/859b21e8-3a32-49ae-be7c-a7249d2e46a7-workload-socket\") pod \"e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz\" (UID: \"859b21e8-3a32-49ae-be7c-a7249d2e46a7\") " pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" Sep 16 07:56:09.426166 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:09.425960 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/859b21e8-3a32-49ae-be7c-a7249d2e46a7-istio-envoy\") pod \"e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz\" (UID: \"859b21e8-3a32-49ae-be7c-a7249d2e46a7\") " pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" Sep 16 07:56:09.426166 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:09.425988 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/859b21e8-3a32-49ae-be7c-a7249d2e46a7-workload-certs\") pod \"e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz\" (UID: \"859b21e8-3a32-49ae-be7c-a7249d2e46a7\") " pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" Sep 16 07:56:09.426166 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:09.426024 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/859b21e8-3a32-49ae-be7c-a7249d2e46a7-istio-podinfo\") pod \"e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz\" (UID: \"859b21e8-3a32-49ae-be7c-a7249d2e46a7\") " pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" Sep 16 07:56:09.426166 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:09.426045 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn4ck\" (UniqueName: \"kubernetes.io/projected/859b21e8-3a32-49ae-be7c-a7249d2e46a7-kube-api-access-rn4ck\") pod \"e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz\" (UID: \"859b21e8-3a32-49ae-be7c-a7249d2e46a7\") " pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" Sep 16 07:56:09.426166 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:09.426065 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/859b21e8-3a32-49ae-be7c-a7249d2e46a7-istiod-ca-cert\") pod \"e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz\" (UID: \"859b21e8-3a32-49ae-be7c-a7249d2e46a7\") " pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" Sep 16 07:56:09.426166 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:09.426131 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/859b21e8-3a32-49ae-be7c-a7249d2e46a7-istio-token\") pod \"e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz\" (UID: \"859b21e8-3a32-49ae-be7c-a7249d2e46a7\") " pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" Sep 16 07:56:09.426491 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:09.426184 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/859b21e8-3a32-49ae-be7c-a7249d2e46a7-credential-socket\") pod \"e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz\" (UID: \"859b21e8-3a32-49ae-be7c-a7249d2e46a7\") " pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" Sep 16 07:56:09.426491 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:09.426256 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/859b21e8-3a32-49ae-be7c-a7249d2e46a7-istio-data\") pod \"e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz\" (UID: \"859b21e8-3a32-49ae-be7c-a7249d2e46a7\") " pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" Sep 16 07:56:09.426491 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:09.426409 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/859b21e8-3a32-49ae-be7c-a7249d2e46a7-workload-socket\") pod \"e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz\" (UID: \"859b21e8-3a32-49ae-be7c-a7249d2e46a7\") " pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" Sep 16 07:56:09.426655 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:09.426518 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/859b21e8-3a32-49ae-be7c-a7249d2e46a7-workload-certs\") pod \"e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz\" (UID: \"859b21e8-3a32-49ae-be7c-a7249d2e46a7\") " pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" Sep 16 07:56:09.426655 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:09.426586 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/859b21e8-3a32-49ae-be7c-a7249d2e46a7-istio-data\") pod \"e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz\" (UID: \"859b21e8-3a32-49ae-be7c-a7249d2e46a7\") " pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" Sep 16 07:56:09.426757 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:09.426668 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/859b21e8-3a32-49ae-be7c-a7249d2e46a7-credential-socket\") pod \"e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz\" (UID: \"859b21e8-3a32-49ae-be7c-a7249d2e46a7\") " pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" Sep 16 07:56:09.427010 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:09.426981 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/859b21e8-3a32-49ae-be7c-a7249d2e46a7-istiod-ca-cert\") pod \"e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz\" (UID: \"859b21e8-3a32-49ae-be7c-a7249d2e46a7\") " pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" Sep 16 07:56:09.428442 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:09.428418 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/859b21e8-3a32-49ae-be7c-a7249d2e46a7-istio-envoy\") pod \"e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz\" (UID: \"859b21e8-3a32-49ae-be7c-a7249d2e46a7\") " pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" Sep 16 07:56:09.428669 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:09.428653 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/859b21e8-3a32-49ae-be7c-a7249d2e46a7-istio-podinfo\") pod \"e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz\" (UID: \"859b21e8-3a32-49ae-be7c-a7249d2e46a7\") " pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" Sep 16 07:56:09.436952 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:09.436922 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/859b21e8-3a32-49ae-be7c-a7249d2e46a7-istio-token\") pod \"e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz\" (UID: \"859b21e8-3a32-49ae-be7c-a7249d2e46a7\") " pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" Sep 16 07:56:09.437309 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:09.437283 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn4ck\" (UniqueName: \"kubernetes.io/projected/859b21e8-3a32-49ae-be7c-a7249d2e46a7-kube-api-access-rn4ck\") pod \"e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz\" (UID: \"859b21e8-3a32-49ae-be7c-a7249d2e46a7\") " pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" Sep 16 07:56:09.476846 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:09.476799 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" Sep 16 07:56:09.592224 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:09.592180 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:09.757916 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:09.757887 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz"] Sep 16 07:56:09.760976 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:56:09.760944 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod859b21e8_3a32_49ae_be7c_a7249d2e46a7.slice/crio-6108f8e2e9f853ccfdd52ec29a1ed34eecc0e48e6667d7413ce808f46fdf22fe WatchSource:0}: Error finding container 6108f8e2e9f853ccfdd52ec29a1ed34eecc0e48e6667d7413ce808f46fdf22fe: Status 404 returned error can't find the container with id 6108f8e2e9f853ccfdd52ec29a1ed34eecc0e48e6667d7413ce808f46fdf22fe Sep 16 07:56:09.763781 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:09.763744 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 16 07:56:09.763886 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:09.763812 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 16 07:56:09.763886 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:09.763843 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 16 07:56:10.032799 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:10.032704 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:10.523034 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:10.522996 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" event={"ID":"859b21e8-3a32-49ae-be7c-a7249d2e46a7","Type":"ContainerStarted","Data":"a08dec342f9f7fe0b0bb901e58807ed0c1624722dbab28cd69856222a56dede8"} Sep 16 07:56:10.523448 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:10.523059 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" event={"ID":"859b21e8-3a32-49ae-be7c-a7249d2e46a7","Type":"ContainerStarted","Data":"6108f8e2e9f853ccfdd52ec29a1ed34eecc0e48e6667d7413ce808f46fdf22fe"} Sep 16 07:56:10.546321 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:10.546249 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" podStartSLOduration=1.5462258869999999 podStartE2EDuration="1.546225887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:56:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:56:10.543681781 +0000 UTC m=+1342.127460119" watchObservedRunningTime="2026-09-16 07:56:10.546225887 +0000 UTC m=+1342.130004241" Sep 16 07:56:10.780302 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:10.780207 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:11.477558 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:11.477512 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" Sep 16 07:56:11.483283 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:11.483248 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" Sep 16 07:56:11.529694 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:11.529664 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" Sep 16 07:56:11.531043 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:11.531015 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" Sep 16 07:56:11.935191 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:11.935156 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:12.465261 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:12.465227 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:12.569586 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:12.569545 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:12.672551 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:12.672512 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:13.565972 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:13.565934 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:14.169299 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:14.169259 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:14.201830 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:14.201792 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fc2a294d-7545bf669d-fsczl"] Sep 16 07:56:14.202089 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:14.202062 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-fc2a294d-7545bf669d-fsczl" podUID="665010a2-d972-420d-bf96-fa90f4da5074" containerName="payload-pre-processing" containerID="cri-o://d5c478ec4a36f71f055c067dbaf77cf9ea5b4fed3d79ecc64385b2c86b8a7f78" gracePeriod=30 Sep 16 07:56:14.269081 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:14.269043 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:14.369681 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:14.369639 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:14.406570 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:14.406533 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fc2a294d-lwc7g"] Sep 16 07:56:14.411012 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:14.410984 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fc2a294d-lwc7g"] Sep 16 07:56:14.428422 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:14.428320 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:14.461046 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:14.461015 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fc2a294d-7545bf669d-fsczl" Sep 16 07:56:14.480149 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:14.480085 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjt4v\" (UniqueName: \"kubernetes.io/projected/665010a2-d972-420d-bf96-fa90f4da5074-kube-api-access-hjt4v\") pod \"665010a2-d972-420d-bf96-fa90f4da5074\" (UID: \"665010a2-d972-420d-bf96-fa90f4da5074\") " Sep 16 07:56:14.480337 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:14.480239 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/665010a2-d972-420d-bf96-fa90f4da5074-plugins-config-volume\") pod \"665010a2-d972-420d-bf96-fa90f4da5074\" (UID: \"665010a2-d972-420d-bf96-fa90f4da5074\") " Sep 16 07:56:14.480619 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:14.480596 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/665010a2-d972-420d-bf96-fa90f4da5074-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "665010a2-d972-420d-bf96-fa90f4da5074" (UID: "665010a2-d972-420d-bf96-fa90f4da5074"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:56:14.482354 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:14.482324 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/665010a2-d972-420d-bf96-fa90f4da5074-kube-api-access-hjt4v" (OuterVolumeSpecName: "kube-api-access-hjt4v") pod "665010a2-d972-420d-bf96-fa90f4da5074" (UID: "665010a2-d972-420d-bf96-fa90f4da5074"). InnerVolumeSpecName "kube-api-access-hjt4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:56:14.491607 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:14.491573 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:14.556550 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:14.556513 2552 generic.go:356] "Generic (PLEG): container finished" podID="665010a2-d972-420d-bf96-fa90f4da5074" containerID="d5c478ec4a36f71f055c067dbaf77cf9ea5b4fed3d79ecc64385b2c86b8a7f78" exitCode=0 Sep 16 07:56:14.556750 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:14.556561 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-fc2a294d-7545bf669d-fsczl" Sep 16 07:56:14.556750 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:14.556610 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fc2a294d-7545bf669d-fsczl" event={"ID":"665010a2-d972-420d-bf96-fa90f4da5074","Type":"ContainerDied","Data":"d5c478ec4a36f71f055c067dbaf77cf9ea5b4fed3d79ecc64385b2c86b8a7f78"} Sep 16 07:56:14.556750 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:14.556672 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-fc2a294d-7545bf669d-fsczl" event={"ID":"665010a2-d972-420d-bf96-fa90f4da5074","Type":"ContainerDied","Data":"7b708b8c87805c1f1caadfebed2fcb20501a27f56586bf272a13b6eb024f9cdb"} Sep 16 07:56:14.556750 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:14.556702 2552 scope.go:117] "RemoveContainer" containerID="d5c478ec4a36f71f055c067dbaf77cf9ea5b4fed3d79ecc64385b2c86b8a7f78" Sep 16 07:56:14.566816 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:14.566796 2552 scope.go:117] "RemoveContainer" containerID="d5c478ec4a36f71f055c067dbaf77cf9ea5b4fed3d79ecc64385b2c86b8a7f78" Sep 16 07:56:14.567174 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:56:14.567153 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5c478ec4a36f71f055c067dbaf77cf9ea5b4fed3d79ecc64385b2c86b8a7f78\": container with ID starting with d5c478ec4a36f71f055c067dbaf77cf9ea5b4fed3d79ecc64385b2c86b8a7f78 not found: ID does not exist" containerID="d5c478ec4a36f71f055c067dbaf77cf9ea5b4fed3d79ecc64385b2c86b8a7f78" Sep 16 07:56:14.567253 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:14.567188 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5c478ec4a36f71f055c067dbaf77cf9ea5b4fed3d79ecc64385b2c86b8a7f78"} err="failed to get container status \"d5c478ec4a36f71f055c067dbaf77cf9ea5b4fed3d79ecc64385b2c86b8a7f78\": rpc error: code = NotFound desc = could not find container \"d5c478ec4a36f71f055c067dbaf77cf9ea5b4fed3d79ecc64385b2c86b8a7f78\": container with ID starting with d5c478ec4a36f71f055c067dbaf77cf9ea5b4fed3d79ecc64385b2c86b8a7f78 not found: ID does not exist" Sep 16 07:56:14.579286 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:14.579252 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fc2a294d-7545bf669d-fsczl"] Sep 16 07:56:14.581392 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:14.581361 2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/665010a2-d972-420d-bf96-fa90f4da5074-plugins-config-volume\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:56:14.581392 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:14.581390 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjt4v\" (UniqueName: \"kubernetes.io/projected/665010a2-d972-420d-bf96-fa90f4da5074-kube-api-access-hjt4v\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:56:14.583418 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:14.583393 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-fc2a294d-7545bf669d-fsczl"] Sep 16 07:56:14.937804 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:14.937765 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="665010a2-d972-420d-bf96-fa90f4da5074" path="/var/lib/kubelet/pods/665010a2-d972-420d-bf96-fa90f4da5074/volumes" Sep 16 07:56:14.938174 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:14.938135 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f195276-8224-4df1-8b70-e98f1b33a49b" path="/var/lib/kubelet/pods/8f195276-8224-4df1-8b70-e98f1b33a49b/volumes" Sep 16 07:56:15.251907 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:15.251796 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:16.057933 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:16.057889 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:16.705161 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:16.705082 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:16.806940 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:16.806902 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:17.154153 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:17.154037 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:17.254874 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:17.254833 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:17.320041 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:17.319983 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e9fc8c-57765cdf-v9t9d"] Sep 16 07:56:17.320636 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:17.320611 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="665010a2-d972-420d-bf96-fa90f4da5074" containerName="payload-pre-processing" Sep 16 07:56:17.320636 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:17.320638 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="665010a2-d972-420d-bf96-fa90f4da5074" containerName="payload-pre-processing" Sep 16 07:56:17.320773 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:17.320741 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="665010a2-d972-420d-bf96-fa90f4da5074" containerName="payload-pre-processing" Sep 16 07:56:17.325622 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:17.325594 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e9fc8c-57765cdf-v9t9d" Sep 16 07:56:17.329218 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:17.329191 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-e9fc8c\"" Sep 16 07:56:17.330339 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:17.330315 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-e9fc8c-dockercfg-rxzrb\"" Sep 16 07:56:17.338854 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:17.338822 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e9fc8c-57765cdf-v9t9d"] Sep 16 07:56:17.344398 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:17.344235 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:17.411084 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:17.410981 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctt44\" (UniqueName: \"kubernetes.io/projected/f1aff88a-f557-4911-a096-859deb592996-kube-api-access-ctt44\") pod \"payload-processing-e2e-shared-b-w3-e9fc8c-57765cdf-v9t9d\" (UID: \"f1aff88a-f557-4911-a096-859deb592996\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e9fc8c-57765cdf-v9t9d" Sep 16 07:56:17.411297 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:17.411115 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1aff88a-f557-4911-a096-859deb592996-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-e9fc8c-57765cdf-v9t9d\" (UID: \"f1aff88a-f557-4911-a096-859deb592996\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e9fc8c-57765cdf-v9t9d" Sep 16 07:56:17.512271 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:17.512223 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1aff88a-f557-4911-a096-859deb592996-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-e9fc8c-57765cdf-v9t9d\" (UID: \"f1aff88a-f557-4911-a096-859deb592996\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e9fc8c-57765cdf-v9t9d" Sep 16 07:56:17.512463 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:17.512364 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctt44\" (UniqueName: \"kubernetes.io/projected/f1aff88a-f557-4911-a096-859deb592996-kube-api-access-ctt44\") pod \"payload-processing-e2e-shared-b-w3-e9fc8c-57765cdf-v9t9d\" (UID: \"f1aff88a-f557-4911-a096-859deb592996\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e9fc8c-57765cdf-v9t9d" Sep 16 07:56:17.513123 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:17.513075 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1aff88a-f557-4911-a096-859deb592996-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-e9fc8c-57765cdf-v9t9d\" (UID: \"f1aff88a-f557-4911-a096-859deb592996\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e9fc8c-57765cdf-v9t9d" Sep 16 07:56:17.520225 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:17.520195 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctt44\" (UniqueName: \"kubernetes.io/projected/f1aff88a-f557-4911-a096-859deb592996-kube-api-access-ctt44\") pod \"payload-processing-e2e-shared-b-w3-e9fc8c-57765cdf-v9t9d\" (UID: \"f1aff88a-f557-4911-a096-859deb592996\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e9fc8c-57765cdf-v9t9d" Sep 16 07:56:17.629369 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:17.629332 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:17.638244 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:17.638208 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e9fc8c-57765cdf-v9t9d" Sep 16 07:56:17.857775 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:17.857738 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:17.919136 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:17.919090 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e9fc8c-57765cdf-v9t9d"] Sep 16 07:56:17.919838 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:56:17.919812 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1aff88a_f557_4911_a096_859deb592996.slice/crio-3649f2afdafed743e9f59046b8eb69e7bd2d5c7847cb15fad344bea71cd59400 WatchSource:0}: Error finding container 3649f2afdafed743e9f59046b8eb69e7bd2d5c7847cb15fad344bea71cd59400: Status 404 returned error can't find the container with id 3649f2afdafed743e9f59046b8eb69e7bd2d5c7847cb15fad344bea71cd59400 Sep 16 07:56:18.345568 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:18.345462 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:18.582083 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:18.582042 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e9fc8c-57765cdf-v9t9d" event={"ID":"f1aff88a-f557-4911-a096-859deb592996","Type":"ContainerStarted","Data":"14ead719e42164f336fab3e988fb856fdf7859bb54b65ec47253ed30f0198a1b"} Sep 16 07:56:18.582253 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:18.582133 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e9fc8c-57765cdf-v9t9d" event={"ID":"f1aff88a-f557-4911-a096-859deb592996","Type":"ContainerStarted","Data":"3649f2afdafed743e9f59046b8eb69e7bd2d5c7847cb15fad344bea71cd59400"} Sep 16 07:56:18.582327 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:18.582289 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e9fc8c-57765cdf-v9t9d" Sep 16 07:56:18.600262 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:18.600139 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e9fc8c-57765cdf-v9t9d" podStartSLOduration=1.600118121 podStartE2EDuration="1.600118121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:56:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:56:18.599204716 +0000 UTC m=+1350.182983055" watchObservedRunningTime="2026-09-16 07:56:18.600118121 +0000 UTC m=+1350.183896458" Sep 16 07:56:18.934284 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:18.934236 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:19.621755 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:19.621713 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:20.261723 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:20.261693 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:22.938967 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:22.938918 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:29.589906 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:29.589873 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e9fc8c-57765cdf-v9t9d" Sep 16 07:56:37.114886 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:37.114841 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:38.657494 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:38.657455 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:38.985454 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:38.985415 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:39.471222 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:39.471160 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:40.244704 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:40.244668 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:40.863271 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:40.863230 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:41.758933 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:41.758895 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:42.538439 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:42.538403 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:42.638479 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:42.638448 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:43.243496 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:43.243460 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:46.745352 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:46.745318 2552 generic.go:356] "Generic (PLEG): container finished" podID="859b21e8-3a32-49ae-be7c-a7249d2e46a7" containerID="a08dec342f9f7fe0b0bb901e58807ed0c1624722dbab28cd69856222a56dede8" exitCode=0 Sep 16 07:56:46.745871 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:46.745400 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" event={"ID":"859b21e8-3a32-49ae-be7c-a7249d2e46a7","Type":"ContainerDied","Data":"a08dec342f9f7fe0b0bb901e58807ed0c1624722dbab28cd69856222a56dede8"} Sep 16 07:56:46.745995 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:46.745976 2552 scope.go:117] "RemoveContainer" containerID="a08dec342f9f7fe0b0bb901e58807ed0c1624722dbab28cd69856222a56dede8" Sep 16 07:56:46.747479 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:46.747447 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 16 07:56:46.747562 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:46.747543 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 16 07:56:46.747607 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:46.747591 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 16 07:56:47.757745 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:47.756931 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" event={"ID":"859b21e8-3a32-49ae-be7c-a7249d2e46a7","Type":"ContainerStarted","Data":"3cbe1e993c33cf89453646a14533595ec09d316ff819a43d99a536e60ad44dec"} Sep 16 07:56:48.477726 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:48.477686 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" Sep 16 07:56:48.477916 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:48.477871 2552 patch_prober.go:28] interesting pod/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Sep 16 07:56:48.477966 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:48.477919 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" podUID="859b21e8-3a32-49ae-be7c-a7249d2e46a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Sep 16 07:56:48.768213 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:48.768120 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:48.925018 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:48.924980 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:49.477968 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:49.477930 2552 patch_prober.go:28] interesting pod/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Sep 16 07:56:49.478174 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:49.478002 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" podUID="859b21e8-3a32-49ae-be7c-a7249d2e46a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Sep 16 07:56:50.477868 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:50.477827 2552 patch_prober.go:28] interesting pod/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Sep 16 07:56:50.478280 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:50.477892 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" podUID="859b21e8-3a32-49ae-be7c-a7249d2e46a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Sep 16 07:56:51.478318 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:51.478275 2552 patch_prober.go:28] interesting pod/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Sep 16 07:56:51.478723 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:51.478336 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" podUID="859b21e8-3a32-49ae-be7c-a7249d2e46a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Sep 16 07:56:52.477868 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:52.477828 2552 patch_prober.go:28] interesting pod/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Sep 16 07:56:52.478058 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:52.477891 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" podUID="859b21e8-3a32-49ae-be7c-a7249d2e46a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Sep 16 07:56:53.483001 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:53.482963 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" Sep 16 07:56:53.483463 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:53.483396 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" Sep 16 07:56:53.484390 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:53.484368 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" Sep 16 07:56:53.494884 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:53.494853 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:56:59.673041 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:56:59.673003 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:57:00.128835 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:57:00.128741 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:57:05.777139 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:57:05.777089 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:57:28.717801 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:57:28.717756 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:57:32.006344 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:57:32.006311 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:57:35.847508 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:57:35.847467 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:57:37.117337 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:57:37.117301 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:57:37.217341 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:57:37.217309 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:57:37.310587 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:57:37.310555 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:57:38.637757 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:57:38.637720 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:57:38.738573 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:57:38.738529 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:57:38.835167 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:57:38.835132 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:57:39.411481 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:57:39.411443 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:57:39.510832 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:57:39.510794 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:57:39.614163 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:57:39.610375 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:57:40.909822 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:57:40.909782 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:57:46.521688 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:57:46.521650 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:57:46.589872 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:57:46.589835 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:57:47.234022 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:57:47.233988 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:57:47.334892 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:57:47.334860 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:57:53.330356 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:57:53.330320 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:57:53.435950 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:57:53.435916 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:57:53.538371 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:57:53.538340 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:57:54.276033 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:57:54.275988 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:58:04.014891 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:58:04.014851 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:58:04.241895 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:58:04.241859 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:58:08.649343 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:58:08.649306 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:58:25.186504 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:58:25.186470 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:58:28.083944 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:58:28.083895 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:58:28.569658 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:58:28.569627 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:58:30.846017 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:58:30.845979 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:58:30.910544 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:58:30.910504 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:58:31.205507 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:58:31.205460 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:58:31.555565 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:58:31.555461 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:58:32.035351 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:58:32.035303 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:58:33.638607 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:58:33.638547 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:58:34.359333 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:58:34.359298 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:58:34.461181 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:58:34.461141 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:58:34.560768 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:58:34.560730 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:58:35.238993 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:58:35.238951 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:58:36.117767 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:58:36.117720 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:58:36.788118 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:58:36.788073 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:58:37.457059 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:58:37.457024 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:58:38.252608 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:58:38.252571 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:58:38.893969 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:58:38.893924 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:58:38.998597 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:58:38.998563 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:58:39.132644 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:58:39.132608 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:58:39.374045 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:58:39.374013 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:58:39.461317 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:58:39.461276 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:58:49.051984 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:58:49.051948 2552 scope.go:117] "RemoveContainer" containerID="f80a6cc7c38b7c3a66fc4ec3f54201316cb84968e607fcb579634d6a44c193a6" Sep 16 07:58:53.218505 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:58:53.218466 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:58:53.314614 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:58:53.314583 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:00.832904 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:00.832864 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:01.465452 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:01.465406 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:02.197036 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:02.196992 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:02.865753 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:02.865715 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:03.561466 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:03.561424 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:04.268426 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:04.268388 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:04.449336 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:04.449303 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:04.902207 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:04.902176 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:05.002582 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:05.002546 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:05.234634 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:05.234603 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:05.518916 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:05.518829 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:05.687706 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:05.687675 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:07.893650 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:07.893620 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:08.523677 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:08.523628 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:08.636899 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:08.636862 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:08.737351 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:08.737303 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:09.488112 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:09.488056 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:10.156947 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:10.156910 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:10.869852 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:10.869808 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:11.736294 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:11.736227 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:13.209421 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:13.209384 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:13.797895 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:13.797865 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:13.898909 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:13.898872 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:14.001245 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:14.001207 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:14.249119 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:14.249064 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:14.338669 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:14.338635 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:14.339564 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:14.339540 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3a5b4c-765cd4576b-hwd2c"] Sep 16 07:59:14.344640 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:14.344614 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3a5b4c-765cd4576b-hwd2c" Sep 16 07:59:14.347204 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:14.347090 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-3a5b4c\"" Sep 16 07:59:14.347204 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:14.347162 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-3a5b4c-dockercfg-kt2cp\"" Sep 16 07:59:14.351595 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:14.351565 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3a5b4c-765cd4576b-hwd2c"] Sep 16 07:59:14.385779 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:14.385741 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3a5b4c-df989476f-np6nd"] Sep 16 07:59:14.389461 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:14.389440 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3a5b4c-df989476f-np6nd" Sep 16 07:59:14.399223 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:14.399193 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3a5b4c-df989476f-np6nd"] Sep 16 07:59:14.432016 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:14.431968 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3aca058-bc0c-4725-ba3f-e368b71ba834-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3a5b4c-765cd4576b-hwd2c\" (UID: \"f3aca058-bc0c-4725-ba3f-e368b71ba834\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3a5b4c-765cd4576b-hwd2c" Sep 16 07:59:14.432272 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:14.432039 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqqnr\" (UniqueName: \"kubernetes.io/projected/f3aca058-bc0c-4725-ba3f-e368b71ba834-kube-api-access-lqqnr\") pod \"payload-pre-processing-e2e-api-b-3a5b4c-765cd4576b-hwd2c\" (UID: \"f3aca058-bc0c-4725-ba3f-e368b71ba834\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3a5b4c-765cd4576b-hwd2c" Sep 16 07:59:14.532926 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:14.532819 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s44d\" (UniqueName: \"kubernetes.io/projected/945aff56-7bbe-4bf6-a1d0-a5edcaf6f355-kube-api-access-5s44d\") pod \"payload-processing-e2e-api-b-3a5b4c-df989476f-np6nd\" (UID: \"945aff56-7bbe-4bf6-a1d0-a5edcaf6f355\") " pod="openshift-ingress/payload-processing-e2e-api-b-3a5b4c-df989476f-np6nd" Sep 16 07:59:14.533093 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:14.532977 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3aca058-bc0c-4725-ba3f-e368b71ba834-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3a5b4c-765cd4576b-hwd2c\" (UID: \"f3aca058-bc0c-4725-ba3f-e368b71ba834\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3a5b4c-765cd4576b-hwd2c" Sep 16 07:59:14.533093 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:14.533024 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqqnr\" (UniqueName: \"kubernetes.io/projected/f3aca058-bc0c-4725-ba3f-e368b71ba834-kube-api-access-lqqnr\") pod \"payload-pre-processing-e2e-api-b-3a5b4c-765cd4576b-hwd2c\" (UID: \"f3aca058-bc0c-4725-ba3f-e368b71ba834\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3a5b4c-765cd4576b-hwd2c" Sep 16 07:59:14.533093 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:14.533066 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/945aff56-7bbe-4bf6-a1d0-a5edcaf6f355-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3a5b4c-df989476f-np6nd\" (UID: \"945aff56-7bbe-4bf6-a1d0-a5edcaf6f355\") " pod="openshift-ingress/payload-processing-e2e-api-b-3a5b4c-df989476f-np6nd" Sep 16 07:59:14.533792 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:14.533765 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3aca058-bc0c-4725-ba3f-e368b71ba834-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-3a5b4c-765cd4576b-hwd2c\" (UID: \"f3aca058-bc0c-4725-ba3f-e368b71ba834\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3a5b4c-765cd4576b-hwd2c" Sep 16 07:59:14.541371 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:14.541346 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqqnr\" (UniqueName: \"kubernetes.io/projected/f3aca058-bc0c-4725-ba3f-e368b71ba834-kube-api-access-lqqnr\") pod \"payload-pre-processing-e2e-api-b-3a5b4c-765cd4576b-hwd2c\" (UID: \"f3aca058-bc0c-4725-ba3f-e368b71ba834\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-3a5b4c-765cd4576b-hwd2c" Sep 16 07:59:14.633983 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:14.633940 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s44d\" (UniqueName: \"kubernetes.io/projected/945aff56-7bbe-4bf6-a1d0-a5edcaf6f355-kube-api-access-5s44d\") pod \"payload-processing-e2e-api-b-3a5b4c-df989476f-np6nd\" (UID: \"945aff56-7bbe-4bf6-a1d0-a5edcaf6f355\") " pod="openshift-ingress/payload-processing-e2e-api-b-3a5b4c-df989476f-np6nd" Sep 16 07:59:14.634212 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:14.634062 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/945aff56-7bbe-4bf6-a1d0-a5edcaf6f355-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3a5b4c-df989476f-np6nd\" (UID: \"945aff56-7bbe-4bf6-a1d0-a5edcaf6f355\") " pod="openshift-ingress/payload-processing-e2e-api-b-3a5b4c-df989476f-np6nd" Sep 16 07:59:14.634787 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:14.634765 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/945aff56-7bbe-4bf6-a1d0-a5edcaf6f355-plugins-config-volume\") pod \"payload-processing-e2e-api-b-3a5b4c-df989476f-np6nd\" (UID: \"945aff56-7bbe-4bf6-a1d0-a5edcaf6f355\") " pod="openshift-ingress/payload-processing-e2e-api-b-3a5b4c-df989476f-np6nd" Sep 16 07:59:14.641985 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:14.641954 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s44d\" (UniqueName: \"kubernetes.io/projected/945aff56-7bbe-4bf6-a1d0-a5edcaf6f355-kube-api-access-5s44d\") pod \"payload-processing-e2e-api-b-3a5b4c-df989476f-np6nd\" (UID: \"945aff56-7bbe-4bf6-a1d0-a5edcaf6f355\") " pod="openshift-ingress/payload-processing-e2e-api-b-3a5b4c-df989476f-np6nd" Sep 16 07:59:14.657703 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:14.657660 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3a5b4c-765cd4576b-hwd2c" Sep 16 07:59:14.701276 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:14.701237 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3a5b4c-df989476f-np6nd" Sep 16 07:59:14.934456 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:59:14.934429 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3aca058_bc0c_4725_ba3f_e368b71ba834.slice/crio-1c7847227ade92c39ed69aab93c5c0988116fcae10bdf111997fa4020ff9108c WatchSource:0}: Error finding container 1c7847227ade92c39ed69aab93c5c0988116fcae10bdf111997fa4020ff9108c: Status 404 returned error can't find the container with id 1c7847227ade92c39ed69aab93c5c0988116fcae10bdf111997fa4020ff9108c Sep 16 07:59:14.939361 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:14.939337 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3a5b4c-765cd4576b-hwd2c"] Sep 16 07:59:14.990573 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:14.990546 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3a5b4c-df989476f-np6nd"] Sep 16 07:59:14.992773 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:59:14.992746 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod945aff56_7bbe_4bf6_a1d0_a5edcaf6f355.slice/crio-69a262c5eecf0fd6c153ffc3892f684fc71cfa816abc9b1e911dba27b004f9bc WatchSource:0}: Error finding container 69a262c5eecf0fd6c153ffc3892f684fc71cfa816abc9b1e911dba27b004f9bc: Status 404 returned error can't find the container with id 69a262c5eecf0fd6c153ffc3892f684fc71cfa816abc9b1e911dba27b004f9bc Sep 16 07:59:15.602066 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:15.602030 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3a5b4c-df989476f-np6nd" event={"ID":"945aff56-7bbe-4bf6-a1d0-a5edcaf6f355","Type":"ContainerStarted","Data":"193bb59c3d2f702bcdb57ca0b2decbad93c2064c8a45297331f78374696bd018"} Sep 16 07:59:15.602569 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:15.602171 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-3a5b4c-df989476f-np6nd" Sep 16 07:59:15.602569 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:15.602210 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3a5b4c-df989476f-np6nd" event={"ID":"945aff56-7bbe-4bf6-a1d0-a5edcaf6f355","Type":"ContainerStarted","Data":"69a262c5eecf0fd6c153ffc3892f684fc71cfa816abc9b1e911dba27b004f9bc"} Sep 16 07:59:15.603790 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:15.603759 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3a5b4c-765cd4576b-hwd2c" event={"ID":"f3aca058-bc0c-4725-ba3f-e368b71ba834","Type":"ContainerStarted","Data":"a74e1b43feaff8ed073769a2e3f8799bb4a64ec477a70a0529dbe5b0b1b3b94c"} Sep 16 07:59:15.603925 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:15.603806 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3a5b4c-765cd4576b-hwd2c" event={"ID":"f3aca058-bc0c-4725-ba3f-e368b71ba834","Type":"ContainerStarted","Data":"1c7847227ade92c39ed69aab93c5c0988116fcae10bdf111997fa4020ff9108c"} Sep 16 07:59:15.603925 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:15.603870 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3a5b4c-765cd4576b-hwd2c" Sep 16 07:59:15.619355 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:15.619297 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-3a5b4c-df989476f-np6nd" podStartSLOduration=1.619281753 podStartE2EDuration="1.619281753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:59:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:59:15.617815289 +0000 UTC m=+1527.201593628" watchObservedRunningTime="2026-09-16 07:59:15.619281753 +0000 UTC m=+1527.203060090" Sep 16 07:59:15.632217 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:15.632152 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3a5b4c-765cd4576b-hwd2c" podStartSLOduration=1.632130409 podStartE2EDuration="1.632130409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 07:59:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 07:59:15.631876569 +0000 UTC m=+1527.215654907" watchObservedRunningTime="2026-09-16 07:59:15.632130409 +0000 UTC m=+1527.215908762" Sep 16 07:59:26.612539 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:26.612504 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-3a5b4c-df989476f-np6nd" Sep 16 07:59:26.612965 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:26.612577 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3a5b4c-765cd4576b-hwd2c" Sep 16 07:59:35.862419 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:35.862380 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:36.672905 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:36.672865 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:37.299157 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:37.299119 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:38.064066 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:38.064019 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:38.596729 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:38.596689 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:38.877594 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:38.877498 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:39.716237 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:39.716199 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:39.813867 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:39.813833 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:39.872676 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:39.872645 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c9020d-km29c"] Sep 16 07:59:39.876560 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:39.876538 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c9020d-km29c" Sep 16 07:59:39.878750 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:39.878728 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 07:59:39.878858 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:39.878793 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zwk5r\"" Sep 16 07:59:39.883232 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:39.883201 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c9020d-km29c"] Sep 16 07:59:39.988595 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:39.988497 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2270b50b-94ea-48c1-b651-57b5a525559d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c9020d-km29c\" (UID: \"2270b50b-94ea-48c1-b651-57b5a525559d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c9020d-km29c" Sep 16 07:59:40.089716 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:40.089671 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2270b50b-94ea-48c1-b651-57b5a525559d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c9020d-km29c\" (UID: \"2270b50b-94ea-48c1-b651-57b5a525559d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c9020d-km29c" Sep 16 07:59:40.090394 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:40.090372 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2270b50b-94ea-48c1-b651-57b5a525559d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c9020d-km29c\" (UID: \"2270b50b-94ea-48c1-b651-57b5a525559d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c9020d-km29c" Sep 16 07:59:40.189744 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:40.189702 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c9020d-km29c" Sep 16 07:59:40.466554 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:40.466525 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c9020d-km29c"] Sep 16 07:59:40.469366 ip-10-0-142-170 kubenswrapper[2552]: W0916 07:59:40.469339 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2270b50b_94ea_48c1_b651_57b5a525559d.slice/crio-0d54c399c63f23a051645c2f5d2d7e72d8e175f2dafd09f677d429eff4b88468 WatchSource:0}: Error finding container 0d54c399c63f23a051645c2f5d2d7e72d8e175f2dafd09f677d429eff4b88468: Status 404 returned error can't find the container with id 0d54c399c63f23a051645c2f5d2d7e72d8e175f2dafd09f677d429eff4b88468 Sep 16 07:59:40.485407 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:40.485362 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:40.741613 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:40.741522 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c9020d-km29c" event={"ID":"2270b50b-94ea-48c1-b651-57b5a525559d","Type":"ContainerStarted","Data":"0d54c399c63f23a051645c2f5d2d7e72d8e175f2dafd09f677d429eff4b88468"} Sep 16 07:59:41.268530 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:41.268492 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:41.749244 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:41.749207 2552 generic.go:356] "Generic (PLEG): container finished" podID="2270b50b-94ea-48c1-b651-57b5a525559d" containerID="4e0c57bc062d0ba93a6cca1b254b5b6a0272b66e6d43421f06ce081534f2f497" exitCode=0 Sep 16 07:59:41.749748 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:41.749301 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c9020d-km29c" event={"ID":"2270b50b-94ea-48c1-b651-57b5a525559d","Type":"ContainerDied","Data":"4e0c57bc062d0ba93a6cca1b254b5b6a0272b66e6d43421f06ce081534f2f497"} Sep 16 07:59:42.069898 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:42.069799 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:42.851390 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:42.851351 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:42.912695 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:42.912664 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c9020d-km29c" Sep 16 07:59:43.019939 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:43.019843 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2270b50b-94ea-48c1-b651-57b5a525559d-maas-api-service-ca\") pod \"2270b50b-94ea-48c1-b651-57b5a525559d\" (UID: \"2270b50b-94ea-48c1-b651-57b5a525559d\") " Sep 16 07:59:43.020254 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:43.020230 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2270b50b-94ea-48c1-b651-57b5a525559d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2270b50b-94ea-48c1-b651-57b5a525559d" (UID: "2270b50b-94ea-48c1-b651-57b5a525559d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:59:43.120915 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:43.120878 2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2270b50b-94ea-48c1-b651-57b5a525559d-maas-api-service-ca\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:59:43.577350 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:43.577306 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:43.761406 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:43.761368 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c9020d-km29c" event={"ID":"2270b50b-94ea-48c1-b651-57b5a525559d","Type":"ContainerDied","Data":"0d54c399c63f23a051645c2f5d2d7e72d8e175f2dafd09f677d429eff4b88468"} Sep 16 07:59:43.761406 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:43.761406 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c9020d-km29c" Sep 16 07:59:43.761631 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:43.761429 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d54c399c63f23a051645c2f5d2d7e72d8e175f2dafd09f677d429eff4b88468" Sep 16 07:59:44.179066 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:44.179031 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:44.742355 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:44.742319 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c9020d/distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn"] Sep 16 07:59:44.769748 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:44.769711 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c9020d/premium-w0-c9020d-kserve-78578c546b-8zspq"] Sep 16 07:59:44.778415 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:44.778379 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c9020d/simulator-w0-c9020d-kserve-5946f6c555-55w4c"] Sep 16 07:59:44.853813 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:44.853774 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:44.954331 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:44.954288 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:45.561854 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:45.561820 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:46.215806 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:46.215766 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:46.335334 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:46.335274 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:46.922884 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:46.922847 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:47.742353 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:47.742314 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:48.140467 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:48.140422 2552 prober.go:120] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s7n7g" podUID="955c6c96-0681-482f-b967-946e3b2eda9f" containerName="manager" probeResult="failure" output="Get \"http://10.133.0.24:8081/readyz\": dial tcp 10.133.0.24:8081: connect: connection refused" Sep 16 07:59:48.791895 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:48.791856 2552 generic.go:356] "Generic (PLEG): container finished" podID="955c6c96-0681-482f-b967-946e3b2eda9f" containerID="dec07385c2e6d7f2a22bfe88fed8e3ec2f4e66935dc3d1fd537a133cee364940" exitCode=137 Sep 16 07:59:48.792083 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:48.791927 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s7n7g" event={"ID":"955c6c96-0681-482f-b967-946e3b2eda9f","Type":"ContainerDied","Data":"dec07385c2e6d7f2a22bfe88fed8e3ec2f4e66935dc3d1fd537a133cee364940"} Sep 16 07:59:48.792326 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:48.792311 2552 scope.go:117] "RemoveContainer" containerID="dec07385c2e6d7f2a22bfe88fed8e3ec2f4e66935dc3d1fd537a133cee364940" Sep 16 07:59:49.799732 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:49.799702 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s7n7g" event={"ID":"955c6c96-0681-482f-b967-946e3b2eda9f","Type":"ContainerStarted","Data":"c4471232bfc1dfdde840118a0572e29e1197f3ced49622cfc06905ca367dc6b2"} Sep 16 07:59:49.800220 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:49.799918 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s7n7g" Sep 16 07:59:50.068917 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:50.068822 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:51.590280 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:51.590244 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:54.210872 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:54.210834 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:54.313864 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:54.313814 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:55.863600 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:55.863566 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:55.896919 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:55.896878 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-c9020d-77dd676546-fzccc"] Sep 16 07:59:55.897237 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:55.897209 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c9020d-77dd676546-fzccc" podUID="f7caee0e-a84b-4c3f-8bf1-3b0ddc2ec2b0" containerName="payload-pre-processing" containerID="cri-o://09638339fe0bf373780445cf120880999aad113f52f3eda508a350013e118beb" gracePeriod=30 Sep 16 07:59:55.929614 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:55.929586 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:56.030618 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:56.030585 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:56.051231 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:56.051193 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c9020d-km29c"] Sep 16 07:59:56.057869 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:56.057830 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c9020d-km29c"] Sep 16 07:59:56.131365 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:56.131257 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:56.166037 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:56.166006 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c9020d-77dd676546-fzccc" Sep 16 07:59:56.267221 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:56.267183 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7caee0e-a84b-4c3f-8bf1-3b0ddc2ec2b0-plugins-config-volume\") pod \"f7caee0e-a84b-4c3f-8bf1-3b0ddc2ec2b0\" (UID: \"f7caee0e-a84b-4c3f-8bf1-3b0ddc2ec2b0\") " Sep 16 07:59:56.267405 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:56.267354 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s47w9\" (UniqueName: \"kubernetes.io/projected/f7caee0e-a84b-4c3f-8bf1-3b0ddc2ec2b0-kube-api-access-s47w9\") pod \"f7caee0e-a84b-4c3f-8bf1-3b0ddc2ec2b0\" (UID: \"f7caee0e-a84b-4c3f-8bf1-3b0ddc2ec2b0\") " Sep 16 07:59:56.267667 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:56.267642 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7caee0e-a84b-4c3f-8bf1-3b0ddc2ec2b0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f7caee0e-a84b-4c3f-8bf1-3b0ddc2ec2b0" (UID: "f7caee0e-a84b-4c3f-8bf1-3b0ddc2ec2b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 07:59:56.269592 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:56.269568 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7caee0e-a84b-4c3f-8bf1-3b0ddc2ec2b0-kube-api-access-s47w9" (OuterVolumeSpecName: "kube-api-access-s47w9") pod "f7caee0e-a84b-4c3f-8bf1-3b0ddc2ec2b0" (UID: "f7caee0e-a84b-4c3f-8bf1-3b0ddc2ec2b0"). InnerVolumeSpecName "kube-api-access-s47w9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 07:59:56.368355 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:56.368304 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s47w9\" (UniqueName: \"kubernetes.io/projected/f7caee0e-a84b-4c3f-8bf1-3b0ddc2ec2b0-kube-api-access-s47w9\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:59:56.368355 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:56.368349 2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7caee0e-a84b-4c3f-8bf1-3b0ddc2ec2b0-plugins-config-volume\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 07:59:56.397698 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:56.397605 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:56.498613 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:56.498571 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:56.846800 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:56.846742 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c9020d-77dd676546-fzccc" event={"ID":"f7caee0e-a84b-4c3f-8bf1-3b0ddc2ec2b0","Type":"ContainerDied","Data":"09638339fe0bf373780445cf120880999aad113f52f3eda508a350013e118beb"} Sep 16 07:59:56.847029 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:56.846864 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c9020d-77dd676546-fzccc" Sep 16 07:59:56.847029 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:56.846886 2552 scope.go:117] "RemoveContainer" containerID="09638339fe0bf373780445cf120880999aad113f52f3eda508a350013e118beb" Sep 16 07:59:56.847175 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:56.846710 2552 generic.go:356] "Generic (PLEG): container finished" podID="f7caee0e-a84b-4c3f-8bf1-3b0ddc2ec2b0" containerID="09638339fe0bf373780445cf120880999aad113f52f3eda508a350013e118beb" exitCode=0 Sep 16 07:59:56.847251 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:56.847229 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-c9020d-77dd676546-fzccc" event={"ID":"f7caee0e-a84b-4c3f-8bf1-3b0ddc2ec2b0","Type":"ContainerDied","Data":"2a1456f1e47669bf82e52f4bc8b7b028315b95b786ad80675f181668b327258a"} Sep 16 07:59:56.864385 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:56.864364 2552 scope.go:117] "RemoveContainer" containerID="09638339fe0bf373780445cf120880999aad113f52f3eda508a350013e118beb" Sep 16 07:59:56.864766 ip-10-0-142-170 kubenswrapper[2552]: E0916 07:59:56.864743 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09638339fe0bf373780445cf120880999aad113f52f3eda508a350013e118beb\": container with ID starting with 09638339fe0bf373780445cf120880999aad113f52f3eda508a350013e118beb not found: ID does not exist" containerID="09638339fe0bf373780445cf120880999aad113f52f3eda508a350013e118beb" Sep 16 07:59:56.864817 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:56.864776 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09638339fe0bf373780445cf120880999aad113f52f3eda508a350013e118beb"} err="failed to get container status \"09638339fe0bf373780445cf120880999aad113f52f3eda508a350013e118beb\": rpc error: code = NotFound desc = could not find container \"09638339fe0bf373780445cf120880999aad113f52f3eda508a350013e118beb\": container with ID starting with 09638339fe0bf373780445cf120880999aad113f52f3eda508a350013e118beb not found: ID does not exist" Sep 16 07:59:56.873251 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:56.873219 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-c9020d-77dd676546-fzccc"] Sep 16 07:59:56.877530 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:56.877495 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-c9020d-77dd676546-fzccc"] Sep 16 07:59:56.939144 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:56.939082 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2270b50b-94ea-48c1-b651-57b5a525559d" path="/var/lib/kubelet/pods/2270b50b-94ea-48c1-b651-57b5a525559d/volumes" Sep 16 07:59:56.939532 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:56.939518 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7caee0e-a84b-4c3f-8bf1-3b0ddc2ec2b0" path="/var/lib/kubelet/pods/f7caee0e-a84b-4c3f-8bf1-3b0ddc2ec2b0/volumes" Sep 16 07:59:57.164495 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:57.164407 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:57.969144 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:57.969062 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:58.595124 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:58.595065 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:58.668560 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:58.668523 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:58.726310 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:58.726273 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:58.815253 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:58.815202 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:58.875788 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:58.875680 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:58.942683 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:58.942649 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:59.177298 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:59.177262 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:59.602841 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:59.602742 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 07:59:59.763596 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:59.763551 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-c9020d/distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn" podUID="e9e4ab9e-482a-459c-b596-f78f156d906c" containerName="main" containerID="cri-o://2bf27c6358eaf2ca4bacc00c9f203c077b8dd7faf0b69627784c254036e78e2b" gracePeriod=45 Sep 16 07:59:59.790534 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:59.790495 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-c9020d/premium-w0-c9020d-kserve-78578c546b-8zspq" podUID="4543c78d-4a9b-437c-8146-89e1eed8b7a8" containerName="main" containerID="cri-o://3fddfd579122ffe1f47b71943c34bd8cbc2300b5872d500cc9df0daa204f45b5" gracePeriod=45 Sep 16 07:59:59.804363 ip-10-0-142-170 kubenswrapper[2552]: I0916 07:59:59.804326 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-c9020d/simulator-w0-c9020d-kserve-5946f6c555-55w4c" podUID="30c1f9e7-d353-4a41-8336-b87728ac9838" containerName="main" containerID="cri-o://fae213868325c5f8c125bd8fe94b9dc7a8b2bbb3150911061db01fcf4ab22128" gracePeriod=45 Sep 16 08:00:00.106729 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.106698 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c9020d/distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn" Sep 16 08:00:00.125045 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.125015 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c9020d/simulator-w0-c9020d-kserve-5946f6c555-55w4c" Sep 16 08:00:00.130517 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.130491 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c9020d/premium-w0-c9020d-kserve-78578c546b-8zspq" Sep 16 08:00:00.154623 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.154532 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fc27a5-29825760-2ktwd"] Sep 16 08:00:00.155179 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.155159 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4543c78d-4a9b-437c-8146-89e1eed8b7a8" containerName="main" Sep 16 08:00:00.155179 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.155180 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="4543c78d-4a9b-437c-8146-89e1eed8b7a8" containerName="main" Sep 16 08:00:00.155280 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.155193 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7caee0e-a84b-4c3f-8bf1-3b0ddc2ec2b0" containerName="payload-pre-processing" Sep 16 08:00:00.155280 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.155199 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7caee0e-a84b-4c3f-8bf1-3b0ddc2ec2b0" containerName="payload-pre-processing" Sep 16 08:00:00.155280 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.155210 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30c1f9e7-d353-4a41-8336-b87728ac9838" containerName="main" Sep 16 08:00:00.155280 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.155216 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="30c1f9e7-d353-4a41-8336-b87728ac9838" containerName="main" Sep 16 08:00:00.155280 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.155223 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2270b50b-94ea-48c1-b651-57b5a525559d" containerName="revoke-keys" Sep 16 08:00:00.155280 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.155228 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="2270b50b-94ea-48c1-b651-57b5a525559d" containerName="revoke-keys" Sep 16 08:00:00.155280 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.155244 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9e4ab9e-482a-459c-b596-f78f156d906c" containerName="main" Sep 16 08:00:00.155280 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.155249 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9e4ab9e-482a-459c-b596-f78f156d906c" containerName="main" Sep 16 08:00:00.155541 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.155319 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="30c1f9e7-d353-4a41-8336-b87728ac9838" containerName="main" Sep 16 08:00:00.155541 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.155328 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="2270b50b-94ea-48c1-b651-57b5a525559d" containerName="revoke-keys" Sep 16 08:00:00.155541 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.155334 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9e4ab9e-482a-459c-b596-f78f156d906c" containerName="main" Sep 16 08:00:00.155541 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.155341 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="4543c78d-4a9b-437c-8146-89e1eed8b7a8" containerName="main" Sep 16 08:00:00.155541 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.155350 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7caee0e-a84b-4c3f-8bf1-3b0ddc2ec2b0" containerName="payload-pre-processing" Sep 16 08:00:00.158423 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.158395 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fc27a5-29825760-2ktwd" Sep 16 08:00:00.158561 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.158482 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-984c1a-29825760-v69p5"] Sep 16 08:00:00.161803 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.161445 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zwk5r\"" Sep 16 08:00:00.163128 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.163071 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-3a5b4c-29825760-fbgc7"] Sep 16 08:00:00.163269 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.163199 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-984c1a-29825760-v69p5" Sep 16 08:00:00.177131 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.175777 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-3a5b4c-29825760-fbgc7"] Sep 16 08:00:00.177131 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.175824 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fc27a5-29825760-2ktwd"] Sep 16 08:00:00.177131 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.175926 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-3a5b4c-29825760-fbgc7" Sep 16 08:00:00.178374 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.178300 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-984c1a-29825760-v69p5"] Sep 16 08:00:00.181578 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.181551 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fc27a5-29825760-2ktwd" Sep 16 08:00:00.181771 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.181740 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3616cc-29825760-b8992"] Sep 16 08:00:00.193459 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.189746 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3616cc-29825760-b8992" Sep 16 08:00:00.198515 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.198469 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3616cc-29825760-b8992"] Sep 16 08:00:00.199616 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.199439 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-984c1a-29825760-v69p5" Sep 16 08:00:00.203254 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.203141 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30c1f9e7-d353-4a41-8336-b87728ac9838-dshm\") pod \"30c1f9e7-d353-4a41-8336-b87728ac9838\" (UID: \"30c1f9e7-d353-4a41-8336-b87728ac9838\") " Sep 16 08:00:00.203254 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.203179 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgmjv\" (UniqueName: \"kubernetes.io/projected/e9e4ab9e-482a-459c-b596-f78f156d906c-kube-api-access-jgmjv\") pod \"e9e4ab9e-482a-459c-b596-f78f156d906c\" (UID: \"e9e4ab9e-482a-459c-b596-f78f156d906c\") " Sep 16 08:00:00.203254 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.203205 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9e4ab9e-482a-459c-b596-f78f156d906c-tls-certs\") pod \"e9e4ab9e-482a-459c-b596-f78f156d906c\" (UID: \"e9e4ab9e-482a-459c-b596-f78f156d906c\") " Sep 16 08:00:00.203254 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.203234 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l68g7\" (UniqueName: \"kubernetes.io/projected/4543c78d-4a9b-437c-8146-89e1eed8b7a8-kube-api-access-l68g7\") pod \"4543c78d-4a9b-437c-8146-89e1eed8b7a8\" (UID: \"4543c78d-4a9b-437c-8146-89e1eed8b7a8\") " Sep 16 08:00:00.203478 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.203266 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9e4ab9e-482a-459c-b596-f78f156d906c-tmp-dir\") pod \"e9e4ab9e-482a-459c-b596-f78f156d906c\" (UID: \"e9e4ab9e-482a-459c-b596-f78f156d906c\") " Sep 16 08:00:00.203478 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.203295 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9e4ab9e-482a-459c-b596-f78f156d906c-dshm\") pod \"e9e4ab9e-482a-459c-b596-f78f156d906c\" (UID: \"e9e4ab9e-482a-459c-b596-f78f156d906c\") " Sep 16 08:00:00.203478 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.203328 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7x5jz\" (UniqueName: \"kubernetes.io/projected/30c1f9e7-d353-4a41-8336-b87728ac9838-kube-api-access-7x5jz\") pod \"30c1f9e7-d353-4a41-8336-b87728ac9838\" (UID: \"30c1f9e7-d353-4a41-8336-b87728ac9838\") " Sep 16 08:00:00.203478 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.203365 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4543c78d-4a9b-437c-8146-89e1eed8b7a8-home\") pod \"4543c78d-4a9b-437c-8146-89e1eed8b7a8\" (UID: \"4543c78d-4a9b-437c-8146-89e1eed8b7a8\") " Sep 16 08:00:00.203478 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.203393 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4543c78d-4a9b-437c-8146-89e1eed8b7a8-tmp-dir\") pod \"4543c78d-4a9b-437c-8146-89e1eed8b7a8\" (UID: \"4543c78d-4a9b-437c-8146-89e1eed8b7a8\") " Sep 16 08:00:00.203478 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.203416 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4543c78d-4a9b-437c-8146-89e1eed8b7a8-dshm\") pod \"4543c78d-4a9b-437c-8146-89e1eed8b7a8\" (UID: \"4543c78d-4a9b-437c-8146-89e1eed8b7a8\") " Sep 16 08:00:00.203478 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.203463 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4543c78d-4a9b-437c-8146-89e1eed8b7a8-model-cache\") pod \"4543c78d-4a9b-437c-8146-89e1eed8b7a8\" (UID: \"4543c78d-4a9b-437c-8146-89e1eed8b7a8\") " Sep 16 08:00:00.203859 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.203494 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30c1f9e7-d353-4a41-8336-b87728ac9838-home\") pod \"30c1f9e7-d353-4a41-8336-b87728ac9838\" (UID: \"30c1f9e7-d353-4a41-8336-b87728ac9838\") " Sep 16 08:00:00.203859 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.203546 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30c1f9e7-d353-4a41-8336-b87728ac9838-tls-certs\") pod \"30c1f9e7-d353-4a41-8336-b87728ac9838\" (UID: \"30c1f9e7-d353-4a41-8336-b87728ac9838\") " Sep 16 08:00:00.203859 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.203591 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30c1f9e7-d353-4a41-8336-b87728ac9838-model-cache\") pod \"30c1f9e7-d353-4a41-8336-b87728ac9838\" (UID: \"30c1f9e7-d353-4a41-8336-b87728ac9838\") " Sep 16 08:00:00.203859 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.203641 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4543c78d-4a9b-437c-8146-89e1eed8b7a8-tls-certs\") pod \"4543c78d-4a9b-437c-8146-89e1eed8b7a8\" (UID: \"4543c78d-4a9b-437c-8146-89e1eed8b7a8\") " Sep 16 08:00:00.204078 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.203863 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30c1f9e7-d353-4a41-8336-b87728ac9838-tmp-dir\") pod \"30c1f9e7-d353-4a41-8336-b87728ac9838\" (UID: \"30c1f9e7-d353-4a41-8336-b87728ac9838\") " Sep 16 08:00:00.204078 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.203895 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9e4ab9e-482a-459c-b596-f78f156d906c-model-cache\") pod \"e9e4ab9e-482a-459c-b596-f78f156d906c\" (UID: \"e9e4ab9e-482a-459c-b596-f78f156d906c\") " Sep 16 08:00:00.204078 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.203971 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9e4ab9e-482a-459c-b596-f78f156d906c-home\") pod \"e9e4ab9e-482a-459c-b596-f78f156d906c\" (UID: \"e9e4ab9e-482a-459c-b596-f78f156d906c\") " Sep 16 08:00:00.204367 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.204263 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4543c78d-4a9b-437c-8146-89e1eed8b7a8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4543c78d-4a9b-437c-8146-89e1eed8b7a8" (UID: "4543c78d-4a9b-437c-8146-89e1eed8b7a8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:00:00.204542 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.204495 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e4ab9e-482a-459c-b596-f78f156d906c-home" (OuterVolumeSpecName: "home") pod "e9e4ab9e-482a-459c-b596-f78f156d906c" (UID: "e9e4ab9e-482a-459c-b596-f78f156d906c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:00:00.204854 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.204548 2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4543c78d-4a9b-437c-8146-89e1eed8b7a8-tmp-dir\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:00:00.205306 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.205286 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c1f9e7-d353-4a41-8336-b87728ac9838-model-cache" (OuterVolumeSpecName: "model-cache") pod "30c1f9e7-d353-4a41-8336-b87728ac9838" (UID: "30c1f9e7-d353-4a41-8336-b87728ac9838"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:00:00.207639 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.207585 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e4ab9e-482a-459c-b596-f78f156d906c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e9e4ab9e-482a-459c-b596-f78f156d906c" (UID: "e9e4ab9e-482a-459c-b596-f78f156d906c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:00:00.207749 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.207681 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c1f9e7-d353-4a41-8336-b87728ac9838-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "30c1f9e7-d353-4a41-8336-b87728ac9838" (UID: "30c1f9e7-d353-4a41-8336-b87728ac9838"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:00:00.207749 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.207719 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30c1f9e7-d353-4a41-8336-b87728ac9838-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "30c1f9e7-d353-4a41-8336-b87728ac9838" (UID: "30c1f9e7-d353-4a41-8336-b87728ac9838"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 08:00:00.207870 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.207732 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4543c78d-4a9b-437c-8146-89e1eed8b7a8-kube-api-access-l68g7" (OuterVolumeSpecName: "kube-api-access-l68g7") pod "4543c78d-4a9b-437c-8146-89e1eed8b7a8" (UID: "4543c78d-4a9b-437c-8146-89e1eed8b7a8"). InnerVolumeSpecName "kube-api-access-l68g7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 08:00:00.208035 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.207880 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e4ab9e-482a-459c-b596-f78f156d906c-model-cache" (OuterVolumeSpecName: "model-cache") pod "e9e4ab9e-482a-459c-b596-f78f156d906c" (UID: "e9e4ab9e-482a-459c-b596-f78f156d906c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:00:00.208147 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.208121 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4543c78d-4a9b-437c-8146-89e1eed8b7a8-model-cache" (OuterVolumeSpecName: "model-cache") pod "4543c78d-4a9b-437c-8146-89e1eed8b7a8" (UID: "4543c78d-4a9b-437c-8146-89e1eed8b7a8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:00:00.208281 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.208258 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4543c78d-4a9b-437c-8146-89e1eed8b7a8-home" (OuterVolumeSpecName: "home") pod "4543c78d-4a9b-437c-8146-89e1eed8b7a8" (UID: "4543c78d-4a9b-437c-8146-89e1eed8b7a8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:00:00.214474 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.208624 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c1f9e7-d353-4a41-8336-b87728ac9838-home" (OuterVolumeSpecName: "home") pod "30c1f9e7-d353-4a41-8336-b87728ac9838" (UID: "30c1f9e7-d353-4a41-8336-b87728ac9838"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:00:00.214474 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.212665 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9e4ab9e-482a-459c-b596-f78f156d906c-dshm" (OuterVolumeSpecName: "dshm") pod "e9e4ab9e-482a-459c-b596-f78f156d906c" (UID: "e9e4ab9e-482a-459c-b596-f78f156d906c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:00:00.214474 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.213057 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9e4ab9e-482a-459c-b596-f78f156d906c-kube-api-access-jgmjv" (OuterVolumeSpecName: "kube-api-access-jgmjv") pod "e9e4ab9e-482a-459c-b596-f78f156d906c" (UID: "e9e4ab9e-482a-459c-b596-f78f156d906c"). InnerVolumeSpecName "kube-api-access-jgmjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 08:00:00.214474 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.213203 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30c1f9e7-d353-4a41-8336-b87728ac9838-dshm" (OuterVolumeSpecName: "dshm") pod "30c1f9e7-d353-4a41-8336-b87728ac9838" (UID: "30c1f9e7-d353-4a41-8336-b87728ac9838"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:00:00.214474 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.213873 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-3a5b4c-29825760-fbgc7" Sep 16 08:00:00.215631 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.215605 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4543c78d-4a9b-437c-8146-89e1eed8b7a8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4543c78d-4a9b-437c-8146-89e1eed8b7a8" (UID: "4543c78d-4a9b-437c-8146-89e1eed8b7a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 08:00:00.215731 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.215711 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9e4ab9e-482a-459c-b596-f78f156d906c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e9e4ab9e-482a-459c-b596-f78f156d906c" (UID: "e9e4ab9e-482a-459c-b596-f78f156d906c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 08:00:00.216035 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.215985 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4543c78d-4a9b-437c-8146-89e1eed8b7a8-dshm" (OuterVolumeSpecName: "dshm") pod "4543c78d-4a9b-437c-8146-89e1eed8b7a8" (UID: "4543c78d-4a9b-437c-8146-89e1eed8b7a8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:00:00.216145 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.216078 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30c1f9e7-d353-4a41-8336-b87728ac9838-kube-api-access-7x5jz" (OuterVolumeSpecName: "kube-api-access-7x5jz") pod "30c1f9e7-d353-4a41-8336-b87728ac9838" (UID: "30c1f9e7-d353-4a41-8336-b87728ac9838"). InnerVolumeSpecName "kube-api-access-7x5jz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 08:00:00.224408 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.224376 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3616cc-29825760-b8992" Sep 16 08:00:00.307050 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.305974 2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4543c78d-4a9b-437c-8146-89e1eed8b7a8-model-cache\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:00:00.307050 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.306008 2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30c1f9e7-d353-4a41-8336-b87728ac9838-home\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:00:00.307050 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.306020 2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30c1f9e7-d353-4a41-8336-b87728ac9838-tls-certs\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:00:00.307050 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.306033 2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30c1f9e7-d353-4a41-8336-b87728ac9838-model-cache\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:00:00.307050 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.306045 2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4543c78d-4a9b-437c-8146-89e1eed8b7a8-tls-certs\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:00:00.307050 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.306056 2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30c1f9e7-d353-4a41-8336-b87728ac9838-tmp-dir\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:00:00.307050 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.306075 2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9e4ab9e-482a-459c-b596-f78f156d906c-model-cache\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:00:00.307050 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.306108 2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9e4ab9e-482a-459c-b596-f78f156d906c-home\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:00:00.307050 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.306119 2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30c1f9e7-d353-4a41-8336-b87728ac9838-dshm\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:00:00.307050 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.306127 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgmjv\" (UniqueName: \"kubernetes.io/projected/e9e4ab9e-482a-459c-b596-f78f156d906c-kube-api-access-jgmjv\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:00:00.307050 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.306136 2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9e4ab9e-482a-459c-b596-f78f156d906c-tls-certs\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:00:00.307050 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.306146 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l68g7\" (UniqueName: \"kubernetes.io/projected/4543c78d-4a9b-437c-8146-89e1eed8b7a8-kube-api-access-l68g7\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:00:00.307050 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.306160 2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9e4ab9e-482a-459c-b596-f78f156d906c-tmp-dir\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:00:00.307050 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.306173 2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9e4ab9e-482a-459c-b596-f78f156d906c-dshm\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:00:00.307050 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.306181 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x5jz\" (UniqueName: \"kubernetes.io/projected/30c1f9e7-d353-4a41-8336-b87728ac9838-kube-api-access-7x5jz\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:00:00.307050 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.306189 2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4543c78d-4a9b-437c-8146-89e1eed8b7a8-home\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:00:00.307050 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.306197 2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4543c78d-4a9b-437c-8146-89e1eed8b7a8-dshm\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:00:00.366796 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.366758 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:00.488423 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.488388 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fc27a5-29825760-2ktwd"] Sep 16 08:00:00.490638 ip-10-0-142-170 kubenswrapper[2552]: W0916 08:00:00.490611 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15d8c3e8_444d_4210_888e_1f1f81d415e0.slice/crio-112fbaccac663eea71fd66d72fc3a018e14e35c446c078bfacb884d02ca7879a WatchSource:0}: Error finding container 112fbaccac663eea71fd66d72fc3a018e14e35c446c078bfacb884d02ca7879a: Status 404 returned error can't find the container with id 112fbaccac663eea71fd66d72fc3a018e14e35c446c078bfacb884d02ca7879a Sep 16 08:00:00.515605 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.515577 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-3a5b4c-29825760-fbgc7"] Sep 16 08:00:00.520240 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.519480 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-984c1a-29825760-v69p5"] Sep 16 08:00:00.527286 ip-10-0-142-170 kubenswrapper[2552]: W0916 08:00:00.527253 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e2b9887_a95e_41dd_96c6_c1bae3704375.slice/crio-60ceeb26e0977038773d2adc2d8a7ff9e7e4ba8629e27a8a07c2159138abd5c6 WatchSource:0}: Error finding container 60ceeb26e0977038773d2adc2d8a7ff9e7e4ba8629e27a8a07c2159138abd5c6: Status 404 returned error can't find the container with id 60ceeb26e0977038773d2adc2d8a7ff9e7e4ba8629e27a8a07c2159138abd5c6 Sep 16 08:00:00.527468 ip-10-0-142-170 kubenswrapper[2552]: W0916 08:00:00.527445 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod229f48c6_1ae0_4a77_8631_96364e247a44.slice/crio-a2be6080deabb2300947dfaf97abb73d0415828a76191a7e3f1829eabc819083 WatchSource:0}: Error finding container a2be6080deabb2300947dfaf97abb73d0415828a76191a7e3f1829eabc819083: Status 404 returned error can't find the container with id a2be6080deabb2300947dfaf97abb73d0415828a76191a7e3f1829eabc819083 Sep 16 08:00:00.540604 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.540579 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3616cc-29825760-b8992"] Sep 16 08:00:00.543281 ip-10-0-142-170 kubenswrapper[2552]: W0916 08:00:00.543253 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc60d2a89_4198_4a2e_b0e3_2b5edb5f7ab5.slice/crio-6ed8a9e26576675e4bedf37a41a5095dcc0f87bd9b09ff184792dbc429789306 WatchSource:0}: Error finding container 6ed8a9e26576675e4bedf37a41a5095dcc0f87bd9b09ff184792dbc429789306: Status 404 returned error can't find the container with id 6ed8a9e26576675e4bedf37a41a5095dcc0f87bd9b09ff184792dbc429789306 Sep 16 08:00:00.807824 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.807790 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s7n7g" Sep 16 08:00:00.874766 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.874731 2552 generic.go:356] "Generic (PLEG): container finished" podID="30c1f9e7-d353-4a41-8336-b87728ac9838" containerID="fae213868325c5f8c125bd8fe94b9dc7a8b2bbb3150911061db01fcf4ab22128" exitCode=0 Sep 16 08:00:00.874966 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.874816 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c9020d/simulator-w0-c9020d-kserve-5946f6c555-55w4c" Sep 16 08:00:00.874966 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.874820 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c9020d/simulator-w0-c9020d-kserve-5946f6c555-55w4c" event={"ID":"30c1f9e7-d353-4a41-8336-b87728ac9838","Type":"ContainerDied","Data":"fae213868325c5f8c125bd8fe94b9dc7a8b2bbb3150911061db01fcf4ab22128"} Sep 16 08:00:00.874966 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.874874 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c9020d/simulator-w0-c9020d-kserve-5946f6c555-55w4c" event={"ID":"30c1f9e7-d353-4a41-8336-b87728ac9838","Type":"ContainerDied","Data":"2f914d10412c7e96b30a69bf32123785a2c3efa48d8c98530ca304910673aa20"} Sep 16 08:00:00.874966 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.874901 2552 scope.go:117] "RemoveContainer" containerID="fae213868325c5f8c125bd8fe94b9dc7a8b2bbb3150911061db01fcf4ab22128" Sep 16 08:00:00.877187 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.877147 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-3a5b4c-29825760-fbgc7" event={"ID":"9e2b9887-a95e-41dd-96c6-c1bae3704375","Type":"ContainerStarted","Data":"60ceeb26e0977038773d2adc2d8a7ff9e7e4ba8629e27a8a07c2159138abd5c6"} Sep 16 08:00:00.879282 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.879139 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-984c1a-29825760-v69p5" event={"ID":"229f48c6-1ae0-4a77-8631-96364e247a44","Type":"ContainerStarted","Data":"40c57cd1272186ad3b83cb1fe3b51e92a1924bb31d8f16989c3c94dc8019f5b8"} Sep 16 08:00:00.879282 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.879194 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-984c1a-29825760-v69p5" event={"ID":"229f48c6-1ae0-4a77-8631-96364e247a44","Type":"ContainerStarted","Data":"a2be6080deabb2300947dfaf97abb73d0415828a76191a7e3f1829eabc819083"} Sep 16 08:00:00.880953 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.880926 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fc27a5-29825760-2ktwd" event={"ID":"15d8c3e8-444d-4210-888e-1f1f81d415e0","Type":"ContainerStarted","Data":"fa12e4d70b9fcba336784f618f2ee6f8ce119cf150243401aea8dc99b86b129e"} Sep 16 08:00:00.881083 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.880976 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fc27a5-29825760-2ktwd" event={"ID":"15d8c3e8-444d-4210-888e-1f1f81d415e0","Type":"ContainerStarted","Data":"112fbaccac663eea71fd66d72fc3a018e14e35c446c078bfacb884d02ca7879a"} Sep 16 08:00:00.883394 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.883365 2552 generic.go:356] "Generic (PLEG): container finished" podID="4543c78d-4a9b-437c-8146-89e1eed8b7a8" containerID="3fddfd579122ffe1f47b71943c34bd8cbc2300b5872d500cc9df0daa204f45b5" exitCode=0 Sep 16 08:00:00.883506 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.883449 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c9020d/premium-w0-c9020d-kserve-78578c546b-8zspq" event={"ID":"4543c78d-4a9b-437c-8146-89e1eed8b7a8","Type":"ContainerDied","Data":"3fddfd579122ffe1f47b71943c34bd8cbc2300b5872d500cc9df0daa204f45b5"} Sep 16 08:00:00.883506 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.883479 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c9020d/premium-w0-c9020d-kserve-78578c546b-8zspq" event={"ID":"4543c78d-4a9b-437c-8146-89e1eed8b7a8","Type":"ContainerDied","Data":"e055ea8afde0b6c9ca3ec6a56f33f0a93b06781f4a5d1969fc1b210ce458656e"} Sep 16 08:00:00.883625 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.883561 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c9020d/premium-w0-c9020d-kserve-78578c546b-8zspq" Sep 16 08:00:00.889367 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.889330 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3616cc-29825760-b8992" event={"ID":"c60d2a89-4198-4a2e-b0e3-2b5edb5f7ab5","Type":"ContainerStarted","Data":"6ed8a9e26576675e4bedf37a41a5095dcc0f87bd9b09ff184792dbc429789306"} Sep 16 08:00:00.890559 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.890537 2552 scope.go:117] "RemoveContainer" containerID="fae213868325c5f8c125bd8fe94b9dc7a8b2bbb3150911061db01fcf4ab22128" Sep 16 08:00:00.890961 ip-10-0-142-170 kubenswrapper[2552]: E0916 08:00:00.890931 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fae213868325c5f8c125bd8fe94b9dc7a8b2bbb3150911061db01fcf4ab22128\": container with ID starting with fae213868325c5f8c125bd8fe94b9dc7a8b2bbb3150911061db01fcf4ab22128 not found: ID does not exist" containerID="fae213868325c5f8c125bd8fe94b9dc7a8b2bbb3150911061db01fcf4ab22128" Sep 16 08:00:00.891062 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.891031 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fae213868325c5f8c125bd8fe94b9dc7a8b2bbb3150911061db01fcf4ab22128"} err="failed to get container status \"fae213868325c5f8c125bd8fe94b9dc7a8b2bbb3150911061db01fcf4ab22128\": rpc error: code = NotFound desc = could not find container \"fae213868325c5f8c125bd8fe94b9dc7a8b2bbb3150911061db01fcf4ab22128\": container with ID starting with fae213868325c5f8c125bd8fe94b9dc7a8b2bbb3150911061db01fcf4ab22128 not found: ID does not exist" Sep 16 08:00:00.891062 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.891057 2552 scope.go:117] "RemoveContainer" containerID="3fddfd579122ffe1f47b71943c34bd8cbc2300b5872d500cc9df0daa204f45b5" Sep 16 08:00:00.891214 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.891191 2552 generic.go:356] "Generic (PLEG): container finished" podID="e9e4ab9e-482a-459c-b596-f78f156d906c" containerID="2bf27c6358eaf2ca4bacc00c9f203c077b8dd7faf0b69627784c254036e78e2b" exitCode=0 Sep 16 08:00:00.891248 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.891234 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c9020d/distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn" event={"ID":"e9e4ab9e-482a-459c-b596-f78f156d906c","Type":"ContainerDied","Data":"2bf27c6358eaf2ca4bacc00c9f203c077b8dd7faf0b69627784c254036e78e2b"} Sep 16 08:00:00.891303 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.891264 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c9020d/distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn" event={"ID":"e9e4ab9e-482a-459c-b596-f78f156d906c","Type":"ContainerDied","Data":"58fb6dae1a2357408dfee5fabc6d4d77c8c2bf0ff9d88414163754e1cd8602ab"} Sep 16 08:00:00.891342 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.891304 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c9020d/distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn" Sep 16 08:00:00.898969 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.898614 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-984c1a-29825760-v69p5" podStartSLOduration=0.898593187 podStartE2EDuration="898.593187ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 08:00:00.8982281 +0000 UTC m=+1572.482006439" watchObservedRunningTime="2026-09-16 08:00:00.898593187 +0000 UTC m=+1572.482371541" Sep 16 08:00:00.903445 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.903408 2552 scope.go:117] "RemoveContainer" containerID="3fddfd579122ffe1f47b71943c34bd8cbc2300b5872d500cc9df0daa204f45b5" Sep 16 08:00:00.903939 ip-10-0-142-170 kubenswrapper[2552]: E0916 08:00:00.903906 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fddfd579122ffe1f47b71943c34bd8cbc2300b5872d500cc9df0daa204f45b5\": container with ID starting with 3fddfd579122ffe1f47b71943c34bd8cbc2300b5872d500cc9df0daa204f45b5 not found: ID does not exist" containerID="3fddfd579122ffe1f47b71943c34bd8cbc2300b5872d500cc9df0daa204f45b5" Sep 16 08:00:00.904065 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.903944 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fddfd579122ffe1f47b71943c34bd8cbc2300b5872d500cc9df0daa204f45b5"} err="failed to get container status \"3fddfd579122ffe1f47b71943c34bd8cbc2300b5872d500cc9df0daa204f45b5\": rpc error: code = NotFound desc = could not find container \"3fddfd579122ffe1f47b71943c34bd8cbc2300b5872d500cc9df0daa204f45b5\": container with ID starting with 3fddfd579122ffe1f47b71943c34bd8cbc2300b5872d500cc9df0daa204f45b5 not found: ID does not exist" Sep 16 08:00:00.904065 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.903964 2552 scope.go:117] "RemoveContainer" containerID="2bf27c6358eaf2ca4bacc00c9f203c077b8dd7faf0b69627784c254036e78e2b" Sep 16 08:00:00.913670 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.913542 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fc27a5-29825760-2ktwd" podStartSLOduration=0.913519886 podStartE2EDuration="913.519886ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 08:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 08:00:00.913082891 +0000 UTC m=+1572.496861230" watchObservedRunningTime="2026-09-16 08:00:00.913519886 +0000 UTC m=+1572.497298225" Sep 16 08:00:00.913839 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.913676 2552 scope.go:117] "RemoveContainer" containerID="2bf27c6358eaf2ca4bacc00c9f203c077b8dd7faf0b69627784c254036e78e2b" Sep 16 08:00:00.914136 ip-10-0-142-170 kubenswrapper[2552]: E0916 08:00:00.914089 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bf27c6358eaf2ca4bacc00c9f203c077b8dd7faf0b69627784c254036e78e2b\": container with ID starting with 2bf27c6358eaf2ca4bacc00c9f203c077b8dd7faf0b69627784c254036e78e2b not found: ID does not exist" containerID="2bf27c6358eaf2ca4bacc00c9f203c077b8dd7faf0b69627784c254036e78e2b" Sep 16 08:00:00.914222 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.914143 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bf27c6358eaf2ca4bacc00c9f203c077b8dd7faf0b69627784c254036e78e2b"} err="failed to get container status \"2bf27c6358eaf2ca4bacc00c9f203c077b8dd7faf0b69627784c254036e78e2b\": rpc error: code = NotFound desc = could not find container \"2bf27c6358eaf2ca4bacc00c9f203c077b8dd7faf0b69627784c254036e78e2b\": container with ID starting with 2bf27c6358eaf2ca4bacc00c9f203c077b8dd7faf0b69627784c254036e78e2b not found: ID does not exist" Sep 16 08:00:00.940358 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.940327 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c9020d/premium-w0-c9020d-kserve-78578c546b-8zspq"] Sep 16 08:00:00.940358 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.940365 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-c9020d/premium-w0-c9020d-kserve-78578c546b-8zspq"] Sep 16 08:00:00.954552 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.954525 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c9020d/simulator-w0-c9020d-kserve-5946f6c555-55w4c"] Sep 16 08:00:00.960462 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.960428 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-c9020d/simulator-w0-c9020d-kserve-5946f6c555-55w4c"] Sep 16 08:00:00.975689 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.975652 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c9020d/distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn"] Sep 16 08:00:00.977868 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:00.977836 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-c9020d/distinct-2-w0-c9020d-kserve-64f6c9fb4b-h9wsn"] Sep 16 08:00:01.187862 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:01.187824 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:01.901208 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:01.901171 2552 generic.go:356] "Generic (PLEG): container finished" podID="9e2b9887-a95e-41dd-96c6-c1bae3704375" containerID="b2e6255bde8ad7a14ae9d094ed5e73dedb804df196bfb88b95a08ba6bfc25cf4" exitCode=0 Sep 16 08:00:01.901420 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:01.901294 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-3a5b4c-29825760-fbgc7" event={"ID":"9e2b9887-a95e-41dd-96c6-c1bae3704375","Type":"ContainerDied","Data":"b2e6255bde8ad7a14ae9d094ed5e73dedb804df196bfb88b95a08ba6bfc25cf4"} Sep 16 08:00:01.902901 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:01.902876 2552 generic.go:356] "Generic (PLEG): container finished" podID="229f48c6-1ae0-4a77-8631-96364e247a44" containerID="40c57cd1272186ad3b83cb1fe3b51e92a1924bb31d8f16989c3c94dc8019f5b8" exitCode=0 Sep 16 08:00:01.903025 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:01.902946 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-984c1a-29825760-v69p5" event={"ID":"229f48c6-1ae0-4a77-8631-96364e247a44","Type":"ContainerDied","Data":"40c57cd1272186ad3b83cb1fe3b51e92a1924bb31d8f16989c3c94dc8019f5b8"} Sep 16 08:00:01.904376 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:01.904353 2552 generic.go:356] "Generic (PLEG): container finished" podID="15d8c3e8-444d-4210-888e-1f1f81d415e0" containerID="fa12e4d70b9fcba336784f618f2ee6f8ce119cf150243401aea8dc99b86b129e" exitCode=0 Sep 16 08:00:01.904483 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:01.904433 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fc27a5-29825760-2ktwd" event={"ID":"15d8c3e8-444d-4210-888e-1f1f81d415e0","Type":"ContainerDied","Data":"fa12e4d70b9fcba336784f618f2ee6f8ce119cf150243401aea8dc99b86b129e"} Sep 16 08:00:01.907039 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:01.907015 2552 generic.go:356] "Generic (PLEG): container finished" podID="c60d2a89-4198-4a2e-b0e3-2b5edb5f7ab5" containerID="977cf3b5b71558487355e3e2dfec1e0b374ba4fd51e2c7e6f311cc1cf3ba9b99" exitCode=0 Sep 16 08:00:01.907171 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:01.907124 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3616cc-29825760-b8992" event={"ID":"c60d2a89-4198-4a2e-b0e3-2b5edb5f7ab5","Type":"ContainerDied","Data":"977cf3b5b71558487355e3e2dfec1e0b374ba4fd51e2c7e6f311cc1cf3ba9b99"} Sep 16 08:00:02.312089 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:02.312053 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:02.943976 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:02.943933 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30c1f9e7-d353-4a41-8336-b87728ac9838" path="/var/lib/kubelet/pods/30c1f9e7-d353-4a41-8336-b87728ac9838/volumes" Sep 16 08:00:02.944641 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:02.944617 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4543c78d-4a9b-437c-8146-89e1eed8b7a8" path="/var/lib/kubelet/pods/4543c78d-4a9b-437c-8146-89e1eed8b7a8/volumes" Sep 16 08:00:02.945854 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:02.945169 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9e4ab9e-482a-459c-b596-f78f156d906c" path="/var/lib/kubelet/pods/e9e4ab9e-482a-459c-b596-f78f156d906c/volumes" Sep 16 08:00:02.996974 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:02.996903 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:03.082507 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:03.082471 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-984c1a-29825760-v69p5" Sep 16 08:00:03.121264 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:03.121232 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fc27a5-29825760-2ktwd" Sep 16 08:00:03.169072 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:03.169046 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3616cc-29825760-b8992" Sep 16 08:00:03.174344 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:03.174317 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-3a5b4c-29825760-fbgc7" Sep 16 08:00:03.900749 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:03.900713 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:03.922002 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:03.921970 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-3a5b4c-29825760-fbgc7" Sep 16 08:00:03.922218 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:03.922001 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-3a5b4c-29825760-fbgc7" event={"ID":"9e2b9887-a95e-41dd-96c6-c1bae3704375","Type":"ContainerDied","Data":"60ceeb26e0977038773d2adc2d8a7ff9e7e4ba8629e27a8a07c2159138abd5c6"} Sep 16 08:00:03.922218 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:03.922049 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60ceeb26e0977038773d2adc2d8a7ff9e7e4ba8629e27a8a07c2159138abd5c6" Sep 16 08:00:03.923736 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:03.923712 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-984c1a-29825760-v69p5" Sep 16 08:00:03.923736 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:03.923724 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-984c1a-29825760-v69p5" event={"ID":"229f48c6-1ae0-4a77-8631-96364e247a44","Type":"ContainerDied","Data":"a2be6080deabb2300947dfaf97abb73d0415828a76191a7e3f1829eabc819083"} Sep 16 08:00:03.923916 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:03.923785 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2be6080deabb2300947dfaf97abb73d0415828a76191a7e3f1829eabc819083" Sep 16 08:00:03.925409 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:03.925375 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fc27a5-29825760-2ktwd" Sep 16 08:00:03.925534 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:03.925379 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fc27a5-29825760-2ktwd" event={"ID":"15d8c3e8-444d-4210-888e-1f1f81d415e0","Type":"ContainerDied","Data":"112fbaccac663eea71fd66d72fc3a018e14e35c446c078bfacb884d02ca7879a"} Sep 16 08:00:03.925534 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:03.925490 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="112fbaccac663eea71fd66d72fc3a018e14e35c446c078bfacb884d02ca7879a" Sep 16 08:00:03.927637 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:03.927610 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3616cc-29825760-b8992" event={"ID":"c60d2a89-4198-4a2e-b0e3-2b5edb5f7ab5","Type":"ContainerDied","Data":"6ed8a9e26576675e4bedf37a41a5095dcc0f87bd9b09ff184792dbc429789306"} Sep 16 08:00:03.927637 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:03.927632 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3616cc-29825760-b8992" Sep 16 08:00:03.927807 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:03.927652 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ed8a9e26576675e4bedf37a41a5095dcc0f87bd9b09ff184792dbc429789306" Sep 16 08:00:04.516909 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:04.516871 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:04.983622 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:04.983588 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:05.094826 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:05.094723 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:06.015458 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:06.015426 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:06.109319 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:06.109282 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:06.176920 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:06.176874 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:06.454338 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:06.454301 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3a5b4c-2c47t"] Sep 16 08:00:06.454756 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:06.454744 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c60d2a89-4198-4a2e-b0e3-2b5edb5f7ab5" containerName="cleanup" Sep 16 08:00:06.454817 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:06.454758 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="c60d2a89-4198-4a2e-b0e3-2b5edb5f7ab5" containerName="cleanup" Sep 16 08:00:06.454817 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:06.454773 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e2b9887-a95e-41dd-96c6-c1bae3704375" containerName="cleanup" Sep 16 08:00:06.454817 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:06.454779 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e2b9887-a95e-41dd-96c6-c1bae3704375" containerName="cleanup" Sep 16 08:00:06.454817 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:06.454789 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15d8c3e8-444d-4210-888e-1f1f81d415e0" containerName="cleanup" Sep 16 08:00:06.454817 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:06.454794 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="15d8c3e8-444d-4210-888e-1f1f81d415e0" containerName="cleanup" Sep 16 08:00:06.454817 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:06.454807 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="229f48c6-1ae0-4a77-8631-96364e247a44" containerName="cleanup" Sep 16 08:00:06.454817 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:06.454813 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="229f48c6-1ae0-4a77-8631-96364e247a44" containerName="cleanup" Sep 16 08:00:06.455019 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:06.454883 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="229f48c6-1ae0-4a77-8631-96364e247a44" containerName="cleanup" Sep 16 08:00:06.455019 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:06.454890 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="15d8c3e8-444d-4210-888e-1f1f81d415e0" containerName="cleanup" Sep 16 08:00:06.455019 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:06.454896 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="c60d2a89-4198-4a2e-b0e3-2b5edb5f7ab5" containerName="cleanup" Sep 16 08:00:06.455019 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:06.454904 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e2b9887-a95e-41dd-96c6-c1bae3704375" containerName="cleanup" Sep 16 08:00:06.458152 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:06.458126 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3a5b4c-2c47t" Sep 16 08:00:06.460995 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:06.460962 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 08:00:06.461162 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:06.461085 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zwk5r\"" Sep 16 08:00:06.463320 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:06.463292 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3a5b4c-2c47t"] Sep 16 08:00:06.573250 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:06.573207 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f657de3d-28ae-423e-a1e5-e2f1e86ddd5d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3a5b4c-2c47t\" (UID: \"f657de3d-28ae-423e-a1e5-e2f1e86ddd5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3a5b4c-2c47t" Sep 16 08:00:06.674553 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:06.674511 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f657de3d-28ae-423e-a1e5-e2f1e86ddd5d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3a5b4c-2c47t\" (UID: \"f657de3d-28ae-423e-a1e5-e2f1e86ddd5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3a5b4c-2c47t" Sep 16 08:00:06.675224 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:06.675200 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f657de3d-28ae-423e-a1e5-e2f1e86ddd5d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-3a5b4c-2c47t\" (UID: \"f657de3d-28ae-423e-a1e5-e2f1e86ddd5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3a5b4c-2c47t" Sep 16 08:00:06.772557 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:06.772462 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3a5b4c-2c47t" Sep 16 08:00:07.049531 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:07.049499 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3a5b4c-2c47t"] Sep 16 08:00:07.053151 ip-10-0-142-170 kubenswrapper[2552]: W0916 08:00:07.053089 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf657de3d_28ae_423e_a1e5_e2f1e86ddd5d.slice/crio-62011b200207ae12d94c488f75044987da111891725f6c2e1d993b0e25d7765d WatchSource:0}: Error finding container 62011b200207ae12d94c488f75044987da111891725f6c2e1d993b0e25d7765d: Status 404 returned error can't find the container with id 62011b200207ae12d94c488f75044987da111891725f6c2e1d993b0e25d7765d Sep 16 08:00:07.294731 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:07.294690 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:07.952850 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:07.952805 2552 generic.go:356] "Generic (PLEG): container finished" podID="f657de3d-28ae-423e-a1e5-e2f1e86ddd5d" containerID="123893790cdcd6a287121c784ff5e7e5d0165d88a8c72623dc69d0f190aad4c1" exitCode=0 Sep 16 08:00:07.953053 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:07.952890 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3a5b4c-2c47t" event={"ID":"f657de3d-28ae-423e-a1e5-e2f1e86ddd5d","Type":"ContainerDied","Data":"123893790cdcd6a287121c784ff5e7e5d0165d88a8c72623dc69d0f190aad4c1"} Sep 16 08:00:07.953053 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:07.952971 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3a5b4c-2c47t" event={"ID":"f657de3d-28ae-423e-a1e5-e2f1e86ddd5d","Type":"ContainerStarted","Data":"62011b200207ae12d94c488f75044987da111891725f6c2e1d993b0e25d7765d"} Sep 16 08:00:08.141232 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:08.141147 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:08.779734 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:08.779697 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:09.103162 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:09.103132 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3a5b4c-2c47t" Sep 16 08:00:09.201027 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:09.200989 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f657de3d-28ae-423e-a1e5-e2f1e86ddd5d-maas-api-service-ca\") pod \"f657de3d-28ae-423e-a1e5-e2f1e86ddd5d\" (UID: \"f657de3d-28ae-423e-a1e5-e2f1e86ddd5d\") " Sep 16 08:00:09.201448 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:09.201397 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f657de3d-28ae-423e-a1e5-e2f1e86ddd5d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f657de3d-28ae-423e-a1e5-e2f1e86ddd5d" (UID: "f657de3d-28ae-423e-a1e5-e2f1e86ddd5d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 08:00:09.302178 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:09.302144 2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f657de3d-28ae-423e-a1e5-e2f1e86ddd5d-maas-api-service-ca\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:00:09.537762 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:09.537721 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:09.968012 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:09.967977 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3a5b4c-2c47t" Sep 16 08:00:09.968212 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:09.967970 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3a5b4c-2c47t" event={"ID":"f657de3d-28ae-423e-a1e5-e2f1e86ddd5d","Type":"ContainerDied","Data":"62011b200207ae12d94c488f75044987da111891725f6c2e1d993b0e25d7765d"} Sep 16 08:00:09.968212 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:09.968125 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="62011b200207ae12d94c488f75044987da111891725f6c2e1d993b0e25d7765d" Sep 16 08:00:10.181218 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:10.181160 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:10.712037 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:10.711992 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:10.862321 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:10.862290 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:11.674556 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:11.674510 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:21.910040 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:21.909985 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:21.911018 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:21.910974 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-3a5b4c-29825760-fbgc7"] Sep 16 08:00:21.913390 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:21.913364 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-3a5b4c-29825760-fbgc7"] Sep 16 08:00:21.934235 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:21.934198 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3a5b4c-df989476f-np6nd"] Sep 16 08:00:21.934546 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:21.934450 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-3a5b4c-df989476f-np6nd" podUID="945aff56-7bbe-4bf6-a1d0-a5edcaf6f355" containerName="payload-processing" containerID="cri-o://193bb59c3d2f702bcdb57ca0b2decbad93c2064c8a45297331f78374696bd018" gracePeriod=30 Sep 16 08:00:21.951185 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:21.945535 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3a5b4c-765cd4576b-hwd2c"] Sep 16 08:00:21.951185 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:21.945899 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3a5b4c-765cd4576b-hwd2c" podUID="f3aca058-bc0c-4725-ba3f-e368b71ba834" containerName="payload-pre-processing" containerID="cri-o://a74e1b43feaff8ed073769a2e3f8799bb4a64ec477a70a0529dbe5b0b1b3b94c" gracePeriod=30 Sep 16 08:00:21.986614 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:21.986578 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:22.057883 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:22.057839 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:22.106693 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:22.106656 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3a5b4c-2c47t"] Sep 16 08:00:22.110166 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:22.110115 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-3a5b4c-2c47t"] Sep 16 08:00:22.120772 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:22.120706 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:22.230642 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:22.230609 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3a5b4c-df989476f-np6nd" Sep 16 08:00:22.257823 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:22.257794 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3a5b4c-765cd4576b-hwd2c" Sep 16 08:00:22.338219 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:22.338181 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5s44d\" (UniqueName: \"kubernetes.io/projected/945aff56-7bbe-4bf6-a1d0-a5edcaf6f355-kube-api-access-5s44d\") pod \"945aff56-7bbe-4bf6-a1d0-a5edcaf6f355\" (UID: \"945aff56-7bbe-4bf6-a1d0-a5edcaf6f355\") " Sep 16 08:00:22.338219 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:22.338226 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3aca058-bc0c-4725-ba3f-e368b71ba834-plugins-config-volume\") pod \"f3aca058-bc0c-4725-ba3f-e368b71ba834\" (UID: \"f3aca058-bc0c-4725-ba3f-e368b71ba834\") " Sep 16 08:00:22.338435 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:22.338303 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/945aff56-7bbe-4bf6-a1d0-a5edcaf6f355-plugins-config-volume\") pod \"945aff56-7bbe-4bf6-a1d0-a5edcaf6f355\" (UID: \"945aff56-7bbe-4bf6-a1d0-a5edcaf6f355\") " Sep 16 08:00:22.338435 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:22.338329 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqqnr\" (UniqueName: \"kubernetes.io/projected/f3aca058-bc0c-4725-ba3f-e368b71ba834-kube-api-access-lqqnr\") pod \"f3aca058-bc0c-4725-ba3f-e368b71ba834\" (UID: \"f3aca058-bc0c-4725-ba3f-e368b71ba834\") " Sep 16 08:00:22.338704 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:22.338669 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/945aff56-7bbe-4bf6-a1d0-a5edcaf6f355-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "945aff56-7bbe-4bf6-a1d0-a5edcaf6f355" (UID: "945aff56-7bbe-4bf6-a1d0-a5edcaf6f355"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 08:00:22.338844 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:22.338720 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3aca058-bc0c-4725-ba3f-e368b71ba834-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f3aca058-bc0c-4725-ba3f-e368b71ba834" (UID: "f3aca058-bc0c-4725-ba3f-e368b71ba834"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 08:00:22.340461 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:22.340434 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3aca058-bc0c-4725-ba3f-e368b71ba834-kube-api-access-lqqnr" (OuterVolumeSpecName: "kube-api-access-lqqnr") pod "f3aca058-bc0c-4725-ba3f-e368b71ba834" (UID: "f3aca058-bc0c-4725-ba3f-e368b71ba834"). InnerVolumeSpecName "kube-api-access-lqqnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 08:00:22.340461 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:22.340459 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/945aff56-7bbe-4bf6-a1d0-a5edcaf6f355-kube-api-access-5s44d" (OuterVolumeSpecName: "kube-api-access-5s44d") pod "945aff56-7bbe-4bf6-a1d0-a5edcaf6f355" (UID: "945aff56-7bbe-4bf6-a1d0-a5edcaf6f355"). InnerVolumeSpecName "kube-api-access-5s44d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 08:00:22.439947 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:22.439850 2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/945aff56-7bbe-4bf6-a1d0-a5edcaf6f355-plugins-config-volume\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:00:22.439947 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:22.439883 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqqnr\" (UniqueName: \"kubernetes.io/projected/f3aca058-bc0c-4725-ba3f-e368b71ba834-kube-api-access-lqqnr\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:00:22.439947 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:22.439895 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s44d\" (UniqueName: \"kubernetes.io/projected/945aff56-7bbe-4bf6-a1d0-a5edcaf6f355-kube-api-access-5s44d\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:00:22.439947 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:22.439904 2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3aca058-bc0c-4725-ba3f-e368b71ba834-plugins-config-volume\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:00:22.481305 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:22.481264 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:22.601554 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:22.601514 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:22.938804 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:22.938765 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e2b9887-a95e-41dd-96c6-c1bae3704375" path="/var/lib/kubelet/pods/9e2b9887-a95e-41dd-96c6-c1bae3704375/volumes" Sep 16 08:00:22.939203 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:22.939081 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f657de3d-28ae-423e-a1e5-e2f1e86ddd5d" path="/var/lib/kubelet/pods/f657de3d-28ae-423e-a1e5-e2f1e86ddd5d/volumes" Sep 16 08:00:23.044754 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:23.044712 2552 generic.go:356] "Generic (PLEG): container finished" podID="f3aca058-bc0c-4725-ba3f-e368b71ba834" containerID="a74e1b43feaff8ed073769a2e3f8799bb4a64ec477a70a0529dbe5b0b1b3b94c" exitCode=0 Sep 16 08:00:23.044974 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:23.044769 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3a5b4c-765cd4576b-hwd2c" Sep 16 08:00:23.044974 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:23.044803 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3a5b4c-765cd4576b-hwd2c" event={"ID":"f3aca058-bc0c-4725-ba3f-e368b71ba834","Type":"ContainerDied","Data":"a74e1b43feaff8ed073769a2e3f8799bb4a64ec477a70a0529dbe5b0b1b3b94c"} Sep 16 08:00:23.044974 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:23.044862 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-3a5b4c-765cd4576b-hwd2c" event={"ID":"f3aca058-bc0c-4725-ba3f-e368b71ba834","Type":"ContainerDied","Data":"1c7847227ade92c39ed69aab93c5c0988116fcae10bdf111997fa4020ff9108c"} Sep 16 08:00:23.044974 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:23.044894 2552 scope.go:117] "RemoveContainer" containerID="a74e1b43feaff8ed073769a2e3f8799bb4a64ec477a70a0529dbe5b0b1b3b94c" Sep 16 08:00:23.046895 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:23.046866 2552 generic.go:356] "Generic (PLEG): container finished" podID="945aff56-7bbe-4bf6-a1d0-a5edcaf6f355" containerID="193bb59c3d2f702bcdb57ca0b2decbad93c2064c8a45297331f78374696bd018" exitCode=0 Sep 16 08:00:23.047016 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:23.046952 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3a5b4c-df989476f-np6nd" event={"ID":"945aff56-7bbe-4bf6-a1d0-a5edcaf6f355","Type":"ContainerDied","Data":"193bb59c3d2f702bcdb57ca0b2decbad93c2064c8a45297331f78374696bd018"} Sep 16 08:00:23.047016 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:23.047000 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-3a5b4c-df989476f-np6nd" event={"ID":"945aff56-7bbe-4bf6-a1d0-a5edcaf6f355","Type":"ContainerDied","Data":"69a262c5eecf0fd6c153ffc3892f684fc71cfa816abc9b1e911dba27b004f9bc"} Sep 16 08:00:23.047161 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:23.047043 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-3a5b4c-df989476f-np6nd" Sep 16 08:00:23.055353 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:23.055329 2552 scope.go:117] "RemoveContainer" containerID="a74e1b43feaff8ed073769a2e3f8799bb4a64ec477a70a0529dbe5b0b1b3b94c" Sep 16 08:00:23.055716 ip-10-0-142-170 kubenswrapper[2552]: E0916 08:00:23.055694 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a74e1b43feaff8ed073769a2e3f8799bb4a64ec477a70a0529dbe5b0b1b3b94c\": container with ID starting with a74e1b43feaff8ed073769a2e3f8799bb4a64ec477a70a0529dbe5b0b1b3b94c not found: ID does not exist" containerID="a74e1b43feaff8ed073769a2e3f8799bb4a64ec477a70a0529dbe5b0b1b3b94c" Sep 16 08:00:23.055801 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:23.055737 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a74e1b43feaff8ed073769a2e3f8799bb4a64ec477a70a0529dbe5b0b1b3b94c"} err="failed to get container status \"a74e1b43feaff8ed073769a2e3f8799bb4a64ec477a70a0529dbe5b0b1b3b94c\": rpc error: code = NotFound desc = could not find container \"a74e1b43feaff8ed073769a2e3f8799bb4a64ec477a70a0529dbe5b0b1b3b94c\": container with ID starting with a74e1b43feaff8ed073769a2e3f8799bb4a64ec477a70a0529dbe5b0b1b3b94c not found: ID does not exist" Sep 16 08:00:23.055801 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:23.055755 2552 scope.go:117] "RemoveContainer" containerID="193bb59c3d2f702bcdb57ca0b2decbad93c2064c8a45297331f78374696bd018" Sep 16 08:00:23.062735 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:23.062707 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3a5b4c-df989476f-np6nd"] Sep 16 08:00:23.067901 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:23.067849 2552 scope.go:117] "RemoveContainer" containerID="193bb59c3d2f702bcdb57ca0b2decbad93c2064c8a45297331f78374696bd018" Sep 16 08:00:23.068585 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:23.068556 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-3a5b4c-df989476f-np6nd"] Sep 16 08:00:23.068704 ip-10-0-142-170 kubenswrapper[2552]: E0916 08:00:23.068579 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"193bb59c3d2f702bcdb57ca0b2decbad93c2064c8a45297331f78374696bd018\": container with ID starting with 193bb59c3d2f702bcdb57ca0b2decbad93c2064c8a45297331f78374696bd018 not found: ID does not exist" containerID="193bb59c3d2f702bcdb57ca0b2decbad93c2064c8a45297331f78374696bd018" Sep 16 08:00:23.068704 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:23.068611 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"193bb59c3d2f702bcdb57ca0b2decbad93c2064c8a45297331f78374696bd018"} err="failed to get container status \"193bb59c3d2f702bcdb57ca0b2decbad93c2064c8a45297331f78374696bd018\": rpc error: code = NotFound desc = could not find container \"193bb59c3d2f702bcdb57ca0b2decbad93c2064c8a45297331f78374696bd018\": container with ID starting with 193bb59c3d2f702bcdb57ca0b2decbad93c2064c8a45297331f78374696bd018 not found: ID does not exist" Sep 16 08:00:23.078842 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:23.078806 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3a5b4c-765cd4576b-hwd2c"] Sep 16 08:00:23.080897 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:23.080868 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-3a5b4c-765cd4576b-hwd2c"] Sep 16 08:00:23.474892 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:23.474855 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:24.159272 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:24.159236 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:24.910258 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:24.910221 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:24.938193 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:24.938157 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="945aff56-7bbe-4bf6-a1d0-a5edcaf6f355" path="/var/lib/kubelet/pods/945aff56-7bbe-4bf6-a1d0-a5edcaf6f355/volumes" Sep 16 08:00:24.938567 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:24.938553 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3aca058-bc0c-4725-ba3f-e368b71ba834" path="/var/lib/kubelet/pods/f3aca058-bc0c-4725-ba3f-e368b71ba834/volumes" Sep 16 08:00:25.474954 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:25.474919 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:26.193484 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:26.193449 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:28.403176 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:28.403064 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:28.504547 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:28.504515 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:28.683762 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:28.683701 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:29.023031 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:29.022935 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4a243c-2gxn2"] Sep 16 08:00:29.023365 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:29.023351 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="945aff56-7bbe-4bf6-a1d0-a5edcaf6f355" containerName="payload-processing" Sep 16 08:00:29.023410 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:29.023365 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="945aff56-7bbe-4bf6-a1d0-a5edcaf6f355" containerName="payload-processing" Sep 16 08:00:29.023410 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:29.023384 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f657de3d-28ae-423e-a1e5-e2f1e86ddd5d" containerName="revoke-keys" Sep 16 08:00:29.023410 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:29.023389 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="f657de3d-28ae-423e-a1e5-e2f1e86ddd5d" containerName="revoke-keys" Sep 16 08:00:29.023410 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:29.023405 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3aca058-bc0c-4725-ba3f-e368b71ba834" containerName="payload-pre-processing" Sep 16 08:00:29.023539 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:29.023412 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3aca058-bc0c-4725-ba3f-e368b71ba834" containerName="payload-pre-processing" Sep 16 08:00:29.023539 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:29.023471 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="f657de3d-28ae-423e-a1e5-e2f1e86ddd5d" containerName="revoke-keys" Sep 16 08:00:29.023539 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:29.023480 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3aca058-bc0c-4725-ba3f-e368b71ba834" containerName="payload-pre-processing" Sep 16 08:00:29.023539 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:29.023488 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="945aff56-7bbe-4bf6-a1d0-a5edcaf6f355" containerName="payload-processing" Sep 16 08:00:29.027881 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:29.027860 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4a243c-2gxn2" Sep 16 08:00:29.030449 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:29.030421 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zwk5r\"" Sep 16 08:00:29.030595 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:29.030458 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 08:00:29.034793 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:29.034760 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4a243c-2gxn2"] Sep 16 08:00:29.106446 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:29.106408 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14e1b3ab-35c4-4edf-9933-4f47afe311c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4a243c-2gxn2\" (UID: \"14e1b3ab-35c4-4edf-9933-4f47afe311c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4a243c-2gxn2" Sep 16 08:00:29.207744 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:29.207701 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14e1b3ab-35c4-4edf-9933-4f47afe311c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4a243c-2gxn2\" (UID: \"14e1b3ab-35c4-4edf-9933-4f47afe311c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4a243c-2gxn2" Sep 16 08:00:29.208416 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:29.208397 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14e1b3ab-35c4-4edf-9933-4f47afe311c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4a243c-2gxn2\" (UID: \"14e1b3ab-35c4-4edf-9933-4f47afe311c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4a243c-2gxn2" Sep 16 08:00:29.285557 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:29.285445 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:29.340813 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:29.340771 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4a243c-2gxn2" Sep 16 08:00:29.616896 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:29.616815 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4a243c-2gxn2"] Sep 16 08:00:29.620557 ip-10-0-142-170 kubenswrapper[2552]: W0916 08:00:29.620525 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14e1b3ab_35c4_4edf_9933_4f47afe311c3.slice/crio-2ec3f19772447b85796b47970b53c8f611130ff5f2e903e3d16d75298d7cd230 WatchSource:0}: Error finding container 2ec3f19772447b85796b47970b53c8f611130ff5f2e903e3d16d75298d7cd230: Status 404 returned error can't find the container with id 2ec3f19772447b85796b47970b53c8f611130ff5f2e903e3d16d75298d7cd230 Sep 16 08:00:30.027904 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:30.027874 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:30.091277 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:30.091236 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4a243c-2gxn2" event={"ID":"14e1b3ab-35c4-4edf-9933-4f47afe311c3","Type":"ContainerStarted","Data":"dc4c13ff0b9ef5933c7620c8421f05300d1e1e1c07e1099b8179fa9db4fa1d93"} Sep 16 08:00:30.091483 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:30.091299 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4a243c-2gxn2" event={"ID":"14e1b3ab-35c4-4edf-9933-4f47afe311c3","Type":"ContainerStarted","Data":"2ec3f19772447b85796b47970b53c8f611130ff5f2e903e3d16d75298d7cd230"} Sep 16 08:00:30.107598 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:30.107542 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4a243c-2gxn2" podStartSLOduration=1.107525452 podStartE2EDuration="1.107525452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 08:00:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 08:00:30.106001049 +0000 UTC m=+1601.689779399" watchObservedRunningTime="2026-09-16 08:00:30.107525452 +0000 UTC m=+1601.691303790" Sep 16 08:00:30.248681 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:30.248637 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:31.025687 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:31.025655 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:31.098575 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:31.098542 2552 generic.go:356] "Generic (PLEG): container finished" podID="14e1b3ab-35c4-4edf-9933-4f47afe311c3" containerID="dc4c13ff0b9ef5933c7620c8421f05300d1e1e1c07e1099b8179fa9db4fa1d93" exitCode=0 Sep 16 08:00:31.098755 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:31.098620 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4a243c-2gxn2" event={"ID":"14e1b3ab-35c4-4edf-9933-4f47afe311c3","Type":"ContainerDied","Data":"dc4c13ff0b9ef5933c7620c8421f05300d1e1e1c07e1099b8179fa9db4fa1d93"} Sep 16 08:00:31.682862 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:31.682832 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:32.252238 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:32.252209 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4a243c-2gxn2" Sep 16 08:00:32.332605 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:32.332559 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14e1b3ab-35c4-4edf-9933-4f47afe311c3-maas-api-service-ca\") pod \"14e1b3ab-35c4-4edf-9933-4f47afe311c3\" (UID: \"14e1b3ab-35c4-4edf-9933-4f47afe311c3\") " Sep 16 08:00:32.332985 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:32.332956 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14e1b3ab-35c4-4edf-9933-4f47afe311c3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "14e1b3ab-35c4-4edf-9933-4f47afe311c3" (UID: "14e1b3ab-35c4-4edf-9933-4f47afe311c3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 08:00:32.433613 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:32.433576 2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14e1b3ab-35c4-4edf-9933-4f47afe311c3-maas-api-service-ca\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:00:32.610386 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:32.610349 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:33.111314 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:33.111275 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4a243c-2gxn2" event={"ID":"14e1b3ab-35c4-4edf-9933-4f47afe311c3","Type":"ContainerDied","Data":"2ec3f19772447b85796b47970b53c8f611130ff5f2e903e3d16d75298d7cd230"} Sep 16 08:00:33.111314 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:33.111310 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4a243c-2gxn2" Sep 16 08:00:33.111526 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:33.111336 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ec3f19772447b85796b47970b53c8f611130ff5f2e903e3d16d75298d7cd230" Sep 16 08:00:33.333076 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:33.332983 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:33.970275 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:33.970241 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:34.662219 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:34.662181 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:35.233947 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:35.233893 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-fc27a5/premium-w4-fc27a5-kserve-54dbcff578-kzxpf"] Sep 16 08:00:35.242578 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:35.242541 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-fc27a5/simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd"] Sep 16 08:00:35.291179 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:35.291141 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:39.146684 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:39.146648 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:39.693713 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:39.693666 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:39.713432 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:39.713392 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4a243c-6d986c5c4c-s42d9"] Sep 16 08:00:39.713684 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:39.713655 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4a243c-6d986c5c4c-s42d9" podUID="a8e51140-8400-4a87-9666-9b3a076dc538" containerName="payload-processing" containerID="cri-o://5141ec06f9223cb772c918ce2612b29649996474e1fb2b880cf6d046371ff460" gracePeriod=30 Sep 16 08:00:39.764913 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:39.764872 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:39.867679 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:39.867638 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:39.868222 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:39.868170 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4a243c-2gxn2"] Sep 16 08:00:39.870374 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:39.870352 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4a243c-2gxn2"] Sep 16 08:00:39.928089 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:39.928058 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:39.981810 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:39.981783 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4a243c-6d986c5c4c-s42d9" Sep 16 08:00:40.005353 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:40.005314 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e51140-8400-4a87-9666-9b3a076dc538-plugins-config-volume\") pod \"a8e51140-8400-4a87-9666-9b3a076dc538\" (UID: \"a8e51140-8400-4a87-9666-9b3a076dc538\") " Sep 16 08:00:40.005891 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:40.005496 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qqbx\" (UniqueName: \"kubernetes.io/projected/a8e51140-8400-4a87-9666-9b3a076dc538-kube-api-access-2qqbx\") pod \"a8e51140-8400-4a87-9666-9b3a076dc538\" (UID: \"a8e51140-8400-4a87-9666-9b3a076dc538\") " Sep 16 08:00:40.005891 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:40.005858 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8e51140-8400-4a87-9666-9b3a076dc538-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a8e51140-8400-4a87-9666-9b3a076dc538" (UID: "a8e51140-8400-4a87-9666-9b3a076dc538"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 08:00:40.007968 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:40.007940 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8e51140-8400-4a87-9666-9b3a076dc538-kube-api-access-2qqbx" (OuterVolumeSpecName: "kube-api-access-2qqbx") pod "a8e51140-8400-4a87-9666-9b3a076dc538" (UID: "a8e51140-8400-4a87-9666-9b3a076dc538"). InnerVolumeSpecName "kube-api-access-2qqbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 08:00:40.106512 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:40.106470 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qqbx\" (UniqueName: \"kubernetes.io/projected/a8e51140-8400-4a87-9666-9b3a076dc538-kube-api-access-2qqbx\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:00:40.106512 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:40.106505 2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e51140-8400-4a87-9666-9b3a076dc538-plugins-config-volume\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:00:40.155173 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:40.155136 2552 generic.go:356] "Generic (PLEG): container finished" podID="a8e51140-8400-4a87-9666-9b3a076dc538" containerID="5141ec06f9223cb772c918ce2612b29649996474e1fb2b880cf6d046371ff460" exitCode=0 Sep 16 08:00:40.155585 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:40.155195 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4a243c-6d986c5c4c-s42d9" Sep 16 08:00:40.155585 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:40.155223 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4a243c-6d986c5c4c-s42d9" event={"ID":"a8e51140-8400-4a87-9666-9b3a076dc538","Type":"ContainerDied","Data":"5141ec06f9223cb772c918ce2612b29649996474e1fb2b880cf6d046371ff460"} Sep 16 08:00:40.155585 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:40.155277 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4a243c-6d986c5c4c-s42d9" event={"ID":"a8e51140-8400-4a87-9666-9b3a076dc538","Type":"ContainerDied","Data":"fa83ba7c776ed10542b30bcca339110e5151739cf114d190da5c0a6ddcb6687a"} Sep 16 08:00:40.155585 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:40.155300 2552 scope.go:117] "RemoveContainer" containerID="5141ec06f9223cb772c918ce2612b29649996474e1fb2b880cf6d046371ff460" Sep 16 08:00:40.165515 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:40.165490 2552 scope.go:117] "RemoveContainer" containerID="5141ec06f9223cb772c918ce2612b29649996474e1fb2b880cf6d046371ff460" Sep 16 08:00:40.165889 ip-10-0-142-170 kubenswrapper[2552]: E0916 08:00:40.165869 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5141ec06f9223cb772c918ce2612b29649996474e1fb2b880cf6d046371ff460\": container with ID starting with 5141ec06f9223cb772c918ce2612b29649996474e1fb2b880cf6d046371ff460 not found: ID does not exist" containerID="5141ec06f9223cb772c918ce2612b29649996474e1fb2b880cf6d046371ff460" Sep 16 08:00:40.165934 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:40.165901 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5141ec06f9223cb772c918ce2612b29649996474e1fb2b880cf6d046371ff460"} err="failed to get container status \"5141ec06f9223cb772c918ce2612b29649996474e1fb2b880cf6d046371ff460\": rpc error: code = NotFound desc = could not find container \"5141ec06f9223cb772c918ce2612b29649996474e1fb2b880cf6d046371ff460\": container with ID starting with 5141ec06f9223cb772c918ce2612b29649996474e1fb2b880cf6d046371ff460 not found: ID does not exist" Sep 16 08:00:40.176906 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:40.176877 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4a243c-6d986c5c4c-s42d9"] Sep 16 08:00:40.178837 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:40.178815 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4a243c-6d986c5c4c-s42d9"] Sep 16 08:00:40.242212 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:40.242125 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:40.343979 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:40.343942 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:40.943265 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:40.943232 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14e1b3ab-35c4-4edf-9933-4f47afe311c3" path="/var/lib/kubelet/pods/14e1b3ab-35c4-4edf-9933-4f47afe311c3/volumes" Sep 16 08:00:40.943632 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:40.943620 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8e51140-8400-4a87-9666-9b3a076dc538" path="/var/lib/kubelet/pods/a8e51140-8400-4a87-9666-9b3a076dc538/volumes" Sep 16 08:00:41.071974 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:41.071888 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:41.719574 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:41.719532 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:42.517348 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:42.517316 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:43.177237 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:43.177195 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:45.894019 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:45.893983 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:45.954014 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:45.953979 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:46.159287 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:46.159194 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:46.480791 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:46.480745 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-984c1a-nml45"] Sep 16 08:00:46.481274 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:46.481253 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8e51140-8400-4a87-9666-9b3a076dc538" containerName="payload-processing" Sep 16 08:00:46.481377 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:46.481277 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8e51140-8400-4a87-9666-9b3a076dc538" containerName="payload-processing" Sep 16 08:00:46.481377 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:46.481313 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14e1b3ab-35c4-4edf-9933-4f47afe311c3" containerName="revoke-keys" Sep 16 08:00:46.481377 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:46.481322 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="14e1b3ab-35c4-4edf-9933-4f47afe311c3" containerName="revoke-keys" Sep 16 08:00:46.481528 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:46.481409 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8e51140-8400-4a87-9666-9b3a076dc538" containerName="payload-processing" Sep 16 08:00:46.481528 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:46.481428 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="14e1b3ab-35c4-4edf-9933-4f47afe311c3" containerName="revoke-keys" Sep 16 08:00:46.484830 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:46.484805 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-984c1a-nml45" Sep 16 08:00:46.487179 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:46.487156 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 08:00:46.487282 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:46.487203 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zwk5r\"" Sep 16 08:00:46.491533 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:46.491503 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-984c1a-nml45"] Sep 16 08:00:46.568299 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:46.568256 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41b62926-0a18-4aeb-9291-62b8bba7eaaa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-984c1a-nml45\" (UID: \"41b62926-0a18-4aeb-9291-62b8bba7eaaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-984c1a-nml45" Sep 16 08:00:46.669258 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:46.669216 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41b62926-0a18-4aeb-9291-62b8bba7eaaa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-984c1a-nml45\" (UID: \"41b62926-0a18-4aeb-9291-62b8bba7eaaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-984c1a-nml45" Sep 16 08:00:46.669862 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:46.669839 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41b62926-0a18-4aeb-9291-62b8bba7eaaa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-984c1a-nml45\" (UID: \"41b62926-0a18-4aeb-9291-62b8bba7eaaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-984c1a-nml45" Sep 16 08:00:46.745371 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:46.745249 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:46.796326 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:46.796291 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-984c1a-nml45" Sep 16 08:00:47.072299 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:47.072203 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-984c1a-nml45"] Sep 16 08:00:47.075613 ip-10-0-142-170 kubenswrapper[2552]: W0916 08:00:47.075584 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41b62926_0a18_4aeb_9291_62b8bba7eaaa.slice/crio-6fb2f54acd5b782668f860ebd98223019a9278aac4ea67b0806f48e30eca22a4 WatchSource:0}: Error finding container 6fb2f54acd5b782668f860ebd98223019a9278aac4ea67b0806f48e30eca22a4: Status 404 returned error can't find the container with id 6fb2f54acd5b782668f860ebd98223019a9278aac4ea67b0806f48e30eca22a4 Sep 16 08:00:47.203439 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:47.203403 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-984c1a-nml45" event={"ID":"41b62926-0a18-4aeb-9291-62b8bba7eaaa","Type":"ContainerStarted","Data":"6fb2f54acd5b782668f860ebd98223019a9278aac4ea67b0806f48e30eca22a4"} Sep 16 08:00:47.344821 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:47.343993 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:47.344821 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:47.344055 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fc27a5-29825760-2ktwd"] Sep 16 08:00:47.347373 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:47.347352 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-fc27a5-29825760-2ktwd"] Sep 16 08:00:47.366981 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:47.366941 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-fc27a5-6bf7fbd5ff-skp6c"] Sep 16 08:00:47.367837 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:47.367225 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-fc27a5-6bf7fbd5ff-skp6c" podUID="fd352ca8-49ca-4482-b54c-499885f04544" containerName="payload-processing" containerID="cri-o://175a6ab504df43187da9eff00ce7fb4625684a78e0d8fda3b43b8df20907d518" gracePeriod=30 Sep 16 08:00:47.376912 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:47.376870 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-fc27a5-54dc7c7d8b-5h7m6"] Sep 16 08:00:47.377737 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:47.377232 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fc27a5-54dc7c7d8b-5h7m6" podUID="dcb8b77a-fd48-4204-890d-b15f2e4ea7b3" containerName="payload-pre-processing" containerID="cri-o://5b8459fdfc95a350e62c531a4afea6fbf3eb28112462f988dd4dff70560a9b42" gracePeriod=30 Sep 16 08:00:47.409204 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:47.409127 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:47.510064 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:47.510010 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:47.572636 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:47.572579 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:47.623638 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:47.623600 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-fc27a5-6bf7fbd5ff-skp6c" Sep 16 08:00:47.663037 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:47.663010 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fc27a5-54dc7c7d8b-5h7m6" Sep 16 08:00:47.678361 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:47.678322 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f6xqx\" (UniqueName: \"kubernetes.io/projected/fd352ca8-49ca-4482-b54c-499885f04544-kube-api-access-f6xqx\") pod \"fd352ca8-49ca-4482-b54c-499885f04544\" (UID: \"fd352ca8-49ca-4482-b54c-499885f04544\") " Sep 16 08:00:47.678533 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:47.678414 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd352ca8-49ca-4482-b54c-499885f04544-plugins-config-volume\") pod \"fd352ca8-49ca-4482-b54c-499885f04544\" (UID: \"fd352ca8-49ca-4482-b54c-499885f04544\") " Sep 16 08:00:47.678822 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:47.678798 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd352ca8-49ca-4482-b54c-499885f04544-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fd352ca8-49ca-4482-b54c-499885f04544" (UID: "fd352ca8-49ca-4482-b54c-499885f04544"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 08:00:47.680878 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:47.680835 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd352ca8-49ca-4482-b54c-499885f04544-kube-api-access-f6xqx" (OuterVolumeSpecName: "kube-api-access-f6xqx") pod "fd352ca8-49ca-4482-b54c-499885f04544" (UID: "fd352ca8-49ca-4482-b54c-499885f04544"). InnerVolumeSpecName "kube-api-access-f6xqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 08:00:47.772356 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:47.772315 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:47.779941 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:47.779911 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcb8b77a-fd48-4204-890d-b15f2e4ea7b3-plugins-config-volume\") pod \"dcb8b77a-fd48-4204-890d-b15f2e4ea7b3\" (UID: \"dcb8b77a-fd48-4204-890d-b15f2e4ea7b3\") " Sep 16 08:00:47.780153 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:47.779960 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qbb7n\" (UniqueName: \"kubernetes.io/projected/dcb8b77a-fd48-4204-890d-b15f2e4ea7b3-kube-api-access-qbb7n\") pod \"dcb8b77a-fd48-4204-890d-b15f2e4ea7b3\" (UID: \"dcb8b77a-fd48-4204-890d-b15f2e4ea7b3\") " Sep 16 08:00:47.780272 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:47.780254 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6xqx\" (UniqueName: \"kubernetes.io/projected/fd352ca8-49ca-4482-b54c-499885f04544-kube-api-access-f6xqx\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:00:47.780333 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:47.780272 2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd352ca8-49ca-4482-b54c-499885f04544-plugins-config-volume\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:00:47.780333 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:47.780286 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcb8b77a-fd48-4204-890d-b15f2e4ea7b3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dcb8b77a-fd48-4204-890d-b15f2e4ea7b3" (UID: "dcb8b77a-fd48-4204-890d-b15f2e4ea7b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 08:00:47.782131 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:47.782091 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcb8b77a-fd48-4204-890d-b15f2e4ea7b3-kube-api-access-qbb7n" (OuterVolumeSpecName: "kube-api-access-qbb7n") pod "dcb8b77a-fd48-4204-890d-b15f2e4ea7b3" (UID: "dcb8b77a-fd48-4204-890d-b15f2e4ea7b3"). InnerVolumeSpecName "kube-api-access-qbb7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 08:00:47.881601 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:47.881514 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbb7n\" (UniqueName: \"kubernetes.io/projected/dcb8b77a-fd48-4204-890d-b15f2e4ea7b3-kube-api-access-qbb7n\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:00:47.881601 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:47.881546 2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcb8b77a-fd48-4204-890d-b15f2e4ea7b3-plugins-config-volume\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:00:47.990734 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:47.990695 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:48.211994 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:48.211959 2552 generic.go:356] "Generic (PLEG): container finished" podID="fd352ca8-49ca-4482-b54c-499885f04544" containerID="175a6ab504df43187da9eff00ce7fb4625684a78e0d8fda3b43b8df20907d518" exitCode=0 Sep 16 08:00:48.212484 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:48.212014 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-fc27a5-6bf7fbd5ff-skp6c" Sep 16 08:00:48.212484 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:48.212040 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-fc27a5-6bf7fbd5ff-skp6c" event={"ID":"fd352ca8-49ca-4482-b54c-499885f04544","Type":"ContainerDied","Data":"175a6ab504df43187da9eff00ce7fb4625684a78e0d8fda3b43b8df20907d518"} Sep 16 08:00:48.212484 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:48.212111 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-fc27a5-6bf7fbd5ff-skp6c" event={"ID":"fd352ca8-49ca-4482-b54c-499885f04544","Type":"ContainerDied","Data":"727d09d6cdcc06c86a9dba0218a425e01bcc057a9c8b822f56aa628fe1e44f09"} Sep 16 08:00:48.212484 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:48.212158 2552 scope.go:117] "RemoveContainer" containerID="175a6ab504df43187da9eff00ce7fb4625684a78e0d8fda3b43b8df20907d518" Sep 16 08:00:48.213747 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:48.213712 2552 generic.go:356] "Generic (PLEG): container finished" podID="dcb8b77a-fd48-4204-890d-b15f2e4ea7b3" containerID="5b8459fdfc95a350e62c531a4afea6fbf3eb28112462f988dd4dff70560a9b42" exitCode=0 Sep 16 08:00:48.213827 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:48.213782 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fc27a5-54dc7c7d8b-5h7m6" event={"ID":"dcb8b77a-fd48-4204-890d-b15f2e4ea7b3","Type":"ContainerDied","Data":"5b8459fdfc95a350e62c531a4afea6fbf3eb28112462f988dd4dff70560a9b42"} Sep 16 08:00:48.213888 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:48.213824 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fc27a5-54dc7c7d8b-5h7m6" event={"ID":"dcb8b77a-fd48-4204-890d-b15f2e4ea7b3","Type":"ContainerDied","Data":"3d5aeef0c8e3493546671304de10f4db71f172c6d747581fea67663cd1f6739d"} Sep 16 08:00:48.213888 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:48.213865 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-fc27a5-54dc7c7d8b-5h7m6" Sep 16 08:00:48.216322 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:48.216294 2552 generic.go:356] "Generic (PLEG): container finished" podID="41b62926-0a18-4aeb-9291-62b8bba7eaaa" containerID="9f93f0cdc7b7d0931eb9451536140b4d6c45e9f6029a746345488803ddda591c" exitCode=0 Sep 16 08:00:48.216454 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:48.216427 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-984c1a-nml45" event={"ID":"41b62926-0a18-4aeb-9291-62b8bba7eaaa","Type":"ContainerDied","Data":"9f93f0cdc7b7d0931eb9451536140b4d6c45e9f6029a746345488803ddda591c"} Sep 16 08:00:48.225277 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:48.225010 2552 scope.go:117] "RemoveContainer" containerID="175a6ab504df43187da9eff00ce7fb4625684a78e0d8fda3b43b8df20907d518" Sep 16 08:00:48.225556 ip-10-0-142-170 kubenswrapper[2552]: E0916 08:00:48.225528 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"175a6ab504df43187da9eff00ce7fb4625684a78e0d8fda3b43b8df20907d518\": container with ID starting with 175a6ab504df43187da9eff00ce7fb4625684a78e0d8fda3b43b8df20907d518 not found: ID does not exist" containerID="175a6ab504df43187da9eff00ce7fb4625684a78e0d8fda3b43b8df20907d518" Sep 16 08:00:48.225666 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:48.225567 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"175a6ab504df43187da9eff00ce7fb4625684a78e0d8fda3b43b8df20907d518"} err="failed to get container status \"175a6ab504df43187da9eff00ce7fb4625684a78e0d8fda3b43b8df20907d518\": rpc error: code = NotFound desc = could not find container \"175a6ab504df43187da9eff00ce7fb4625684a78e0d8fda3b43b8df20907d518\": container with ID starting with 175a6ab504df43187da9eff00ce7fb4625684a78e0d8fda3b43b8df20907d518 not found: ID does not exist" Sep 16 08:00:48.225666 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:48.225594 2552 scope.go:117] "RemoveContainer" containerID="5b8459fdfc95a350e62c531a4afea6fbf3eb28112462f988dd4dff70560a9b42" Sep 16 08:00:48.236724 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:48.236702 2552 scope.go:117] "RemoveContainer" containerID="5b8459fdfc95a350e62c531a4afea6fbf3eb28112462f988dd4dff70560a9b42" Sep 16 08:00:48.237070 ip-10-0-142-170 kubenswrapper[2552]: E0916 08:00:48.237047 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b8459fdfc95a350e62c531a4afea6fbf3eb28112462f988dd4dff70560a9b42\": container with ID starting with 5b8459fdfc95a350e62c531a4afea6fbf3eb28112462f988dd4dff70560a9b42 not found: ID does not exist" containerID="5b8459fdfc95a350e62c531a4afea6fbf3eb28112462f988dd4dff70560a9b42" Sep 16 08:00:48.237179 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:48.237078 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b8459fdfc95a350e62c531a4afea6fbf3eb28112462f988dd4dff70560a9b42"} err="failed to get container status \"5b8459fdfc95a350e62c531a4afea6fbf3eb28112462f988dd4dff70560a9b42\": rpc error: code = NotFound desc = could not find container \"5b8459fdfc95a350e62c531a4afea6fbf3eb28112462f988dd4dff70560a9b42\": container with ID starting with 5b8459fdfc95a350e62c531a4afea6fbf3eb28112462f988dd4dff70560a9b42 not found: ID does not exist" Sep 16 08:00:48.249794 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:48.249742 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-fc27a5-54dc7c7d8b-5h7m6"] Sep 16 08:00:48.253138 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:48.253084 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-fc27a5-54dc7c7d8b-5h7m6"] Sep 16 08:00:48.261539 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:48.261504 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-fc27a5-6bf7fbd5ff-skp6c"] Sep 16 08:00:48.263395 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:48.263372 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-fc27a5-6bf7fbd5ff-skp6c"] Sep 16 08:00:48.639030 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:48.638932 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:48.938111 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:48.938063 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15d8c3e8-444d-4210-888e-1f1f81d415e0" path="/var/lib/kubelet/pods/15d8c3e8-444d-4210-888e-1f1f81d415e0/volumes" Sep 16 08:00:48.938528 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:48.938508 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcb8b77a-fd48-4204-890d-b15f2e4ea7b3" path="/var/lib/kubelet/pods/dcb8b77a-fd48-4204-890d-b15f2e4ea7b3/volumes" Sep 16 08:00:48.938904 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:48.938886 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd352ca8-49ca-4482-b54c-499885f04544" path="/var/lib/kubelet/pods/fd352ca8-49ca-4482-b54c-499885f04544/volumes" Sep 16 08:00:49.093619 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:49.093586 2552 scope.go:117] "RemoveContainer" containerID="5544bc4b789089294bcd2c5ef3d84f442450c0b952f49e7b340b776ac7956b7e" Sep 16 08:00:49.284288 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:49.284253 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:49.374178 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:49.374147 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-984c1a-nml45" Sep 16 08:00:49.497623 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:49.497519 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41b62926-0a18-4aeb-9291-62b8bba7eaaa-maas-api-service-ca\") pod \"41b62926-0a18-4aeb-9291-62b8bba7eaaa\" (UID: \"41b62926-0a18-4aeb-9291-62b8bba7eaaa\") " Sep 16 08:00:49.497846 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:49.497821 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41b62926-0a18-4aeb-9291-62b8bba7eaaa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "41b62926-0a18-4aeb-9291-62b8bba7eaaa" (UID: "41b62926-0a18-4aeb-9291-62b8bba7eaaa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 08:00:49.598738 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:49.598697 2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/41b62926-0a18-4aeb-9291-62b8bba7eaaa-maas-api-service-ca\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:00:49.934116 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:49.934065 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:50.231642 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:50.231558 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-984c1a-nml45" Sep 16 08:00:50.231642 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:50.231596 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-984c1a-nml45" event={"ID":"41b62926-0a18-4aeb-9291-62b8bba7eaaa","Type":"ContainerDied","Data":"6fb2f54acd5b782668f860ebd98223019a9278aac4ea67b0806f48e30eca22a4"} Sep 16 08:00:50.231642 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:50.231639 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fb2f54acd5b782668f860ebd98223019a9278aac4ea67b0806f48e30eca22a4" Sep 16 08:00:50.259040 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:50.258998 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-fc27a5/premium-w4-fc27a5-kserve-54dbcff578-kzxpf" podUID="3edcc01e-d0c7-4f87-8a18-eb46302572df" containerName="main" containerID="cri-o://5ee0010b9c738fb5eb16240327bad86081d7168a42d9a10c0cf26cc1053b58df" gracePeriod=45 Sep 16 08:00:50.262615 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:50.262572 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-fc27a5/simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd" podUID="8a094926-e874-4ed6-94ef-f5e0616f7026" containerName="main" containerID="cri-o://fae799527d6331b34872566431319d4686f8bd4dfe6baad0adec4b44888b2e35" gracePeriod=45 Sep 16 08:00:50.534467 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:50.534436 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fc27a5/simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd" Sep 16 08:00:50.575093 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:50.575062 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fc27a5/premium-w4-fc27a5-kserve-54dbcff578-kzxpf" Sep 16 08:00:50.607391 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:50.607358 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a094926-e874-4ed6-94ef-f5e0616f7026-model-cache\") pod \"8a094926-e874-4ed6-94ef-f5e0616f7026\" (UID: \"8a094926-e874-4ed6-94ef-f5e0616f7026\") " Sep 16 08:00:50.607588 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:50.607419 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a094926-e874-4ed6-94ef-f5e0616f7026-tls-certs\") pod \"8a094926-e874-4ed6-94ef-f5e0616f7026\" (UID: \"8a094926-e874-4ed6-94ef-f5e0616f7026\") " Sep 16 08:00:50.607588 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:50.607454 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a094926-e874-4ed6-94ef-f5e0616f7026-home\") pod \"8a094926-e874-4ed6-94ef-f5e0616f7026\" (UID: \"8a094926-e874-4ed6-94ef-f5e0616f7026\") " Sep 16 08:00:50.607588 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:50.607509 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a094926-e874-4ed6-94ef-f5e0616f7026-tmp-dir\") pod \"8a094926-e874-4ed6-94ef-f5e0616f7026\" (UID: \"8a094926-e874-4ed6-94ef-f5e0616f7026\") " Sep 16 08:00:50.607588 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:50.607556 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wp7sj\" (UniqueName: \"kubernetes.io/projected/8a094926-e874-4ed6-94ef-f5e0616f7026-kube-api-access-wp7sj\") pod \"8a094926-e874-4ed6-94ef-f5e0616f7026\" (UID: \"8a094926-e874-4ed6-94ef-f5e0616f7026\") " Sep 16 08:00:50.607802 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:50.607661 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a094926-e874-4ed6-94ef-f5e0616f7026-dshm\") pod \"8a094926-e874-4ed6-94ef-f5e0616f7026\" (UID: \"8a094926-e874-4ed6-94ef-f5e0616f7026\") " Sep 16 08:00:50.607802 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:50.607713 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a094926-e874-4ed6-94ef-f5e0616f7026-model-cache" (OuterVolumeSpecName: "model-cache") pod "8a094926-e874-4ed6-94ef-f5e0616f7026" (UID: "8a094926-e874-4ed6-94ef-f5e0616f7026"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:00:50.607802 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:50.607765 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a094926-e874-4ed6-94ef-f5e0616f7026-home" (OuterVolumeSpecName: "home") pod "8a094926-e874-4ed6-94ef-f5e0616f7026" (UID: "8a094926-e874-4ed6-94ef-f5e0616f7026"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:00:50.607802 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:50.607772 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a094926-e874-4ed6-94ef-f5e0616f7026-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8a094926-e874-4ed6-94ef-f5e0616f7026" (UID: "8a094926-e874-4ed6-94ef-f5e0616f7026"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:00:50.608057 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:50.608020 2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a094926-e874-4ed6-94ef-f5e0616f7026-model-cache\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:00:50.608057 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:50.608046 2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a094926-e874-4ed6-94ef-f5e0616f7026-home\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:00:50.608201 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:50.608085 2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a094926-e874-4ed6-94ef-f5e0616f7026-tmp-dir\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:00:50.610001 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:50.609936 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a094926-e874-4ed6-94ef-f5e0616f7026-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8a094926-e874-4ed6-94ef-f5e0616f7026" (UID: "8a094926-e874-4ed6-94ef-f5e0616f7026"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 08:00:50.610176 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:50.610049 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a094926-e874-4ed6-94ef-f5e0616f7026-kube-api-access-wp7sj" (OuterVolumeSpecName: "kube-api-access-wp7sj") pod "8a094926-e874-4ed6-94ef-f5e0616f7026" (UID: "8a094926-e874-4ed6-94ef-f5e0616f7026"). InnerVolumeSpecName "kube-api-access-wp7sj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 08:00:50.610176 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:50.610059 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a094926-e874-4ed6-94ef-f5e0616f7026-dshm" (OuterVolumeSpecName: "dshm") pod "8a094926-e874-4ed6-94ef-f5e0616f7026" (UID: "8a094926-e874-4ed6-94ef-f5e0616f7026"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:00:50.709073 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:50.709034 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3edcc01e-d0c7-4f87-8a18-eb46302572df-tmp-dir\") pod \"3edcc01e-d0c7-4f87-8a18-eb46302572df\" (UID: \"3edcc01e-d0c7-4f87-8a18-eb46302572df\") " Sep 16 08:00:50.709073 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:50.709080 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3edcc01e-d0c7-4f87-8a18-eb46302572df-dshm\") pod \"3edcc01e-d0c7-4f87-8a18-eb46302572df\" (UID: \"3edcc01e-d0c7-4f87-8a18-eb46302572df\") " Sep 16 08:00:50.709352 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:50.709116 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45k2k\" (UniqueName: \"kubernetes.io/projected/3edcc01e-d0c7-4f87-8a18-eb46302572df-kube-api-access-45k2k\") pod \"3edcc01e-d0c7-4f87-8a18-eb46302572df\" (UID: \"3edcc01e-d0c7-4f87-8a18-eb46302572df\") " Sep 16 08:00:50.709352 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:50.709141 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3edcc01e-d0c7-4f87-8a18-eb46302572df-model-cache\") pod \"3edcc01e-d0c7-4f87-8a18-eb46302572df\" (UID: \"3edcc01e-d0c7-4f87-8a18-eb46302572df\") " Sep 16 08:00:50.709352 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:50.709288 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3edcc01e-d0c7-4f87-8a18-eb46302572df-tls-certs\") pod \"3edcc01e-d0c7-4f87-8a18-eb46302572df\" (UID: \"3edcc01e-d0c7-4f87-8a18-eb46302572df\") " Sep 16 08:00:50.709352 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:50.709324 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3edcc01e-d0c7-4f87-8a18-eb46302572df-home\") pod \"3edcc01e-d0c7-4f87-8a18-eb46302572df\" (UID: \"3edcc01e-d0c7-4f87-8a18-eb46302572df\") " Sep 16 08:00:50.709549 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:50.709382 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3edcc01e-d0c7-4f87-8a18-eb46302572df-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3edcc01e-d0c7-4f87-8a18-eb46302572df" (UID: "3edcc01e-d0c7-4f87-8a18-eb46302572df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:00:50.709549 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:50.709482 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3edcc01e-d0c7-4f87-8a18-eb46302572df-model-cache" (OuterVolumeSpecName: "model-cache") pod "3edcc01e-d0c7-4f87-8a18-eb46302572df" (UID: "3edcc01e-d0c7-4f87-8a18-eb46302572df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:00:50.709664 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:50.709579 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3edcc01e-d0c7-4f87-8a18-eb46302572df-home" (OuterVolumeSpecName: "home") pod "3edcc01e-d0c7-4f87-8a18-eb46302572df" (UID: "3edcc01e-d0c7-4f87-8a18-eb46302572df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:00:50.709711 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:50.709668 2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3edcc01e-d0c7-4f87-8a18-eb46302572df-home\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:00:50.709711 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:50.709679 2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3edcc01e-d0c7-4f87-8a18-eb46302572df-tmp-dir\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:00:50.709711 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:50.709691 2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3edcc01e-d0c7-4f87-8a18-eb46302572df-model-cache\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:00:50.709711 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:50.709699 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp7sj\" (UniqueName: \"kubernetes.io/projected/8a094926-e874-4ed6-94ef-f5e0616f7026-kube-api-access-wp7sj\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:00:50.709914 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:50.709713 2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a094926-e874-4ed6-94ef-f5e0616f7026-dshm\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:00:50.709914 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:50.709726 2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a094926-e874-4ed6-94ef-f5e0616f7026-tls-certs\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:00:50.711340 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:50.711311 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3edcc01e-d0c7-4f87-8a18-eb46302572df-dshm" (OuterVolumeSpecName: "dshm") pod "3edcc01e-d0c7-4f87-8a18-eb46302572df" (UID: "3edcc01e-d0c7-4f87-8a18-eb46302572df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:00:50.711340 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:50.711313 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3edcc01e-d0c7-4f87-8a18-eb46302572df-kube-api-access-45k2k" (OuterVolumeSpecName: "kube-api-access-45k2k") pod "3edcc01e-d0c7-4f87-8a18-eb46302572df" (UID: "3edcc01e-d0c7-4f87-8a18-eb46302572df"). InnerVolumeSpecName "kube-api-access-45k2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 08:00:50.711494 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:50.711352 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3edcc01e-d0c7-4f87-8a18-eb46302572df-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3edcc01e-d0c7-4f87-8a18-eb46302572df" (UID: "3edcc01e-d0c7-4f87-8a18-eb46302572df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 08:00:50.810923 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:50.810830 2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3edcc01e-d0c7-4f87-8a18-eb46302572df-tls-certs\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:00:50.810923 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:50.810860 2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3edcc01e-d0c7-4f87-8a18-eb46302572df-dshm\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:00:50.810923 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:50.810869 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45k2k\" (UniqueName: \"kubernetes.io/projected/3edcc01e-d0c7-4f87-8a18-eb46302572df-kube-api-access-45k2k\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:00:50.854591 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:50.854553 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:51.239229 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:51.239195 2552 generic.go:356] "Generic (PLEG): container finished" podID="8a094926-e874-4ed6-94ef-f5e0616f7026" containerID="fae799527d6331b34872566431319d4686f8bd4dfe6baad0adec4b44888b2e35" exitCode=0 Sep 16 08:00:51.239389 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:51.239239 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fc27a5/simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd" Sep 16 08:00:51.239389 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:51.239270 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fc27a5/simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd" event={"ID":"8a094926-e874-4ed6-94ef-f5e0616f7026","Type":"ContainerDied","Data":"fae799527d6331b34872566431319d4686f8bd4dfe6baad0adec4b44888b2e35"} Sep 16 08:00:51.239389 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:51.239311 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fc27a5/simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd" event={"ID":"8a094926-e874-4ed6-94ef-f5e0616f7026","Type":"ContainerDied","Data":"5a8e5e23fdcb8ef86174aee22b5119e3161b08b8835d931b05f4b60688e64656"} Sep 16 08:00:51.239389 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:51.239337 2552 scope.go:117] "RemoveContainer" containerID="fae799527d6331b34872566431319d4686f8bd4dfe6baad0adec4b44888b2e35" Sep 16 08:00:51.240812 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:51.240786 2552 generic.go:356] "Generic (PLEG): container finished" podID="3edcc01e-d0c7-4f87-8a18-eb46302572df" containerID="5ee0010b9c738fb5eb16240327bad86081d7168a42d9a10c0cf26cc1053b58df" exitCode=0 Sep 16 08:00:51.240943 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:51.240828 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-fc27a5/premium-w4-fc27a5-kserve-54dbcff578-kzxpf" Sep 16 08:00:51.240943 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:51.240853 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fc27a5/premium-w4-fc27a5-kserve-54dbcff578-kzxpf" event={"ID":"3edcc01e-d0c7-4f87-8a18-eb46302572df","Type":"ContainerDied","Data":"5ee0010b9c738fb5eb16240327bad86081d7168a42d9a10c0cf26cc1053b58df"} Sep 16 08:00:51.240943 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:51.240884 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-fc27a5/premium-w4-fc27a5-kserve-54dbcff578-kzxpf" event={"ID":"3edcc01e-d0c7-4f87-8a18-eb46302572df","Type":"ContainerDied","Data":"b8db2622f8f43399745bd541c845f3a9ed3daf69b7abdea65dc7c0cce824bc52"} Sep 16 08:00:51.248399 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:51.248377 2552 scope.go:117] "RemoveContainer" containerID="fae799527d6331b34872566431319d4686f8bd4dfe6baad0adec4b44888b2e35" Sep 16 08:00:51.248760 ip-10-0-142-170 kubenswrapper[2552]: E0916 08:00:51.248733 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fae799527d6331b34872566431319d4686f8bd4dfe6baad0adec4b44888b2e35\": container with ID starting with fae799527d6331b34872566431319d4686f8bd4dfe6baad0adec4b44888b2e35 not found: ID does not exist" containerID="fae799527d6331b34872566431319d4686f8bd4dfe6baad0adec4b44888b2e35" Sep 16 08:00:51.248822 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:51.248774 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fae799527d6331b34872566431319d4686f8bd4dfe6baad0adec4b44888b2e35"} err="failed to get container status \"fae799527d6331b34872566431319d4686f8bd4dfe6baad0adec4b44888b2e35\": rpc error: code = NotFound desc = could not find container \"fae799527d6331b34872566431319d4686f8bd4dfe6baad0adec4b44888b2e35\": container with ID starting with fae799527d6331b34872566431319d4686f8bd4dfe6baad0adec4b44888b2e35 not found: ID does not exist" Sep 16 08:00:51.248822 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:51.248799 2552 scope.go:117] "RemoveContainer" containerID="5ee0010b9c738fb5eb16240327bad86081d7168a42d9a10c0cf26cc1053b58df" Sep 16 08:00:51.257967 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:51.257933 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-fc27a5/premium-w4-fc27a5-kserve-54dbcff578-kzxpf"] Sep 16 08:00:51.258547 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:51.258521 2552 scope.go:117] "RemoveContainer" containerID="5ee0010b9c738fb5eb16240327bad86081d7168a42d9a10c0cf26cc1053b58df" Sep 16 08:00:51.258929 ip-10-0-142-170 kubenswrapper[2552]: E0916 08:00:51.258907 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ee0010b9c738fb5eb16240327bad86081d7168a42d9a10c0cf26cc1053b58df\": container with ID starting with 5ee0010b9c738fb5eb16240327bad86081d7168a42d9a10c0cf26cc1053b58df not found: ID does not exist" containerID="5ee0010b9c738fb5eb16240327bad86081d7168a42d9a10c0cf26cc1053b58df" Sep 16 08:00:51.258998 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:51.258937 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ee0010b9c738fb5eb16240327bad86081d7168a42d9a10c0cf26cc1053b58df"} err="failed to get container status \"5ee0010b9c738fb5eb16240327bad86081d7168a42d9a10c0cf26cc1053b58df\": rpc error: code = NotFound desc = could not find container \"5ee0010b9c738fb5eb16240327bad86081d7168a42d9a10c0cf26cc1053b58df\": container with ID starting with 5ee0010b9c738fb5eb16240327bad86081d7168a42d9a10c0cf26cc1053b58df not found: ID does not exist" Sep 16 08:00:51.263847 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:51.263818 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-fc27a5/premium-w4-fc27a5-kserve-54dbcff578-kzxpf"] Sep 16 08:00:51.274032 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:51.273997 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-fc27a5/simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd"] Sep 16 08:00:51.277646 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:51.277617 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-fc27a5/simulator-w4-fc27a5-kserve-5bc9f597dc-7cdxd"] Sep 16 08:00:52.937348 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:52.937313 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3edcc01e-d0c7-4f87-8a18-eb46302572df" path="/var/lib/kubelet/pods/3edcc01e-d0c7-4f87-8a18-eb46302572df/volumes" Sep 16 08:00:52.937735 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:52.937714 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a094926-e874-4ed6-94ef-f5e0616f7026" path="/var/lib/kubelet/pods/8a094926-e874-4ed6-94ef-f5e0616f7026/volumes" Sep 16 08:00:53.832456 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:53.832422 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:53.933012 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:53.932978 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:54.100342 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:54.100243 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:56.772475 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:56.772438 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:57.295628 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:57.295582 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-984c1a-29825760-v69p5"] Sep 16 08:00:57.299537 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:57.299496 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:57.300065 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:57.300040 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-984c1a-29825760-v69p5"] Sep 16 08:00:57.367356 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:57.367321 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:57.468333 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:57.468301 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:57.476177 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:57.476139 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-984c1a-nml45"] Sep 16 08:00:57.480701 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:57.480669 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-984c1a-nml45"] Sep 16 08:00:57.521827 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:57.521783 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:57.856648 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:57.856605 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:57.956931 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:57.956894 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:58.546500 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:58.546461 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:00:58.937177 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:58.937145 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="229f48c6-1ae0-4a77-8631-96364e247a44" path="/var/lib/kubelet/pods/229f48c6-1ae0-4a77-8631-96364e247a44/volumes" Sep 16 08:00:58.937563 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:58.937477 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41b62926-0a18-4aeb-9291-62b8bba7eaaa" path="/var/lib/kubelet/pods/41b62926-0a18-4aeb-9291-62b8bba7eaaa/volumes" Sep 16 08:00:59.292087 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:00:59.291979 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:01:04.084519 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:04.084476 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:01:04.184384 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:04.184345 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:01:04.369430 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:04.369335 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:01:11.834989 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:11.834951 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:01:11.958170 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:11.958069 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:01:12.670536 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:12.670497 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:01:18.546520 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:18.546482 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:01:30.190764 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:30.190709 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:01:30.291581 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:30.291520 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:01:37.216076 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:37.216036 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:01:37.947296 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:37.947253 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:01:38.192848 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:38.192812 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:01:42.878740 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:42.878705 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-3616cc/sub-test-model-3616cc-kserve-59687759f5-dfxnv"] Sep 16 08:01:42.879260 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:42.879071 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd352ca8-49ca-4482-b54c-499885f04544" containerName="payload-processing" Sep 16 08:01:42.879260 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:42.879082 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd352ca8-49ca-4482-b54c-499885f04544" containerName="payload-processing" Sep 16 08:01:42.879260 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:42.879141 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3edcc01e-d0c7-4f87-8a18-eb46302572df" containerName="main" Sep 16 08:01:42.879260 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:42.879148 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="3edcc01e-d0c7-4f87-8a18-eb46302572df" containerName="main" Sep 16 08:01:42.879260 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:42.879156 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41b62926-0a18-4aeb-9291-62b8bba7eaaa" containerName="revoke-keys" Sep 16 08:01:42.879260 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:42.879162 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="41b62926-0a18-4aeb-9291-62b8bba7eaaa" containerName="revoke-keys" Sep 16 08:01:42.879260 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:42.879182 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcb8b77a-fd48-4204-890d-b15f2e4ea7b3" containerName="payload-pre-processing" Sep 16 08:01:42.879260 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:42.879188 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcb8b77a-fd48-4204-890d-b15f2e4ea7b3" containerName="payload-pre-processing" Sep 16 08:01:42.879260 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:42.879194 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a094926-e874-4ed6-94ef-f5e0616f7026" containerName="main" Sep 16 08:01:42.879260 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:42.879199 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a094926-e874-4ed6-94ef-f5e0616f7026" containerName="main" Sep 16 08:01:42.879260 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:42.879253 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a094926-e874-4ed6-94ef-f5e0616f7026" containerName="main" Sep 16 08:01:42.879260 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:42.879264 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcb8b77a-fd48-4204-890d-b15f2e4ea7b3" containerName="payload-pre-processing" Sep 16 08:01:42.879672 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:42.879270 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd352ca8-49ca-4482-b54c-499885f04544" containerName="payload-processing" Sep 16 08:01:42.879672 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:42.879275 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="3edcc01e-d0c7-4f87-8a18-eb46302572df" containerName="main" Sep 16 08:01:42.879672 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:42.879280 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="41b62926-0a18-4aeb-9291-62b8bba7eaaa" containerName="revoke-keys" Sep 16 08:01:42.882366 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:42.882346 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3616cc/sub-test-model-3616cc-kserve-59687759f5-dfxnv" Sep 16 08:01:42.886822 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:42.886793 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-3616cc\"/\"default-dockercfg-lm9f7\"" Sep 16 08:01:42.886973 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:42.886855 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-3616cc\"/\"openshift-service-ca.crt\"" Sep 16 08:01:42.886973 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:42.886876 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-3616cc\"/\"kube-root-ca.crt\"" Sep 16 08:01:42.886973 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:42.886793 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-3616cc\"/\"sub-test-model-3616cc-kserve-self-signed-certs\"" Sep 16 08:01:42.906871 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:42.906834 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3616cc/sub-test-model-3616cc-kserve-59687759f5-dfxnv"] Sep 16 08:01:42.938130 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:42.936934 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:01:43.012357 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:43.012322 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6-home\") pod \"sub-test-model-3616cc-kserve-59687759f5-dfxnv\" (UID: \"ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/sub-test-model-3616cc-kserve-59687759f5-dfxnv" Sep 16 08:01:43.012550 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:43.012374 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6-tmp-dir\") pod \"sub-test-model-3616cc-kserve-59687759f5-dfxnv\" (UID: \"ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/sub-test-model-3616cc-kserve-59687759f5-dfxnv" Sep 16 08:01:43.012550 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:43.012428 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6-model-cache\") pod \"sub-test-model-3616cc-kserve-59687759f5-dfxnv\" (UID: \"ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/sub-test-model-3616cc-kserve-59687759f5-dfxnv" Sep 16 08:01:43.012550 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:43.012448 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6-dshm\") pod \"sub-test-model-3616cc-kserve-59687759f5-dfxnv\" (UID: \"ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/sub-test-model-3616cc-kserve-59687759f5-dfxnv" Sep 16 08:01:43.012550 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:43.012482 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6-tls-certs\") pod \"sub-test-model-3616cc-kserve-59687759f5-dfxnv\" (UID: \"ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/sub-test-model-3616cc-kserve-59687759f5-dfxnv" Sep 16 08:01:43.012550 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:43.012522 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpw79\" (UniqueName: \"kubernetes.io/projected/ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6-kube-api-access-vpw79\") pod \"sub-test-model-3616cc-kserve-59687759f5-dfxnv\" (UID: \"ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/sub-test-model-3616cc-kserve-59687759f5-dfxnv" Sep 16 08:01:43.113326 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:43.113285 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpw79\" (UniqueName: \"kubernetes.io/projected/ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6-kube-api-access-vpw79\") pod \"sub-test-model-3616cc-kserve-59687759f5-dfxnv\" (UID: \"ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/sub-test-model-3616cc-kserve-59687759f5-dfxnv" Sep 16 08:01:43.113518 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:43.113343 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6-home\") pod \"sub-test-model-3616cc-kserve-59687759f5-dfxnv\" (UID: \"ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/sub-test-model-3616cc-kserve-59687759f5-dfxnv" Sep 16 08:01:43.113518 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:43.113379 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6-tmp-dir\") pod \"sub-test-model-3616cc-kserve-59687759f5-dfxnv\" (UID: \"ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/sub-test-model-3616cc-kserve-59687759f5-dfxnv" Sep 16 08:01:43.113518 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:43.113411 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6-model-cache\") pod \"sub-test-model-3616cc-kserve-59687759f5-dfxnv\" (UID: \"ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/sub-test-model-3616cc-kserve-59687759f5-dfxnv" Sep 16 08:01:43.113518 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:43.113442 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6-dshm\") pod \"sub-test-model-3616cc-kserve-59687759f5-dfxnv\" (UID: \"ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/sub-test-model-3616cc-kserve-59687759f5-dfxnv" Sep 16 08:01:43.113518 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:43.113479 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6-tls-certs\") pod \"sub-test-model-3616cc-kserve-59687759f5-dfxnv\" (UID: \"ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/sub-test-model-3616cc-kserve-59687759f5-dfxnv" Sep 16 08:01:43.113806 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:43.113779 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6-home\") pod \"sub-test-model-3616cc-kserve-59687759f5-dfxnv\" (UID: \"ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/sub-test-model-3616cc-kserve-59687759f5-dfxnv" Sep 16 08:01:43.113870 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:43.113811 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6-tmp-dir\") pod \"sub-test-model-3616cc-kserve-59687759f5-dfxnv\" (UID: \"ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/sub-test-model-3616cc-kserve-59687759f5-dfxnv" Sep 16 08:01:43.113929 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:43.113881 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6-model-cache\") pod \"sub-test-model-3616cc-kserve-59687759f5-dfxnv\" (UID: \"ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/sub-test-model-3616cc-kserve-59687759f5-dfxnv" Sep 16 08:01:43.115792 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:43.115756 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6-dshm\") pod \"sub-test-model-3616cc-kserve-59687759f5-dfxnv\" (UID: \"ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/sub-test-model-3616cc-kserve-59687759f5-dfxnv" Sep 16 08:01:43.115952 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:43.115934 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6-tls-certs\") pod \"sub-test-model-3616cc-kserve-59687759f5-dfxnv\" (UID: \"ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/sub-test-model-3616cc-kserve-59687759f5-dfxnv" Sep 16 08:01:43.123892 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:43.123866 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpw79\" (UniqueName: \"kubernetes.io/projected/ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6-kube-api-access-vpw79\") pod \"sub-test-model-3616cc-kserve-59687759f5-dfxnv\" (UID: \"ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6\") " pod="ai-tenant-e2e-shared-a-w3-3616cc/sub-test-model-3616cc-kserve-59687759f5-dfxnv" Sep 16 08:01:43.192288 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:43.192244 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3616cc/sub-test-model-3616cc-kserve-59687759f5-dfxnv" Sep 16 08:01:43.467958 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:43.467870 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3616cc/sub-test-model-3616cc-kserve-59687759f5-dfxnv"] Sep 16 08:01:43.470591 ip-10-0-142-170 kubenswrapper[2552]: W0916 08:01:43.470560 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac65ffde_4ea2_4dbe_9414_0d2eed10e8f6.slice/crio-37d0fbaa6e695a62dc661ff34df3acb37d1a12f8fecff159896567cd2af82bf0 WatchSource:0}: Error finding container 37d0fbaa6e695a62dc661ff34df3acb37d1a12f8fecff159896567cd2af82bf0: Status 404 returned error can't find the container with id 37d0fbaa6e695a62dc661ff34df3acb37d1a12f8fecff159896567cd2af82bf0 Sep 16 08:01:43.536633 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:43.536600 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3616cc/sub-test-model-3616cc-kserve-59687759f5-dfxnv" event={"ID":"ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6","Type":"ContainerStarted","Data":"37d0fbaa6e695a62dc661ff34df3acb37d1a12f8fecff159896567cd2af82bf0"} Sep 16 08:01:44.543822 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:44.543788 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3616cc/sub-test-model-3616cc-kserve-59687759f5-dfxnv" event={"ID":"ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6","Type":"ContainerStarted","Data":"a59198d7713332826e4c2c1a435bb59b17f1188b8dd84f71a2d182c6dd0b5bae"} Sep 16 08:01:44.559936 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:44.559873 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-3616cc/sub-test-model-3616cc-kserve-59687759f5-dfxnv" podStartSLOduration=2.559857686 podStartE2EDuration="2.559857686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 08:01:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 08:01:44.55929309 +0000 UTC m=+1676.143071444" watchObservedRunningTime="2026-09-16 08:01:44.559857686 +0000 UTC m=+1676.143636024" Sep 16 08:01:53.193089 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:53.193043 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-3616cc/sub-test-model-3616cc-kserve-59687759f5-dfxnv" Sep 16 08:01:53.193551 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:53.193134 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-3616cc/sub-test-model-3616cc-kserve-59687759f5-dfxnv" Sep 16 08:01:53.205981 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:53.205949 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-3616cc/sub-test-model-3616cc-kserve-59687759f5-dfxnv" Sep 16 08:01:53.604548 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:01:53.604456 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-3616cc/sub-test-model-3616cc-kserve-59687759f5-dfxnv" Sep 16 08:02:24.394725 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:02:24.394685 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:02:48.705579 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:02:48.705542 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:02:49.174024 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:02:49.173993 2552 scope.go:117] "RemoveContainer" containerID="1cbf22575a6ff24a053a82717554103551ac11aad6a09ca818e0968d0f855e9e" Sep 16 08:03:03.377126 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:03:03.377074 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3616cc/sub-test-model-3616cc-kserve-59687759f5-dfxnv"] Sep 16 08:03:03.397857 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:03:03.397821 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:03:04.279747 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:03:04.279715 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:03:16.396629 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:03:16.396587 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:03:18.399186 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:03:18.399143 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-3616cc/sub-test-model-3616cc-kserve-59687759f5-dfxnv" podUID="ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6" containerName="main" containerID="cri-o://a59198d7713332826e4c2c1a435bb59b17f1188b8dd84f71a2d182c6dd0b5bae" gracePeriod=45 Sep 16 08:03:18.661672 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:03:18.661601 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3616cc/sub-test-model-3616cc-kserve-59687759f5-dfxnv" Sep 16 08:03:18.818817 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:03:18.818780 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6-home\") pod \"ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6\" (UID: \"ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6\") " Sep 16 08:03:18.818992 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:03:18.818829 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vpw79\" (UniqueName: \"kubernetes.io/projected/ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6-kube-api-access-vpw79\") pod \"ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6\" (UID: \"ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6\") " Sep 16 08:03:18.818992 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:03:18.818875 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6-tls-certs\") pod \"ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6\" (UID: \"ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6\") " Sep 16 08:03:18.818992 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:03:18.818911 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6-model-cache\") pod \"ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6\" (UID: \"ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6\") " Sep 16 08:03:18.818992 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:03:18.818940 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6-dshm\") pod \"ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6\" (UID: \"ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6\") " Sep 16 08:03:18.819271 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:03:18.818998 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6-tmp-dir\") pod \"ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6\" (UID: \"ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6\") " Sep 16 08:03:18.819271 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:03:18.819140 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6-home" (OuterVolumeSpecName: "home") pod "ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6" (UID: "ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:03:18.819424 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:03:18.819405 2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6-home\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:03:18.819484 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:03:18.819431 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6" (UID: "ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:03:18.819534 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:03:18.819479 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6-model-cache" (OuterVolumeSpecName: "model-cache") pod "ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6" (UID: "ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:03:18.822010 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:03:18.821982 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6-kube-api-access-vpw79" (OuterVolumeSpecName: "kube-api-access-vpw79") pod "ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6" (UID: "ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6"). InnerVolumeSpecName "kube-api-access-vpw79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 08:03:18.822166 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:03:18.822010 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6" (UID: "ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 08:03:18.822852 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:03:18.822823 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6-dshm" (OuterVolumeSpecName: "dshm") pod "ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6" (UID: "ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:03:18.920477 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:03:18.920441 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpw79\" (UniqueName: \"kubernetes.io/projected/ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6-kube-api-access-vpw79\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:03:18.920477 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:03:18.920475 2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6-tls-certs\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:03:18.920674 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:03:18.920488 2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6-model-cache\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:03:18.920674 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:03:18.920497 2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6-dshm\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:03:18.920674 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:03:18.920506 2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6-tmp-dir\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:03:19.056491 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:03:19.056453 2552 generic.go:356] "Generic (PLEG): container finished" podID="ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6" containerID="a59198d7713332826e4c2c1a435bb59b17f1188b8dd84f71a2d182c6dd0b5bae" exitCode=0 Sep 16 08:03:19.056679 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:03:19.056511 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3616cc/sub-test-model-3616cc-kserve-59687759f5-dfxnv" Sep 16 08:03:19.056679 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:03:19.056535 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3616cc/sub-test-model-3616cc-kserve-59687759f5-dfxnv" event={"ID":"ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6","Type":"ContainerDied","Data":"a59198d7713332826e4c2c1a435bb59b17f1188b8dd84f71a2d182c6dd0b5bae"} Sep 16 08:03:19.056679 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:03:19.056579 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3616cc/sub-test-model-3616cc-kserve-59687759f5-dfxnv" event={"ID":"ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6","Type":"ContainerDied","Data":"37d0fbaa6e695a62dc661ff34df3acb37d1a12f8fecff159896567cd2af82bf0"} Sep 16 08:03:19.056679 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:03:19.056597 2552 scope.go:117] "RemoveContainer" containerID="a59198d7713332826e4c2c1a435bb59b17f1188b8dd84f71a2d182c6dd0b5bae" Sep 16 08:03:19.066872 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:03:19.066852 2552 scope.go:117] "RemoveContainer" containerID="a59198d7713332826e4c2c1a435bb59b17f1188b8dd84f71a2d182c6dd0b5bae" Sep 16 08:03:19.067248 ip-10-0-142-170 kubenswrapper[2552]: E0916 08:03:19.067225 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a59198d7713332826e4c2c1a435bb59b17f1188b8dd84f71a2d182c6dd0b5bae\": container with ID starting with a59198d7713332826e4c2c1a435bb59b17f1188b8dd84f71a2d182c6dd0b5bae not found: ID does not exist" containerID="a59198d7713332826e4c2c1a435bb59b17f1188b8dd84f71a2d182c6dd0b5bae" Sep 16 08:03:19.067324 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:03:19.067263 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a59198d7713332826e4c2c1a435bb59b17f1188b8dd84f71a2d182c6dd0b5bae"} err="failed to get container status \"a59198d7713332826e4c2c1a435bb59b17f1188b8dd84f71a2d182c6dd0b5bae\": rpc error: code = NotFound desc = could not find container \"a59198d7713332826e4c2c1a435bb59b17f1188b8dd84f71a2d182c6dd0b5bae\": container with ID starting with a59198d7713332826e4c2c1a435bb59b17f1188b8dd84f71a2d182c6dd0b5bae not found: ID does not exist" Sep 16 08:03:19.073086 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:03:19.073057 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3616cc/sub-test-model-3616cc-kserve-59687759f5-dfxnv"] Sep 16 08:03:19.076851 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:03:19.076820 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3616cc/sub-test-model-3616cc-kserve-59687759f5-dfxnv"] Sep 16 08:03:20.937537 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:03:20.937502 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6" path="/var/lib/kubelet/pods/ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6/volumes" Sep 16 08:03:42.819457 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:03:42.819371 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:04:22.802595 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:22.802557 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:04:24.807125 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:24.807067 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:04:26.766291 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:26.766257 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:04:27.341156 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:27.341119 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:04:27.472526 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:27.472484 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:04:27.576117 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:27.576070 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:04:28.398347 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:28.398219 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:04:29.271480 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:29.271443 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:04:29.373974 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:29.373927 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:04:29.480423 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:29.480386 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:04:29.814561 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:29.814526 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:04:29.913270 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:29.913233 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:04:44.174714 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:44.174675 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:04:51.068287 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:51.068247 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:04:52.232686 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:52.232653 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:04:52.393907 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:52.393869 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:04:52.912787 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:52.912755 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:04:53.121293 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:53.121255 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:04:53.415863 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:53.415826 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz"] Sep 16 08:04:53.416280 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:53.416215 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6" containerName="main" Sep 16 08:04:53.416280 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:53.416230 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6" containerName="main" Sep 16 08:04:53.416363 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:53.416308 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac65ffde-4ea2-4dbe-9414-0d2eed10e8f6" containerName="main" Sep 16 08:04:53.419521 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:53.419502 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" Sep 16 08:04:53.421887 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:53.421865 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-5493e8-openshift-default-dockercfg-swfbw\"" Sep 16 08:04:53.433320 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:53.433286 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz"] Sep 16 08:04:53.448757 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:53.448721 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:04:53.513252 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:53.513195 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-credential-socket\") pod \"e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz\" (UID: \"9f0e732a-2468-4ae9-a704-12ceb0ae4eff\") " pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" Sep 16 08:04:53.513252 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:53.513249 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-istio-envoy\") pod \"e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz\" (UID: \"9f0e732a-2468-4ae9-a704-12ceb0ae4eff\") " pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" Sep 16 08:04:53.513522 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:53.513282 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-workload-socket\") pod \"e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz\" (UID: \"9f0e732a-2468-4ae9-a704-12ceb0ae4eff\") " pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" Sep 16 08:04:53.513522 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:53.513360 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-istio-data\") pod \"e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz\" (UID: \"9f0e732a-2468-4ae9-a704-12ceb0ae4eff\") " pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" Sep 16 08:04:53.513522 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:53.513413 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvlkd\" (UniqueName: \"kubernetes.io/projected/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-kube-api-access-tvlkd\") pod \"e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz\" (UID: \"9f0e732a-2468-4ae9-a704-12ceb0ae4eff\") " pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" Sep 16 08:04:53.513522 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:53.513442 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-workload-certs\") pod \"e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz\" (UID: \"9f0e732a-2468-4ae9-a704-12ceb0ae4eff\") " pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" Sep 16 08:04:53.513522 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:53.513476 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-istiod-ca-cert\") pod \"e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz\" (UID: \"9f0e732a-2468-4ae9-a704-12ceb0ae4eff\") " pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" Sep 16 08:04:53.513705 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:53.513551 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-istio-token\") pod \"e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz\" (UID: \"9f0e732a-2468-4ae9-a704-12ceb0ae4eff\") " pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" Sep 16 08:04:53.513705 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:53.513579 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-istio-podinfo\") pod \"e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz\" (UID: \"9f0e732a-2468-4ae9-a704-12ceb0ae4eff\") " pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" Sep 16 08:04:53.614599 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:53.614552 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-istio-token\") pod \"e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz\" (UID: \"9f0e732a-2468-4ae9-a704-12ceb0ae4eff\") " pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" Sep 16 08:04:53.614599 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:53.614600 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-istio-podinfo\") pod \"e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz\" (UID: \"9f0e732a-2468-4ae9-a704-12ceb0ae4eff\") " pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" Sep 16 08:04:53.614880 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:53.614628 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-credential-socket\") pod \"e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz\" (UID: \"9f0e732a-2468-4ae9-a704-12ceb0ae4eff\") " pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" Sep 16 08:04:53.614880 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:53.614693 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-istio-envoy\") pod \"e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz\" (UID: \"9f0e732a-2468-4ae9-a704-12ceb0ae4eff\") " pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" Sep 16 08:04:53.614880 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:53.614739 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-workload-socket\") pod \"e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz\" (UID: \"9f0e732a-2468-4ae9-a704-12ceb0ae4eff\") " pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" Sep 16 08:04:53.614880 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:53.614784 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-istio-data\") pod \"e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz\" (UID: \"9f0e732a-2468-4ae9-a704-12ceb0ae4eff\") " pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" Sep 16 08:04:53.614880 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:53.614854 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvlkd\" (UniqueName: \"kubernetes.io/projected/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-kube-api-access-tvlkd\") pod \"e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz\" (UID: \"9f0e732a-2468-4ae9-a704-12ceb0ae4eff\") " pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" Sep 16 08:04:53.615161 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:53.614885 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-workload-certs\") pod \"e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz\" (UID: \"9f0e732a-2468-4ae9-a704-12ceb0ae4eff\") " pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" Sep 16 08:04:53.615161 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:53.614944 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-istiod-ca-cert\") pod \"e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz\" (UID: \"9f0e732a-2468-4ae9-a704-12ceb0ae4eff\") " pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" Sep 16 08:04:53.615280 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:53.615167 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-workload-socket\") pod \"e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz\" (UID: \"9f0e732a-2468-4ae9-a704-12ceb0ae4eff\") " pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" Sep 16 08:04:53.615280 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:53.615268 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-istio-data\") pod \"e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz\" (UID: \"9f0e732a-2468-4ae9-a704-12ceb0ae4eff\") " pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" Sep 16 08:04:53.615394 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:53.615299 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-credential-socket\") pod \"e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz\" (UID: \"9f0e732a-2468-4ae9-a704-12ceb0ae4eff\") " pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" Sep 16 08:04:53.615394 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:53.615358 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-workload-certs\") pod \"e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz\" (UID: \"9f0e732a-2468-4ae9-a704-12ceb0ae4eff\") " pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" Sep 16 08:04:53.615673 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:53.615652 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-istiod-ca-cert\") pod \"e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz\" (UID: \"9f0e732a-2468-4ae9-a704-12ceb0ae4eff\") " pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" Sep 16 08:04:53.617061 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:53.617042 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-istio-envoy\") pod \"e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz\" (UID: \"9f0e732a-2468-4ae9-a704-12ceb0ae4eff\") " pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" Sep 16 08:04:53.617200 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:53.617184 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-istio-podinfo\") pod \"e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz\" (UID: \"9f0e732a-2468-4ae9-a704-12ceb0ae4eff\") " pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" Sep 16 08:04:53.622550 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:53.622522 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-istio-token\") pod \"e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz\" (UID: \"9f0e732a-2468-4ae9-a704-12ceb0ae4eff\") " pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" Sep 16 08:04:53.622934 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:53.622913 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvlkd\" (UniqueName: \"kubernetes.io/projected/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-kube-api-access-tvlkd\") pod \"e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz\" (UID: \"9f0e732a-2468-4ae9-a704-12ceb0ae4eff\") " pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" Sep 16 08:04:53.730719 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:53.730675 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" Sep 16 08:04:53.794940 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:53.794859 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:04:53.935298 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:53.935261 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:04:54.010401 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:54.010373 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz"] Sep 16 08:04:54.013395 ip-10-0-142-170 kubenswrapper[2552]: W0916 08:04:54.013362 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f0e732a_2468_4ae9_a704_12ceb0ae4eff.slice/crio-586149e653e6fae15e5fab7960f7588c8e519e54108669004422017932c7a024 WatchSource:0}: Error finding container 586149e653e6fae15e5fab7960f7588c8e519e54108669004422017932c7a024: Status 404 returned error can't find the container with id 586149e653e6fae15e5fab7960f7588c8e519e54108669004422017932c7a024 Sep 16 08:04:54.016011 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:54.015967 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 16 08:04:54.016151 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:54.016062 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 16 08:04:54.016151 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:54.016125 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 16 08:04:54.459270 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:54.459235 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:04:54.590773 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:54.590733 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" event={"ID":"9f0e732a-2468-4ae9-a704-12ceb0ae4eff","Type":"ContainerStarted","Data":"2c0ce00a8bdff1ab5f1038f6cf7d652bfed7e731cd6d191c2db2510d06dba874"} Sep 16 08:04:54.590952 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:54.590788 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" event={"ID":"9f0e732a-2468-4ae9-a704-12ceb0ae4eff","Type":"ContainerStarted","Data":"586149e653e6fae15e5fab7960f7588c8e519e54108669004422017932c7a024"} Sep 16 08:04:54.610013 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:54.609954 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" podStartSLOduration=1.6099319410000001 podStartE2EDuration="1.609931941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 08:04:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 08:04:54.607931386 +0000 UTC m=+1866.191709723" watchObservedRunningTime="2026-09-16 08:04:54.609931941 +0000 UTC m=+1866.193710282" Sep 16 08:04:54.731245 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:54.731154 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" Sep 16 08:04:55.737553 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:55.737515 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" Sep 16 08:04:55.849092 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:55.849049 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:04:56.602229 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:56.602198 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" Sep 16 08:04:56.603461 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:56.603431 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" Sep 16 08:04:56.660968 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:56.660919 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:04:56.913333 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:56.913247 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:04:57.533579 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:57.533545 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:04:58.379399 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:58.379360 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:04:59.424558 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:59.424507 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:04:59.525179 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:59.525139 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:04:59.625269 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:59.625237 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:04:59.703371 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:04:59.703266 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:05:15.715919 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:15.715885 2552 generic.go:356] "Generic (PLEG): container finished" podID="9f0e732a-2468-4ae9-a704-12ceb0ae4eff" containerID="2c0ce00a8bdff1ab5f1038f6cf7d652bfed7e731cd6d191c2db2510d06dba874" exitCode=0 Sep 16 08:05:15.716402 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:15.715950 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" event={"ID":"9f0e732a-2468-4ae9-a704-12ceb0ae4eff","Type":"ContainerDied","Data":"2c0ce00a8bdff1ab5f1038f6cf7d652bfed7e731cd6d191c2db2510d06dba874"} Sep 16 08:05:15.716457 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:15.716411 2552 scope.go:117] "RemoveContainer" containerID="2c0ce00a8bdff1ab5f1038f6cf7d652bfed7e731cd6d191c2db2510d06dba874" Sep 16 08:05:15.717470 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:15.717443 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 16 08:05:15.717547 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:15.717512 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 16 08:05:15.717589 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:15.717546 2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"} Sep 16 08:05:16.723729 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:16.723685 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" event={"ID":"9f0e732a-2468-4ae9-a704-12ceb0ae4eff","Type":"ContainerStarted","Data":"3e286e6f2b01669680e8d109e3612bc478a780a13a6d61b9cbb6a121feb6e22e"} Sep 16 08:05:16.731421 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:16.731390 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" Sep 16 08:05:16.731620 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:16.731595 2552 patch_prober.go:28] interesting pod/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 16 08:05:16.731699 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:16.731675 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" podUID="9f0e732a-2468-4ae9-a704-12ceb0ae4eff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 16 08:05:17.318200 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:17.318159 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:05:17.370929 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:17.370885 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:05:17.731135 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:17.731086 2552 patch_prober.go:28] interesting pod/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 16 08:05:17.731552 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:17.731167 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" podUID="9f0e732a-2468-4ae9-a704-12ceb0ae4eff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 16 08:05:18.731167 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:18.731091 2552 patch_prober.go:28] interesting pod/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 16 08:05:18.731565 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:18.731200 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" podUID="9f0e732a-2468-4ae9-a704-12ceb0ae4eff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 16 08:05:19.732089 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:19.732046 2552 patch_prober.go:28] interesting pod/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 16 08:05:19.732497 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:19.732140 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" podUID="9f0e732a-2468-4ae9-a704-12ceb0ae4eff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 16 08:05:20.731731 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:20.731699 2552 patch_prober.go:28] interesting pod/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 16 08:05:20.731939 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:20.731822 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" podUID="9f0e732a-2468-4ae9-a704-12ceb0ae4eff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 16 08:05:21.173256 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:21.173212 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:05:21.628490 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:21.628402 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:05:21.731335 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:21.731294 2552 patch_prober.go:28] interesting pod/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" start-of-body= Sep 16 08:05:21.731520 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:21.731355 2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" podUID="9f0e732a-2468-4ae9-a704-12ceb0ae4eff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.70:15021/healthz/ready\": dial tcp 10.133.0.70:15021: connect: connection refused" Sep 16 08:05:21.838333 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:21.838281 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:05:22.473765 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:22.473732 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:05:22.735705 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:22.735613 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" Sep 16 08:05:22.735927 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:22.735899 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" Sep 16 08:05:22.736673 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:22.736656 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" Sep 16 08:05:23.371590 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:23.371553 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:05:24.005593 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:24.005548 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:05:34.062949 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:34.062906 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-796e11/ipp-route-796e11-kserve-66fb6897f5-wq275"] Sep 16 08:05:34.066573 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:34.066545 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-796e11/ipp-route-796e11-kserve-66fb6897f5-wq275" Sep 16 08:05:34.068983 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:34.068949 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-796e11\"/\"kube-root-ca.crt\"" Sep 16 08:05:34.069829 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:34.069802 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-796e11\"/\"default-dockercfg-4hdvs\"" Sep 16 08:05:34.070044 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:34.070009 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-796e11\"/\"ipp-route-796e11-kserve-self-signed-certs\"" Sep 16 08:05:34.070185 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:34.070044 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-796e11\"/\"openshift-service-ca.crt\"" Sep 16 08:05:34.076265 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:34.076233 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-796e11/ipp-route-796e11-kserve-66fb6897f5-wq275"] Sep 16 08:05:34.094037 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:34.093999 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:05:34.225862 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:34.225811 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zfnp\" (UniqueName: \"kubernetes.io/projected/d8929a09-cb89-4979-b219-e6d28df7cce8-kube-api-access-2zfnp\") pod \"ipp-route-796e11-kserve-66fb6897f5-wq275\" (UID: \"d8929a09-cb89-4979-b219-e6d28df7cce8\") " pod="ai-tenant-e2e-ipp-a-796e11/ipp-route-796e11-kserve-66fb6897f5-wq275" Sep 16 08:05:34.226067 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:34.225883 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8929a09-cb89-4979-b219-e6d28df7cce8-home\") pod \"ipp-route-796e11-kserve-66fb6897f5-wq275\" (UID: \"d8929a09-cb89-4979-b219-e6d28df7cce8\") " pod="ai-tenant-e2e-ipp-a-796e11/ipp-route-796e11-kserve-66fb6897f5-wq275" Sep 16 08:05:34.226067 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:34.225971 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8929a09-cb89-4979-b219-e6d28df7cce8-dshm\") pod \"ipp-route-796e11-kserve-66fb6897f5-wq275\" (UID: \"d8929a09-cb89-4979-b219-e6d28df7cce8\") " pod="ai-tenant-e2e-ipp-a-796e11/ipp-route-796e11-kserve-66fb6897f5-wq275" Sep 16 08:05:34.226235 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:34.226206 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8929a09-cb89-4979-b219-e6d28df7cce8-model-cache\") pod \"ipp-route-796e11-kserve-66fb6897f5-wq275\" (UID: \"d8929a09-cb89-4979-b219-e6d28df7cce8\") " pod="ai-tenant-e2e-ipp-a-796e11/ipp-route-796e11-kserve-66fb6897f5-wq275" Sep 16 08:05:34.226288 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:34.226249 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8929a09-cb89-4979-b219-e6d28df7cce8-tls-certs\") pod \"ipp-route-796e11-kserve-66fb6897f5-wq275\" (UID: \"d8929a09-cb89-4979-b219-e6d28df7cce8\") " pod="ai-tenant-e2e-ipp-a-796e11/ipp-route-796e11-kserve-66fb6897f5-wq275" Sep 16 08:05:34.226389 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:34.226373 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8929a09-cb89-4979-b219-e6d28df7cce8-tmp-dir\") pod \"ipp-route-796e11-kserve-66fb6897f5-wq275\" (UID: \"d8929a09-cb89-4979-b219-e6d28df7cce8\") " pod="ai-tenant-e2e-ipp-a-796e11/ipp-route-796e11-kserve-66fb6897f5-wq275" Sep 16 08:05:34.327785 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:34.327683 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8929a09-cb89-4979-b219-e6d28df7cce8-dshm\") pod \"ipp-route-796e11-kserve-66fb6897f5-wq275\" (UID: \"d8929a09-cb89-4979-b219-e6d28df7cce8\") " pod="ai-tenant-e2e-ipp-a-796e11/ipp-route-796e11-kserve-66fb6897f5-wq275" Sep 16 08:05:34.327785 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:34.327728 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8929a09-cb89-4979-b219-e6d28df7cce8-model-cache\") pod \"ipp-route-796e11-kserve-66fb6897f5-wq275\" (UID: \"d8929a09-cb89-4979-b219-e6d28df7cce8\") " pod="ai-tenant-e2e-ipp-a-796e11/ipp-route-796e11-kserve-66fb6897f5-wq275" Sep 16 08:05:34.327785 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:34.327749 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8929a09-cb89-4979-b219-e6d28df7cce8-tls-certs\") pod \"ipp-route-796e11-kserve-66fb6897f5-wq275\" (UID: \"d8929a09-cb89-4979-b219-e6d28df7cce8\") " pod="ai-tenant-e2e-ipp-a-796e11/ipp-route-796e11-kserve-66fb6897f5-wq275" Sep 16 08:05:34.327785 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:34.327782 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8929a09-cb89-4979-b219-e6d28df7cce8-tmp-dir\") pod \"ipp-route-796e11-kserve-66fb6897f5-wq275\" (UID: \"d8929a09-cb89-4979-b219-e6d28df7cce8\") " pod="ai-tenant-e2e-ipp-a-796e11/ipp-route-796e11-kserve-66fb6897f5-wq275" Sep 16 08:05:34.328194 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:34.327821 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zfnp\" (UniqueName: \"kubernetes.io/projected/d8929a09-cb89-4979-b219-e6d28df7cce8-kube-api-access-2zfnp\") pod \"ipp-route-796e11-kserve-66fb6897f5-wq275\" (UID: \"d8929a09-cb89-4979-b219-e6d28df7cce8\") " pod="ai-tenant-e2e-ipp-a-796e11/ipp-route-796e11-kserve-66fb6897f5-wq275" Sep 16 08:05:34.328194 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:34.327852 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8929a09-cb89-4979-b219-e6d28df7cce8-home\") pod \"ipp-route-796e11-kserve-66fb6897f5-wq275\" (UID: \"d8929a09-cb89-4979-b219-e6d28df7cce8\") " pod="ai-tenant-e2e-ipp-a-796e11/ipp-route-796e11-kserve-66fb6897f5-wq275" Sep 16 08:05:34.328303 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:34.328220 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8929a09-cb89-4979-b219-e6d28df7cce8-model-cache\") pod \"ipp-route-796e11-kserve-66fb6897f5-wq275\" (UID: \"d8929a09-cb89-4979-b219-e6d28df7cce8\") " pod="ai-tenant-e2e-ipp-a-796e11/ipp-route-796e11-kserve-66fb6897f5-wq275" Sep 16 08:05:34.328303 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:34.328275 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8929a09-cb89-4979-b219-e6d28df7cce8-tmp-dir\") pod \"ipp-route-796e11-kserve-66fb6897f5-wq275\" (UID: \"d8929a09-cb89-4979-b219-e6d28df7cce8\") " pod="ai-tenant-e2e-ipp-a-796e11/ipp-route-796e11-kserve-66fb6897f5-wq275" Sep 16 08:05:34.328372 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:34.328305 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8929a09-cb89-4979-b219-e6d28df7cce8-home\") pod \"ipp-route-796e11-kserve-66fb6897f5-wq275\" (UID: \"d8929a09-cb89-4979-b219-e6d28df7cce8\") " pod="ai-tenant-e2e-ipp-a-796e11/ipp-route-796e11-kserve-66fb6897f5-wq275" Sep 16 08:05:34.330300 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:34.330275 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8929a09-cb89-4979-b219-e6d28df7cce8-dshm\") pod \"ipp-route-796e11-kserve-66fb6897f5-wq275\" (UID: \"d8929a09-cb89-4979-b219-e6d28df7cce8\") " pod="ai-tenant-e2e-ipp-a-796e11/ipp-route-796e11-kserve-66fb6897f5-wq275" Sep 16 08:05:34.330486 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:34.330467 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8929a09-cb89-4979-b219-e6d28df7cce8-tls-certs\") pod \"ipp-route-796e11-kserve-66fb6897f5-wq275\" (UID: \"d8929a09-cb89-4979-b219-e6d28df7cce8\") " pod="ai-tenant-e2e-ipp-a-796e11/ipp-route-796e11-kserve-66fb6897f5-wq275" Sep 16 08:05:34.337297 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:34.337265 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zfnp\" (UniqueName: \"kubernetes.io/projected/d8929a09-cb89-4979-b219-e6d28df7cce8-kube-api-access-2zfnp\") pod \"ipp-route-796e11-kserve-66fb6897f5-wq275\" (UID: \"d8929a09-cb89-4979-b219-e6d28df7cce8\") " pod="ai-tenant-e2e-ipp-a-796e11/ipp-route-796e11-kserve-66fb6897f5-wq275" Sep 16 08:05:34.377499 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:34.377455 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-796e11/ipp-route-796e11-kserve-66fb6897f5-wq275" Sep 16 08:05:34.651307 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:34.651157 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-796e11/ipp-route-796e11-kserve-66fb6897f5-wq275"] Sep 16 08:05:34.654363 ip-10-0-142-170 kubenswrapper[2552]: W0916 08:05:34.654319 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd8929a09_cb89_4979_b219_e6d28df7cce8.slice/crio-bc0660f84e422d34ad3eeff01507b0dfc021f16c271822d3eba9a477acadb6a8 WatchSource:0}: Error finding container bc0660f84e422d34ad3eeff01507b0dfc021f16c271822d3eba9a477acadb6a8: Status 404 returned error can't find the container with id bc0660f84e422d34ad3eeff01507b0dfc021f16c271822d3eba9a477acadb6a8 Sep 16 08:05:34.830913 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:34.830880 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-796e11/ipp-route-796e11-kserve-66fb6897f5-wq275" event={"ID":"d8929a09-cb89-4979-b219-e6d28df7cce8","Type":"ContainerStarted","Data":"a5093b28cad6a4c15863d25c92d79ccfc46f8e58a53bc7e3cecfc30d15f291b1"} Sep 16 08:05:34.830913 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:34.830920 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-796e11/ipp-route-796e11-kserve-66fb6897f5-wq275" event={"ID":"d8929a09-cb89-4979-b219-e6d28df7cce8","Type":"ContainerStarted","Data":"bc0660f84e422d34ad3eeff01507b0dfc021f16c271822d3eba9a477acadb6a8"} Sep 16 08:05:34.849565 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:34.849498 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-796e11/ipp-route-796e11-kserve-66fb6897f5-wq275" podStartSLOduration=0.849471183 podStartE2EDuration="849.471183ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 08:05:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 08:05:34.846578349 +0000 UTC m=+1906.430356708" watchObservedRunningTime="2026-09-16 08:05:34.849471183 +0000 UTC m=+1906.433249523" Sep 16 08:05:44.378060 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:44.378010 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-796e11/ipp-route-796e11-kserve-66fb6897f5-wq275" Sep 16 08:05:44.378641 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:44.378074 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-796e11/ipp-route-796e11-kserve-66fb6897f5-wq275" Sep 16 08:05:44.391690 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:44.391652 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-796e11/ipp-route-796e11-kserve-66fb6897f5-wq275" Sep 16 08:05:44.897378 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:44.897346 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-796e11/ipp-route-796e11-kserve-66fb6897f5-wq275" Sep 16 08:05:49.215785 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:49.215754 2552 scope.go:117] "RemoveContainer" containerID="4e0c57bc062d0ba93a6cca1b254b5b6a0272b66e6d43421f06ce081534f2f497" Sep 16 08:05:53.563529 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:53.563496 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:05:53.617164 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:05:53.617128 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:06:03.505907 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:03.505870 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:06:03.803143 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:03.803031 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:06:03.932847 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:03.932806 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:06:06.674402 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:06.674364 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:06:06.877474 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:06.877438 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:06:06.970940 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:06.970850 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:06:07.582812 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:07.582775 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:06:08.472445 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:08.472412 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:06:09.209113 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:09.209069 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:06:09.864935 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:09.864892 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:06:09.931372 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:09.931331 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:06:10.029964 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:10.029848 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:06:10.306599 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:10.306528 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:06:10.406087 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:10.406045 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:06:10.407059 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:10.407036 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d26a85-6bd488c94b-pqc4q"] Sep 16 08:06:10.410744 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:10.410722 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d26a85-6bd488c94b-pqc4q" Sep 16 08:06:10.412949 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:10.412929 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-d26a85-dockercfg-z956z\"" Sep 16 08:06:10.413018 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:10.412930 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-d26a85\"" Sep 16 08:06:10.421895 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:10.421865 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d26a85-6bd488c94b-pqc4q"] Sep 16 08:06:10.566789 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:10.566685 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckcd7\" (UniqueName: \"kubernetes.io/projected/10611b7d-e979-4b16-b87f-0c0fd083899e-kube-api-access-ckcd7\") pod \"payload-pre-processing-e2e-ipp-cleanup-d26a85-6bd488c94b-pqc4q\" (UID: \"10611b7d-e979-4b16-b87f-0c0fd083899e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d26a85-6bd488c94b-pqc4q" Sep 16 08:06:10.566991 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:10.566838 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10611b7d-e979-4b16-b87f-0c0fd083899e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-d26a85-6bd488c94b-pqc4q\" (UID: \"10611b7d-e979-4b16-b87f-0c0fd083899e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d26a85-6bd488c94b-pqc4q" Sep 16 08:06:10.668276 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:10.668234 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckcd7\" (UniqueName: \"kubernetes.io/projected/10611b7d-e979-4b16-b87f-0c0fd083899e-kube-api-access-ckcd7\") pod \"payload-pre-processing-e2e-ipp-cleanup-d26a85-6bd488c94b-pqc4q\" (UID: \"10611b7d-e979-4b16-b87f-0c0fd083899e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d26a85-6bd488c94b-pqc4q" Sep 16 08:06:10.668464 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:10.668337 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10611b7d-e979-4b16-b87f-0c0fd083899e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-d26a85-6bd488c94b-pqc4q\" (UID: \"10611b7d-e979-4b16-b87f-0c0fd083899e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d26a85-6bd488c94b-pqc4q" Sep 16 08:06:10.669050 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:10.669025 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10611b7d-e979-4b16-b87f-0c0fd083899e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-d26a85-6bd488c94b-pqc4q\" (UID: \"10611b7d-e979-4b16-b87f-0c0fd083899e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d26a85-6bd488c94b-pqc4q" Sep 16 08:06:10.676266 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:10.676235 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckcd7\" (UniqueName: \"kubernetes.io/projected/10611b7d-e979-4b16-b87f-0c0fd083899e-kube-api-access-ckcd7\") pod \"payload-pre-processing-e2e-ipp-cleanup-d26a85-6bd488c94b-pqc4q\" (UID: \"10611b7d-e979-4b16-b87f-0c0fd083899e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d26a85-6bd488c94b-pqc4q" Sep 16 08:06:10.722214 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:10.722172 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d26a85-6bd488c94b-pqc4q" Sep 16 08:06:10.775075 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:10.775014 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:06:10.997815 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:10.997786 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d26a85-6bd488c94b-pqc4q"] Sep 16 08:06:11.000157 ip-10-0-142-170 kubenswrapper[2552]: W0916 08:06:11.000093 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10611b7d_e979_4b16_b87f_0c0fd083899e.slice/crio-c555896248b9321eb6961a67d05dfb18720cbe4db4c1c488bd7a6c1a2c55632a WatchSource:0}: Error finding container c555896248b9321eb6961a67d05dfb18720cbe4db4c1c488bd7a6c1a2c55632a: Status 404 returned error can't find the container with id c555896248b9321eb6961a67d05dfb18720cbe4db4c1c488bd7a6c1a2c55632a Sep 16 08:06:11.034509 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:11.034476 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d26a85-6bd488c94b-pqc4q" event={"ID":"10611b7d-e979-4b16-b87f-0c0fd083899e","Type":"ContainerStarted","Data":"c555896248b9321eb6961a67d05dfb18720cbe4db4c1c488bd7a6c1a2c55632a"} Sep 16 08:06:11.414351 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:11.414262 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:06:12.046161 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:12.046116 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d26a85-6bd488c94b-pqc4q" event={"ID":"10611b7d-e979-4b16-b87f-0c0fd083899e","Type":"ContainerStarted","Data":"471f25054ec201375081c5dbdd0f476582230aacf21b7265c31d59103bbc0ecc"} Sep 16 08:06:12.046653 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:12.046243 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d26a85-6bd488c94b-pqc4q" Sep 16 08:06:12.062884 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:12.062828 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d26a85-6bd488c94b-pqc4q" podStartSLOduration=2.062813378 podStartE2EDuration="2.062813378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 08:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 08:06:12.061302583 +0000 UTC m=+1943.645080922" watchObservedRunningTime="2026-09-16 08:06:12.062813378 +0000 UTC m=+1943.646591715" Sep 16 08:06:23.054557 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:23.054526 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d26a85-6bd488c94b-pqc4q" Sep 16 08:06:31.636805 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:31.636774 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:06:31.910989 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:31.910904 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:06:32.366568 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:32.366529 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:06:32.655548 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:32.655463 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d26a85-xk2qr"] Sep 16 08:06:32.659042 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:32.659021 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d26a85-xk2qr" Sep 16 08:06:32.661323 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:32.661301 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 08:06:32.661457 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:32.661366 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zwk5r\"" Sep 16 08:06:32.668304 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:32.668275 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d26a85-xk2qr"] Sep 16 08:06:32.790233 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:32.790194 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f4b9791-3964-46e0-8b08-4d66c5d73734-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-d26a85-xk2qr\" (UID: \"9f4b9791-3964-46e0-8b08-4d66c5d73734\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d26a85-xk2qr" Sep 16 08:06:32.891551 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:32.891508 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f4b9791-3964-46e0-8b08-4d66c5d73734-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-d26a85-xk2qr\" (UID: \"9f4b9791-3964-46e0-8b08-4d66c5d73734\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d26a85-xk2qr" Sep 16 08:06:32.892175 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:32.892154 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f4b9791-3964-46e0-8b08-4d66c5d73734-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-d26a85-xk2qr\" (UID: \"9f4b9791-3964-46e0-8b08-4d66c5d73734\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d26a85-xk2qr" Sep 16 08:06:32.971334 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:32.971286 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d26a85-xk2qr" Sep 16 08:06:33.230740 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:33.230656 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:06:33.243166 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:33.243140 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d26a85-xk2qr"] Sep 16 08:06:33.245404 ip-10-0-142-170 kubenswrapper[2552]: W0916 08:06:33.245379 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f4b9791_3964_46e0_8b08_4d66c5d73734.slice/crio-c20eb64bc647be6cb0452cb3897dc5533441d8acd240906ac12e317a14ea1e42 WatchSource:0}: Error finding container c20eb64bc647be6cb0452cb3897dc5533441d8acd240906ac12e317a14ea1e42: Status 404 returned error can't find the container with id c20eb64bc647be6cb0452cb3897dc5533441d8acd240906ac12e317a14ea1e42 Sep 16 08:06:33.820592 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:33.820498 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:06:34.175256 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:34.175222 2552 generic.go:356] "Generic (PLEG): container finished" podID="9f4b9791-3964-46e0-8b08-4d66c5d73734" containerID="f794b17cb75d0f9ef584231378df43b7a3f3210390adf99cfb398c2787f253eb" exitCode=0 Sep 16 08:06:34.175465 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:34.175305 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d26a85-xk2qr" event={"ID":"9f4b9791-3964-46e0-8b08-4d66c5d73734","Type":"ContainerDied","Data":"f794b17cb75d0f9ef584231378df43b7a3f3210390adf99cfb398c2787f253eb"} Sep 16 08:06:34.175465 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:34.175364 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d26a85-xk2qr" event={"ID":"9f4b9791-3964-46e0-8b08-4d66c5d73734","Type":"ContainerStarted","Data":"c20eb64bc647be6cb0452cb3897dc5533441d8acd240906ac12e317a14ea1e42"} Sep 16 08:06:34.285765 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:34.285729 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:06:34.559718 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:34.559626 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:06:35.197926 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:35.197878 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:06:35.319820 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:35.319792 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d26a85-xk2qr" Sep 16 08:06:35.416837 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:35.416793 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f4b9791-3964-46e0-8b08-4d66c5d73734-maas-api-service-ca\") pod \"9f4b9791-3964-46e0-8b08-4d66c5d73734\" (UID: \"9f4b9791-3964-46e0-8b08-4d66c5d73734\") " Sep 16 08:06:35.417219 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:35.417197 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f4b9791-3964-46e0-8b08-4d66c5d73734-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9f4b9791-3964-46e0-8b08-4d66c5d73734" (UID: "9f4b9791-3964-46e0-8b08-4d66c5d73734"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 08:06:35.518548 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:35.518441 2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f4b9791-3964-46e0-8b08-4d66c5d73734-maas-api-service-ca\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:06:35.861934 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:35.861813 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:06:36.187697 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:36.187669 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d26a85-xk2qr" Sep 16 08:06:36.187898 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:36.187704 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d26a85-xk2qr" event={"ID":"9f4b9791-3964-46e0-8b08-4d66c5d73734","Type":"ContainerDied","Data":"c20eb64bc647be6cb0452cb3897dc5533441d8acd240906ac12e317a14ea1e42"} Sep 16 08:06:36.187898 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:36.187750 2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c20eb64bc647be6cb0452cb3897dc5533441d8acd240906ac12e317a14ea1e42" Sep 16 08:06:36.761370 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:36.761331 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:06:38.179400 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:38.179372 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:06:38.953261 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:38.953224 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:06:39.504774 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:39.504731 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:06:39.540678 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:39.540642 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d26a85-6bd488c94b-pqc4q"] Sep 16 08:06:39.540961 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:39.540929 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d26a85-6bd488c94b-pqc4q" podUID="10611b7d-e979-4b16-b87f-0c0fd083899e" containerName="payload-pre-processing" containerID="cri-o://471f25054ec201375081c5dbdd0f476582230aacf21b7265c31d59103bbc0ecc" gracePeriod=30 Sep 16 08:06:39.572148 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:39.572088 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:06:39.672440 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:39.672400 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:06:39.683425 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:39.683358 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d26a85-xk2qr"] Sep 16 08:06:39.685828 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:39.685799 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d26a85-xk2qr"] Sep 16 08:06:39.737864 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:39.737827 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:06:39.804075 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:39.804043 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d26a85-6bd488c94b-pqc4q" Sep 16 08:06:39.961072 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:39.961024 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10611b7d-e979-4b16-b87f-0c0fd083899e-plugins-config-volume\") pod \"10611b7d-e979-4b16-b87f-0c0fd083899e\" (UID: \"10611b7d-e979-4b16-b87f-0c0fd083899e\") " Sep 16 08:06:39.961302 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:39.961158 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ckcd7\" (UniqueName: \"kubernetes.io/projected/10611b7d-e979-4b16-b87f-0c0fd083899e-kube-api-access-ckcd7\") pod \"10611b7d-e979-4b16-b87f-0c0fd083899e\" (UID: \"10611b7d-e979-4b16-b87f-0c0fd083899e\") " Sep 16 08:06:39.961542 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:39.961500 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10611b7d-e979-4b16-b87f-0c0fd083899e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "10611b7d-e979-4b16-b87f-0c0fd083899e" (UID: "10611b7d-e979-4b16-b87f-0c0fd083899e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 08:06:39.963605 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:39.963570 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10611b7d-e979-4b16-b87f-0c0fd083899e-kube-api-access-ckcd7" (OuterVolumeSpecName: "kube-api-access-ckcd7") pod "10611b7d-e979-4b16-b87f-0c0fd083899e" (UID: "10611b7d-e979-4b16-b87f-0c0fd083899e"). InnerVolumeSpecName "kube-api-access-ckcd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 08:06:40.062171 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:40.062062 2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10611b7d-e979-4b16-b87f-0c0fd083899e-plugins-config-volume\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:06:40.062171 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:40.062126 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckcd7\" (UniqueName: \"kubernetes.io/projected/10611b7d-e979-4b16-b87f-0c0fd083899e-kube-api-access-ckcd7\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:06:40.151843 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:40.151808 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:06:40.210341 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:40.210304 2552 generic.go:356] "Generic (PLEG): container finished" podID="10611b7d-e979-4b16-b87f-0c0fd083899e" containerID="471f25054ec201375081c5dbdd0f476582230aacf21b7265c31d59103bbc0ecc" exitCode=0 Sep 16 08:06:40.210546 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:40.210347 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d26a85-6bd488c94b-pqc4q" Sep 16 08:06:40.210546 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:40.210391 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d26a85-6bd488c94b-pqc4q" event={"ID":"10611b7d-e979-4b16-b87f-0c0fd083899e","Type":"ContainerDied","Data":"471f25054ec201375081c5dbdd0f476582230aacf21b7265c31d59103bbc0ecc"} Sep 16 08:06:40.210546 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:40.210450 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d26a85-6bd488c94b-pqc4q" event={"ID":"10611b7d-e979-4b16-b87f-0c0fd083899e","Type":"ContainerDied","Data":"c555896248b9321eb6961a67d05dfb18720cbe4db4c1c488bd7a6c1a2c55632a"} Sep 16 08:06:40.210546 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:40.210482 2552 scope.go:117] "RemoveContainer" containerID="471f25054ec201375081c5dbdd0f476582230aacf21b7265c31d59103bbc0ecc" Sep 16 08:06:40.222571 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:40.222543 2552 scope.go:117] "RemoveContainer" containerID="471f25054ec201375081c5dbdd0f476582230aacf21b7265c31d59103bbc0ecc" Sep 16 08:06:40.223152 ip-10-0-142-170 kubenswrapper[2552]: E0916 08:06:40.223092 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"471f25054ec201375081c5dbdd0f476582230aacf21b7265c31d59103bbc0ecc\": container with ID starting with 471f25054ec201375081c5dbdd0f476582230aacf21b7265c31d59103bbc0ecc not found: ID does not exist" containerID="471f25054ec201375081c5dbdd0f476582230aacf21b7265c31d59103bbc0ecc" Sep 16 08:06:40.223248 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:40.223173 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"471f25054ec201375081c5dbdd0f476582230aacf21b7265c31d59103bbc0ecc"} err="failed to get container status \"471f25054ec201375081c5dbdd0f476582230aacf21b7265c31d59103bbc0ecc\": rpc error: code = NotFound desc = could not find container \"471f25054ec201375081c5dbdd0f476582230aacf21b7265c31d59103bbc0ecc\": container with ID starting with 471f25054ec201375081c5dbdd0f476582230aacf21b7265c31d59103bbc0ecc not found: ID does not exist" Sep 16 08:06:40.232318 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:40.232284 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d26a85-6bd488c94b-pqc4q"] Sep 16 08:06:40.237480 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:40.237450 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:06:40.238688 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:40.238661 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d26a85-6bd488c94b-pqc4q"] Sep 16 08:06:40.827292 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:40.827258 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:06:40.938293 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:40.938259 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10611b7d-e979-4b16-b87f-0c0fd083899e" path="/var/lib/kubelet/pods/10611b7d-e979-4b16-b87f-0c0fd083899e/volumes" Sep 16 08:06:40.938640 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:40.938627 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f4b9791-3964-46e0-8b08-4d66c5d73734" path="/var/lib/kubelet/pods/9f4b9791-3964-46e0-8b08-4d66c5d73734/volumes" Sep 16 08:06:41.565201 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:41.565167 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:06:42.253053 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:42.253019 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:06:42.961518 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:42.961480 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:06:46.093880 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:46.093841 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:06:46.193862 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:46.193828 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:06:46.361418 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:46.361315 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:06:49.237577 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:49.237542 2552 scope.go:117] "RemoveContainer" containerID="9f93f0cdc7b7d0931eb9451536140b4d6c45e9f6029a746345488803ddda591c" Sep 16 08:06:49.247184 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:49.247162 2552 scope.go:117] "RemoveContainer" containerID="b2e6255bde8ad7a14ae9d094ed5e73dedb804df196bfb88b95a08ba6bfc25cf4" Sep 16 08:06:49.256596 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:49.256568 2552 scope.go:117] "RemoveContainer" containerID="fa12e4d70b9fcba336784f618f2ee6f8ce119cf150243401aea8dc99b86b129e" Sep 16 08:06:49.265997 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:49.265967 2552 scope.go:117] "RemoveContainer" containerID="dc4c13ff0b9ef5933c7620c8421f05300d1e1e1c07e1099b8179fa9db4fa1d93" Sep 16 08:06:49.274242 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:49.274221 2552 scope.go:117] "RemoveContainer" containerID="123893790cdcd6a287121c784ff5e7e5d0165d88a8c72623dc69d0f190aad4c1" Sep 16 08:06:49.282371 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:49.282350 2552 scope.go:117] "RemoveContainer" containerID="40c57cd1272186ad3b83cb1fe3b51e92a1924bb31d8f16989c3c94dc8019f5b8" Sep 16 08:06:50.364066 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:50.364007 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:06:51.205884 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:51.205831 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:06:51.977459 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:51.977425 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:06:52.623596 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:52.623566 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:06:53.252608 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:53.252569 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:06:54.140419 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:06:54.140382 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:07:05.696606 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:05.696569 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:07:05.768077 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:05.768012 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:07:05.837644 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:05.837612 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:07:05.905311 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:05.905280 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:07:06.279631 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:06.279590 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:07:06.348169 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:06.348127 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:07:07.180620 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:07.180556 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:07:07.744965 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:07.744924 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:07:08.182552 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:08.182511 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:07:08.487081 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:08.486984 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:07:11.273554 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:11.273514 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-796e11/ipp-route-796e11-kserve-66fb6897f5-wq275"] Sep 16 08:07:11.323022 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:11.322982 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:07:26.293694 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:26.293627 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-796e11/ipp-route-796e11-kserve-66fb6897f5-wq275" podUID="d8929a09-cb89-4979-b219-e6d28df7cce8" containerName="main" containerID="cri-o://a5093b28cad6a4c15863d25c92d79ccfc46f8e58a53bc7e3cecfc30d15f291b1" gracePeriod=45 Sep 16 08:07:26.467306 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:26.467273 2552 generic.go:356] "Generic (PLEG): container finished" podID="d8929a09-cb89-4979-b219-e6d28df7cce8" containerID="a5093b28cad6a4c15863d25c92d79ccfc46f8e58a53bc7e3cecfc30d15f291b1" exitCode=0 Sep 16 08:07:26.467489 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:26.467344 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-796e11/ipp-route-796e11-kserve-66fb6897f5-wq275" event={"ID":"d8929a09-cb89-4979-b219-e6d28df7cce8","Type":"ContainerDied","Data":"a5093b28cad6a4c15863d25c92d79ccfc46f8e58a53bc7e3cecfc30d15f291b1"} Sep 16 08:07:26.557386 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:26.557309 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-796e11/ipp-route-796e11-kserve-66fb6897f5-wq275" Sep 16 08:07:26.713367 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:26.713324 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8929a09-cb89-4979-b219-e6d28df7cce8-dshm\") pod \"d8929a09-cb89-4979-b219-e6d28df7cce8\" (UID: \"d8929a09-cb89-4979-b219-e6d28df7cce8\") " Sep 16 08:07:26.713367 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:26.713373 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8929a09-cb89-4979-b219-e6d28df7cce8-model-cache\") pod \"d8929a09-cb89-4979-b219-e6d28df7cce8\" (UID: \"d8929a09-cb89-4979-b219-e6d28df7cce8\") " Sep 16 08:07:26.713631 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:26.713407 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2zfnp\" (UniqueName: \"kubernetes.io/projected/d8929a09-cb89-4979-b219-e6d28df7cce8-kube-api-access-2zfnp\") pod \"d8929a09-cb89-4979-b219-e6d28df7cce8\" (UID: \"d8929a09-cb89-4979-b219-e6d28df7cce8\") " Sep 16 08:07:26.713631 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:26.713439 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8929a09-cb89-4979-b219-e6d28df7cce8-tmp-dir\") pod \"d8929a09-cb89-4979-b219-e6d28df7cce8\" (UID: \"d8929a09-cb89-4979-b219-e6d28df7cce8\") " Sep 16 08:07:26.713631 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:26.713486 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8929a09-cb89-4979-b219-e6d28df7cce8-tls-certs\") pod \"d8929a09-cb89-4979-b219-e6d28df7cce8\" (UID: \"d8929a09-cb89-4979-b219-e6d28df7cce8\") " Sep 16 08:07:26.713631 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:26.713503 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8929a09-cb89-4979-b219-e6d28df7cce8-home\") pod \"d8929a09-cb89-4979-b219-e6d28df7cce8\" (UID: \"d8929a09-cb89-4979-b219-e6d28df7cce8\") " Sep 16 08:07:26.713843 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:26.713706 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8929a09-cb89-4979-b219-e6d28df7cce8-model-cache" (OuterVolumeSpecName: "model-cache") pod "d8929a09-cb89-4979-b219-e6d28df7cce8" (UID: "d8929a09-cb89-4979-b219-e6d28df7cce8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:07:26.713843 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:26.713802 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8929a09-cb89-4979-b219-e6d28df7cce8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d8929a09-cb89-4979-b219-e6d28df7cce8" (UID: "d8929a09-cb89-4979-b219-e6d28df7cce8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:07:26.713937 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:26.713916 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8929a09-cb89-4979-b219-e6d28df7cce8-home" (OuterVolumeSpecName: "home") pod "d8929a09-cb89-4979-b219-e6d28df7cce8" (UID: "d8929a09-cb89-4979-b219-e6d28df7cce8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:07:26.715819 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:26.715791 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8929a09-cb89-4979-b219-e6d28df7cce8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d8929a09-cb89-4979-b219-e6d28df7cce8" (UID: "d8929a09-cb89-4979-b219-e6d28df7cce8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 08:07:26.715819 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:26.715801 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8929a09-cb89-4979-b219-e6d28df7cce8-dshm" (OuterVolumeSpecName: "dshm") pod "d8929a09-cb89-4979-b219-e6d28df7cce8" (UID: "d8929a09-cb89-4979-b219-e6d28df7cce8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:07:26.715986 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:26.715805 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8929a09-cb89-4979-b219-e6d28df7cce8-kube-api-access-2zfnp" (OuterVolumeSpecName: "kube-api-access-2zfnp") pod "d8929a09-cb89-4979-b219-e6d28df7cce8" (UID: "d8929a09-cb89-4979-b219-e6d28df7cce8"). InnerVolumeSpecName "kube-api-access-2zfnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 08:07:26.814617 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:26.814520 2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d8929a09-cb89-4979-b219-e6d28df7cce8-dshm\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:07:26.814617 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:26.814555 2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d8929a09-cb89-4979-b219-e6d28df7cce8-model-cache\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:07:26.814617 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:26.814570 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zfnp\" (UniqueName: \"kubernetes.io/projected/d8929a09-cb89-4979-b219-e6d28df7cce8-kube-api-access-2zfnp\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:07:26.814617 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:26.814580 2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d8929a09-cb89-4979-b219-e6d28df7cce8-tmp-dir\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:07:26.814617 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:26.814588 2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8929a09-cb89-4979-b219-e6d28df7cce8-tls-certs\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:07:26.814617 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:26.814596 2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d8929a09-cb89-4979-b219-e6d28df7cce8-home\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:07:27.474656 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:27.474625 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-796e11/ipp-route-796e11-kserve-66fb6897f5-wq275" Sep 16 08:07:27.475135 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:27.474673 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-796e11/ipp-route-796e11-kserve-66fb6897f5-wq275" event={"ID":"d8929a09-cb89-4979-b219-e6d28df7cce8","Type":"ContainerDied","Data":"bc0660f84e422d34ad3eeff01507b0dfc021f16c271822d3eba9a477acadb6a8"} Sep 16 08:07:27.475135 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:27.474719 2552 scope.go:117] "RemoveContainer" containerID="a5093b28cad6a4c15863d25c92d79ccfc46f8e58a53bc7e3cecfc30d15f291b1" Sep 16 08:07:27.490073 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:27.490036 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-796e11/ipp-route-796e11-kserve-66fb6897f5-wq275"] Sep 16 08:07:27.493337 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:27.493306 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-796e11/ipp-route-796e11-kserve-66fb6897f5-wq275"] Sep 16 08:07:28.937059 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:28.937019 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8929a09-cb89-4979-b219-e6d28df7cce8" path="/var/lib/kubelet/pods/d8929a09-cb89-4979-b219-e6d28df7cce8/volumes" Sep 16 08:07:47.497364 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:47.497304 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:07:48.268766 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:48.268726 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:07:53.806495 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:53.806462 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:07:53.907538 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:53.907500 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:07:54.146410 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:54.146303 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:07:54.591433 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:54.591399 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:07:55.339730 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:55.339693 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:07:56.067911 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:56.067870 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:07:56.800952 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:56.800916 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:07:57.528496 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:07:57.528467 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:08:04.915529 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:04.915490 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:08:04.983033 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:04.982987 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:08:05.043970 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:05.043923 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:08:05.105675 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:05.105640 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:08:05.519177 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:05.519143 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:08:05.620762 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:05.620723 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:08:06.359777 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:06.359741 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:08:07.001034 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:07.000983 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:08:11.122091 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:11.121996 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:08:11.195361 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:11.195322 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:08:11.373866 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:11.373755 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz"] Sep 16 08:08:11.374089 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:11.374031 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" podUID="9f0e732a-2468-4ae9-a704-12ceb0ae4eff" containerName="istio-proxy" containerID="cri-o://3e286e6f2b01669680e8d109e3612bc478a780a13a6d61b9cbb6a121feb6e22e" gracePeriod=30 Sep 16 08:08:11.397373 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:11.397336 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:08:11.989711 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:11.989671 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:08:12.792672 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:12.792637 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:08:13.415775 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:13.415739 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:08:14.155083 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:14.155045 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:08:16.655672 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:16.655583 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" Sep 16 08:08:16.733995 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:16.733961 2552 generic.go:356] "Generic (PLEG): container finished" podID="9f0e732a-2468-4ae9-a704-12ceb0ae4eff" containerID="3e286e6f2b01669680e8d109e3612bc478a780a13a6d61b9cbb6a121feb6e22e" exitCode=0 Sep 16 08:08:16.734202 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:16.734022 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" Sep 16 08:08:16.734202 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:16.734044 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" event={"ID":"9f0e732a-2468-4ae9-a704-12ceb0ae4eff","Type":"ContainerDied","Data":"3e286e6f2b01669680e8d109e3612bc478a780a13a6d61b9cbb6a121feb6e22e"} Sep 16 08:08:16.734202 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:16.734142 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz" event={"ID":"9f0e732a-2468-4ae9-a704-12ceb0ae4eff","Type":"ContainerDied","Data":"586149e653e6fae15e5fab7960f7588c8e519e54108669004422017932c7a024"} Sep 16 08:08:16.734202 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:16.734170 2552 scope.go:117] "RemoveContainer" containerID="3e286e6f2b01669680e8d109e3612bc478a780a13a6d61b9cbb6a121feb6e22e" Sep 16 08:08:16.742980 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:16.742952 2552 scope.go:117] "RemoveContainer" containerID="2c0ce00a8bdff1ab5f1038f6cf7d652bfed7e731cd6d191c2db2510d06dba874" Sep 16 08:08:16.751732 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:16.751704 2552 scope.go:117] "RemoveContainer" containerID="3e286e6f2b01669680e8d109e3612bc478a780a13a6d61b9cbb6a121feb6e22e" Sep 16 08:08:16.752146 ip-10-0-142-170 kubenswrapper[2552]: E0916 08:08:16.752118 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e286e6f2b01669680e8d109e3612bc478a780a13a6d61b9cbb6a121feb6e22e\": container with ID starting with 3e286e6f2b01669680e8d109e3612bc478a780a13a6d61b9cbb6a121feb6e22e not found: ID does not exist" containerID="3e286e6f2b01669680e8d109e3612bc478a780a13a6d61b9cbb6a121feb6e22e" Sep 16 08:08:16.752257 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:16.752155 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e286e6f2b01669680e8d109e3612bc478a780a13a6d61b9cbb6a121feb6e22e"} err="failed to get container status \"3e286e6f2b01669680e8d109e3612bc478a780a13a6d61b9cbb6a121feb6e22e\": rpc error: code = NotFound desc = could not find container \"3e286e6f2b01669680e8d109e3612bc478a780a13a6d61b9cbb6a121feb6e22e\": container with ID starting with 3e286e6f2b01669680e8d109e3612bc478a780a13a6d61b9cbb6a121feb6e22e not found: ID does not exist" Sep 16 08:08:16.752257 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:16.752175 2552 scope.go:117] "RemoveContainer" containerID="2c0ce00a8bdff1ab5f1038f6cf7d652bfed7e731cd6d191c2db2510d06dba874" Sep 16 08:08:16.752496 ip-10-0-142-170 kubenswrapper[2552]: E0916 08:08:16.752476 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c0ce00a8bdff1ab5f1038f6cf7d652bfed7e731cd6d191c2db2510d06dba874\": container with ID starting with 2c0ce00a8bdff1ab5f1038f6cf7d652bfed7e731cd6d191c2db2510d06dba874 not found: ID does not exist" containerID="2c0ce00a8bdff1ab5f1038f6cf7d652bfed7e731cd6d191c2db2510d06dba874" Sep 16 08:08:16.752541 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:16.752503 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c0ce00a8bdff1ab5f1038f6cf7d652bfed7e731cd6d191c2db2510d06dba874"} err="failed to get container status \"2c0ce00a8bdff1ab5f1038f6cf7d652bfed7e731cd6d191c2db2510d06dba874\": rpc error: code = NotFound desc = could not find container \"2c0ce00a8bdff1ab5f1038f6cf7d652bfed7e731cd6d191c2db2510d06dba874\": container with ID starting with 2c0ce00a8bdff1ab5f1038f6cf7d652bfed7e731cd6d191c2db2510d06dba874 not found: ID does not exist" Sep 16 08:08:16.791850 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:16.791808 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvlkd\" (UniqueName: \"kubernetes.io/projected/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-kube-api-access-tvlkd\") pod \"9f0e732a-2468-4ae9-a704-12ceb0ae4eff\" (UID: \"9f0e732a-2468-4ae9-a704-12ceb0ae4eff\") " Sep 16 08:08:16.792048 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:16.791864 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-workload-certs\") pod \"9f0e732a-2468-4ae9-a704-12ceb0ae4eff\" (UID: \"9f0e732a-2468-4ae9-a704-12ceb0ae4eff\") " Sep 16 08:08:16.792048 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:16.791887 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-credential-socket\") pod \"9f0e732a-2468-4ae9-a704-12ceb0ae4eff\" (UID: \"9f0e732a-2468-4ae9-a704-12ceb0ae4eff\") " Sep 16 08:08:16.792048 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:16.791950 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-istio-token\") pod \"9f0e732a-2468-4ae9-a704-12ceb0ae4eff\" (UID: \"9f0e732a-2468-4ae9-a704-12ceb0ae4eff\") " Sep 16 08:08:16.792048 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:16.791975 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-workload-socket\") pod \"9f0e732a-2468-4ae9-a704-12ceb0ae4eff\" (UID: \"9f0e732a-2468-4ae9-a704-12ceb0ae4eff\") " Sep 16 08:08:16.792048 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:16.792005 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-istio-podinfo\") pod \"9f0e732a-2468-4ae9-a704-12ceb0ae4eff\" (UID: \"9f0e732a-2468-4ae9-a704-12ceb0ae4eff\") " Sep 16 08:08:16.792356 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:16.792053 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-istiod-ca-cert\") pod \"9f0e732a-2468-4ae9-a704-12ceb0ae4eff\" (UID: \"9f0e732a-2468-4ae9-a704-12ceb0ae4eff\") " Sep 16 08:08:16.792356 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:16.792076 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-istio-envoy\") pod \"9f0e732a-2468-4ae9-a704-12ceb0ae4eff\" (UID: \"9f0e732a-2468-4ae9-a704-12ceb0ae4eff\") " Sep 16 08:08:16.792356 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:16.792119 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-istio-data\") pod \"9f0e732a-2468-4ae9-a704-12ceb0ae4eff\" (UID: \"9f0e732a-2468-4ae9-a704-12ceb0ae4eff\") " Sep 16 08:08:16.792356 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:16.792159 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9f0e732a-2468-4ae9-a704-12ceb0ae4eff" (UID: "9f0e732a-2468-4ae9-a704-12ceb0ae4eff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:08:16.792356 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:16.792297 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9f0e732a-2468-4ae9-a704-12ceb0ae4eff" (UID: "9f0e732a-2468-4ae9-a704-12ceb0ae4eff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:08:16.792621 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:16.792428 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-istio-data" (OuterVolumeSpecName: "istio-data") pod "9f0e732a-2468-4ae9-a704-12ceb0ae4eff" (UID: "9f0e732a-2468-4ae9-a704-12ceb0ae4eff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:08:16.792621 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:16.792510 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9f0e732a-2468-4ae9-a704-12ceb0ae4eff" (UID: "9f0e732a-2468-4ae9-a704-12ceb0ae4eff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:08:16.792621 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:16.792544 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9f0e732a-2468-4ae9-a704-12ceb0ae4eff" (UID: "9f0e732a-2468-4ae9-a704-12ceb0ae4eff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 08:08:16.792785 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:16.792751 2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-workload-certs\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:08:16.792785 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:16.792771 2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-credential-socket\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:08:16.792898 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:16.792787 2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-workload-socket\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:08:16.792898 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:16.792801 2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-istiod-ca-cert\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:08:16.792898 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:16.792815 2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-istio-data\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:08:16.794275 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:16.794245 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-kube-api-access-tvlkd" (OuterVolumeSpecName: "kube-api-access-tvlkd") pod "9f0e732a-2468-4ae9-a704-12ceb0ae4eff" (UID: "9f0e732a-2468-4ae9-a704-12ceb0ae4eff"). InnerVolumeSpecName "kube-api-access-tvlkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 08:08:16.794390 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:16.794304 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9f0e732a-2468-4ae9-a704-12ceb0ae4eff" (UID: "9f0e732a-2468-4ae9-a704-12ceb0ae4eff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 08:08:16.794390 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:16.794378 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9f0e732a-2468-4ae9-a704-12ceb0ae4eff" (UID: "9f0e732a-2468-4ae9-a704-12ceb0ae4eff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:08:16.794467 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:16.794404 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-istio-token" (OuterVolumeSpecName: "istio-token") pod "9f0e732a-2468-4ae9-a704-12ceb0ae4eff" (UID: "9f0e732a-2468-4ae9-a704-12ceb0ae4eff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 08:08:16.893352 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:16.893315 2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-istio-token\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:08:16.893352 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:16.893348 2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-istio-podinfo\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:08:16.893352 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:16.893357 2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-istio-envoy\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:08:16.893597 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:16.893367 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvlkd\" (UniqueName: \"kubernetes.io/projected/9f0e732a-2468-4ae9-a704-12ceb0ae4eff-kube-api-access-tvlkd\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:08:17.049921 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:17.049880 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz"] Sep 16 08:08:17.053839 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:17.053808 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-5493e8-openshift-default-674bf7bcb-2jjqz"] Sep 16 08:08:18.938214 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:18.938180 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f0e732a-2468-4ae9-a704-12ceb0ae4eff" path="/var/lib/kubelet/pods/9f0e732a-2468-4ae9-a704-12ceb0ae4eff/volumes" Sep 16 08:08:22.258712 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:22.258670 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3616cc-29825760-b8992"] Sep 16 08:08:22.261667 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:22.261639 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3616cc-29825760-b8992"] Sep 16 08:08:22.261940 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:22.261918 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:08:22.282227 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:22.282188 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-3616cc-6c5d7cd96c-8ht5l"] Sep 16 08:08:22.282447 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:22.282417 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3616cc-6c5d7cd96c-8ht5l" podUID="4a81a2c2-a21d-47d4-bfb7-0454e8e1f12b" containerName="payload-processing" containerID="cri-o://5c0c065a017d9c18ccecc5fb6ae680251ee894ab14f95a229d0b6766d68810f0" gracePeriod=30 Sep 16 08:08:22.288743 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:22.288705 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3616cc-55fcf67858-6qb2b"] Sep 16 08:08:22.289046 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:22.289005 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3616cc-55fcf67858-6qb2b" podUID="04195d61-0c03-4757-9612-d9e58e5d19e3" containerName="payload-pre-processing" containerID="cri-o://20df1296bdba9b56d42450c80b71be0a94176d9306438ec4568d819ce7a881e1" gracePeriod=30 Sep 16 08:08:22.324273 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:22.324209 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:08:22.407670 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:22.407630 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:08:22.471470 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:22.471432 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:08:22.573063 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:22.573027 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3616cc-6c5d7cd96c-8ht5l" Sep 16 08:08:22.576011 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:22.575978 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3616cc-55fcf67858-6qb2b" Sep 16 08:08:22.648322 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:22.648281 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a81a2c2-a21d-47d4-bfb7-0454e8e1f12b-plugins-config-volume\") pod \"4a81a2c2-a21d-47d4-bfb7-0454e8e1f12b\" (UID: \"4a81a2c2-a21d-47d4-bfb7-0454e8e1f12b\") " Sep 16 08:08:22.648502 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:22.648338 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04195d61-0c03-4757-9612-d9e58e5d19e3-plugins-config-volume\") pod \"04195d61-0c03-4757-9612-d9e58e5d19e3\" (UID: \"04195d61-0c03-4757-9612-d9e58e5d19e3\") " Sep 16 08:08:22.648502 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:22.648432 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmwws\" (UniqueName: \"kubernetes.io/projected/04195d61-0c03-4757-9612-d9e58e5d19e3-kube-api-access-fmwws\") pod \"04195d61-0c03-4757-9612-d9e58e5d19e3\" (UID: \"04195d61-0c03-4757-9612-d9e58e5d19e3\") " Sep 16 08:08:22.648502 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:22.648477 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5dg92\" (UniqueName: \"kubernetes.io/projected/4a81a2c2-a21d-47d4-bfb7-0454e8e1f12b-kube-api-access-5dg92\") pod \"4a81a2c2-a21d-47d4-bfb7-0454e8e1f12b\" (UID: \"4a81a2c2-a21d-47d4-bfb7-0454e8e1f12b\") " Sep 16 08:08:22.648708 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:22.648687 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a81a2c2-a21d-47d4-bfb7-0454e8e1f12b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4a81a2c2-a21d-47d4-bfb7-0454e8e1f12b" (UID: "4a81a2c2-a21d-47d4-bfb7-0454e8e1f12b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 08:08:22.648772 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:22.648726 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04195d61-0c03-4757-9612-d9e58e5d19e3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "04195d61-0c03-4757-9612-d9e58e5d19e3" (UID: "04195d61-0c03-4757-9612-d9e58e5d19e3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 08:08:22.648843 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:22.648825 2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a81a2c2-a21d-47d4-bfb7-0454e8e1f12b-plugins-config-volume\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:08:22.648895 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:22.648850 2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04195d61-0c03-4757-9612-d9e58e5d19e3-plugins-config-volume\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:08:22.650689 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:22.650659 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a81a2c2-a21d-47d4-bfb7-0454e8e1f12b-kube-api-access-5dg92" (OuterVolumeSpecName: "kube-api-access-5dg92") pod "4a81a2c2-a21d-47d4-bfb7-0454e8e1f12b" (UID: "4a81a2c2-a21d-47d4-bfb7-0454e8e1f12b"). InnerVolumeSpecName "kube-api-access-5dg92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 08:08:22.650689 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:22.650682 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04195d61-0c03-4757-9612-d9e58e5d19e3-kube-api-access-fmwws" (OuterVolumeSpecName: "kube-api-access-fmwws") pod "04195d61-0c03-4757-9612-d9e58e5d19e3" (UID: "04195d61-0c03-4757-9612-d9e58e5d19e3"). InnerVolumeSpecName "kube-api-access-fmwws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 08:08:22.749405 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:22.749367 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmwws\" (UniqueName: \"kubernetes.io/projected/04195d61-0c03-4757-9612-d9e58e5d19e3-kube-api-access-fmwws\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:08:22.749405 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:22.749400 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dg92\" (UniqueName: \"kubernetes.io/projected/4a81a2c2-a21d-47d4-bfb7-0454e8e1f12b-kube-api-access-5dg92\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:08:22.769070 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:22.769036 2552 generic.go:356] "Generic (PLEG): container finished" podID="4a81a2c2-a21d-47d4-bfb7-0454e8e1f12b" containerID="5c0c065a017d9c18ccecc5fb6ae680251ee894ab14f95a229d0b6766d68810f0" exitCode=0 Sep 16 08:08:22.769284 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:22.769082 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3616cc-6c5d7cd96c-8ht5l" Sep 16 08:08:22.769284 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:22.769135 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3616cc-6c5d7cd96c-8ht5l" event={"ID":"4a81a2c2-a21d-47d4-bfb7-0454e8e1f12b","Type":"ContainerDied","Data":"5c0c065a017d9c18ccecc5fb6ae680251ee894ab14f95a229d0b6766d68810f0"} Sep 16 08:08:22.769284 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:22.769188 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3616cc-6c5d7cd96c-8ht5l" event={"ID":"4a81a2c2-a21d-47d4-bfb7-0454e8e1f12b","Type":"ContainerDied","Data":"42993ab1678dee94d23330f686ac79d3c08ff0c25b891998ecb30927ae957538"} Sep 16 08:08:22.769284 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:22.769212 2552 scope.go:117] "RemoveContainer" containerID="5c0c065a017d9c18ccecc5fb6ae680251ee894ab14f95a229d0b6766d68810f0" Sep 16 08:08:22.770507 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:22.770482 2552 generic.go:356] "Generic (PLEG): container finished" podID="04195d61-0c03-4757-9612-d9e58e5d19e3" containerID="20df1296bdba9b56d42450c80b71be0a94176d9306438ec4568d819ce7a881e1" exitCode=0 Sep 16 08:08:22.770599 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:22.770519 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3616cc-55fcf67858-6qb2b" Sep 16 08:08:22.770599 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:22.770565 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3616cc-55fcf67858-6qb2b" event={"ID":"04195d61-0c03-4757-9612-d9e58e5d19e3","Type":"ContainerDied","Data":"20df1296bdba9b56d42450c80b71be0a94176d9306438ec4568d819ce7a881e1"} Sep 16 08:08:22.770689 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:22.770612 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3616cc-55fcf67858-6qb2b" event={"ID":"04195d61-0c03-4757-9612-d9e58e5d19e3","Type":"ContainerDied","Data":"5f2b9ad3f1fbe562050fbf3e9ba1e1b52502cbaa254d4653eea3d4261161a68c"} Sep 16 08:08:22.778992 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:22.778968 2552 scope.go:117] "RemoveContainer" containerID="5c0c065a017d9c18ccecc5fb6ae680251ee894ab14f95a229d0b6766d68810f0" Sep 16 08:08:22.779326 ip-10-0-142-170 kubenswrapper[2552]: E0916 08:08:22.779307 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c0c065a017d9c18ccecc5fb6ae680251ee894ab14f95a229d0b6766d68810f0\": container with ID starting with 5c0c065a017d9c18ccecc5fb6ae680251ee894ab14f95a229d0b6766d68810f0 not found: ID does not exist" containerID="5c0c065a017d9c18ccecc5fb6ae680251ee894ab14f95a229d0b6766d68810f0" Sep 16 08:08:22.779414 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:22.779339 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c0c065a017d9c18ccecc5fb6ae680251ee894ab14f95a229d0b6766d68810f0"} err="failed to get container status \"5c0c065a017d9c18ccecc5fb6ae680251ee894ab14f95a229d0b6766d68810f0\": rpc error: code = NotFound desc = could not find container \"5c0c065a017d9c18ccecc5fb6ae680251ee894ab14f95a229d0b6766d68810f0\": container with ID starting with 5c0c065a017d9c18ccecc5fb6ae680251ee894ab14f95a229d0b6766d68810f0 not found: ID does not exist" Sep 16 08:08:22.779414 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:22.779362 2552 scope.go:117] "RemoveContainer" containerID="20df1296bdba9b56d42450c80b71be0a94176d9306438ec4568d819ce7a881e1" Sep 16 08:08:22.788356 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:22.788333 2552 scope.go:117] "RemoveContainer" containerID="20df1296bdba9b56d42450c80b71be0a94176d9306438ec4568d819ce7a881e1" Sep 16 08:08:22.788684 ip-10-0-142-170 kubenswrapper[2552]: E0916 08:08:22.788665 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20df1296bdba9b56d42450c80b71be0a94176d9306438ec4568d819ce7a881e1\": container with ID starting with 20df1296bdba9b56d42450c80b71be0a94176d9306438ec4568d819ce7a881e1 not found: ID does not exist" containerID="20df1296bdba9b56d42450c80b71be0a94176d9306438ec4568d819ce7a881e1" Sep 16 08:08:22.788729 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:22.788693 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20df1296bdba9b56d42450c80b71be0a94176d9306438ec4568d819ce7a881e1"} err="failed to get container status \"20df1296bdba9b56d42450c80b71be0a94176d9306438ec4568d819ce7a881e1\": rpc error: code = NotFound desc = could not find container \"20df1296bdba9b56d42450c80b71be0a94176d9306438ec4568d819ce7a881e1\": container with ID starting with 20df1296bdba9b56d42450c80b71be0a94176d9306438ec4568d819ce7a881e1 not found: ID does not exist" Sep 16 08:08:22.802776 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:22.802687 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3616cc-55fcf67858-6qb2b"] Sep 16 08:08:22.816558 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:22.816522 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3616cc-55fcf67858-6qb2b"] Sep 16 08:08:22.828064 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:22.828026 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-3616cc-6c5d7cd96c-8ht5l"] Sep 16 08:08:22.833718 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:22.833688 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-3616cc-6c5d7cd96c-8ht5l"] Sep 16 08:08:22.936771 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:22.936735 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04195d61-0c03-4757-9612-d9e58e5d19e3" path="/var/lib/kubelet/pods/04195d61-0c03-4757-9612-d9e58e5d19e3/volumes" Sep 16 08:08:22.937665 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:22.937647 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a81a2c2-a21d-47d4-bfb7-0454e8e1f12b" path="/var/lib/kubelet/pods/4a81a2c2-a21d-47d4-bfb7-0454e8e1f12b/volumes" Sep 16 08:08:22.937949 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:22.937935 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c60d2a89-4198-4a2e-b0e3-2b5edb5f7ab5" path="/var/lib/kubelet/pods/c60d2a89-4198-4a2e-b0e3-2b5edb5f7ab5/volumes" Sep 16 08:08:22.953318 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:22.953279 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:08:23.040755 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:23.040721 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:08:23.602763 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:23.602672 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:08:28.666424 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:28.666391 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:08:28.729383 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:28.729346 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:08:28.965817 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:28.965780 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:08:30.155140 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:30.155078 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:08:30.854480 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:30.854448 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:08:39.868608 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:39.868567 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:08:39.897718 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:39.897673 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e9fc8c-57765cdf-v9t9d"] Sep 16 08:08:39.897948 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:39.897916 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e9fc8c-57765cdf-v9t9d" podUID="f1aff88a-f557-4911-a096-859deb592996" containerName="payload-processing" containerID="cri-o://14ead719e42164f336fab3e988fb856fdf7859bb54b65ec47253ed30f0198a1b" gracePeriod=30 Sep 16 08:08:39.931875 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:39.931834 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:08:40.004028 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:40.003988 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:08:40.071337 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:40.071299 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:08:40.159470 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:40.159443 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e9fc8c-57765cdf-v9t9d" Sep 16 08:08:40.211811 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:40.211772 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctt44\" (UniqueName: \"kubernetes.io/projected/f1aff88a-f557-4911-a096-859deb592996-kube-api-access-ctt44\") pod \"f1aff88a-f557-4911-a096-859deb592996\" (UID: \"f1aff88a-f557-4911-a096-859deb592996\") " Sep 16 08:08:40.212010 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:40.211870 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1aff88a-f557-4911-a096-859deb592996-plugins-config-volume\") pod \"f1aff88a-f557-4911-a096-859deb592996\" (UID: \"f1aff88a-f557-4911-a096-859deb592996\") " Sep 16 08:08:40.212254 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:40.212227 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1aff88a-f557-4911-a096-859deb592996-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f1aff88a-f557-4911-a096-859deb592996" (UID: "f1aff88a-f557-4911-a096-859deb592996"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 08:08:40.213994 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:40.213971 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1aff88a-f557-4911-a096-859deb592996-kube-api-access-ctt44" (OuterVolumeSpecName: "kube-api-access-ctt44") pod "f1aff88a-f557-4911-a096-859deb592996" (UID: "f1aff88a-f557-4911-a096-859deb592996"). InnerVolumeSpecName "kube-api-access-ctt44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 08:08:40.312567 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:40.312524 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctt44\" (UniqueName: \"kubernetes.io/projected/f1aff88a-f557-4911-a096-859deb592996-kube-api-access-ctt44\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:08:40.312567 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:40.312559 2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1aff88a-f557-4911-a096-859deb592996-plugins-config-volume\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:08:40.424477 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:40.424440 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:08:40.525161 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:40.525089 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:08:40.864493 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:40.864404 2552 generic.go:356] "Generic (PLEG): container finished" podID="f1aff88a-f557-4911-a096-859deb592996" containerID="14ead719e42164f336fab3e988fb856fdf7859bb54b65ec47253ed30f0198a1b" exitCode=0 Sep 16 08:08:40.864493 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:40.864453 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e9fc8c-57765cdf-v9t9d" Sep 16 08:08:40.864493 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:40.864479 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e9fc8c-57765cdf-v9t9d" event={"ID":"f1aff88a-f557-4911-a096-859deb592996","Type":"ContainerDied","Data":"14ead719e42164f336fab3e988fb856fdf7859bb54b65ec47253ed30f0198a1b"} Sep 16 08:08:40.864745 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:40.864522 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-e9fc8c-57765cdf-v9t9d" event={"ID":"f1aff88a-f557-4911-a096-859deb592996","Type":"ContainerDied","Data":"3649f2afdafed743e9f59046b8eb69e7bd2d5c7847cb15fad344bea71cd59400"} Sep 16 08:08:40.864745 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:40.864544 2552 scope.go:117] "RemoveContainer" containerID="14ead719e42164f336fab3e988fb856fdf7859bb54b65ec47253ed30f0198a1b" Sep 16 08:08:40.874161 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:40.874134 2552 scope.go:117] "RemoveContainer" containerID="14ead719e42164f336fab3e988fb856fdf7859bb54b65ec47253ed30f0198a1b" Sep 16 08:08:40.874533 ip-10-0-142-170 kubenswrapper[2552]: E0916 08:08:40.874472 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14ead719e42164f336fab3e988fb856fdf7859bb54b65ec47253ed30f0198a1b\": container with ID starting with 14ead719e42164f336fab3e988fb856fdf7859bb54b65ec47253ed30f0198a1b not found: ID does not exist" containerID="14ead719e42164f336fab3e988fb856fdf7859bb54b65ec47253ed30f0198a1b" Sep 16 08:08:40.874533 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:40.874512 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14ead719e42164f336fab3e988fb856fdf7859bb54b65ec47253ed30f0198a1b"} err="failed to get container status \"14ead719e42164f336fab3e988fb856fdf7859bb54b65ec47253ed30f0198a1b\": rpc error: code = NotFound desc = could not find container \"14ead719e42164f336fab3e988fb856fdf7859bb54b65ec47253ed30f0198a1b\": container with ID starting with 14ead719e42164f336fab3e988fb856fdf7859bb54b65ec47253ed30f0198a1b not found: ID does not exist" Sep 16 08:08:40.885668 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:40.885631 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e9fc8c-57765cdf-v9t9d"] Sep 16 08:08:40.889361 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:40.889332 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-e9fc8c-57765cdf-v9t9d"] Sep 16 08:08:40.936677 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:40.936637 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1aff88a-f557-4911-a096-859deb592996" path="/var/lib/kubelet/pods/f1aff88a-f557-4911-a096-859deb592996/volumes" Sep 16 08:08:46.389840 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:46.389803 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:08:46.452990 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:46.452954 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:08:46.651165 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:46.651058 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz"] Sep 16 08:08:46.651323 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:46.651293 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" podUID="859b21e8-3a32-49ae-be7c-a7249d2e46a7" containerName="istio-proxy" containerID="cri-o://3cbe1e993c33cf89453646a14533595ec09d316ff819a43d99a536e60ad44dec" gracePeriod=30 Sep 16 08:08:46.670451 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:46.670414 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:08:49.335846 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:49.335810 2552 scope.go:117] "RemoveContainer" containerID="977cf3b5b71558487355e3e2dfec1e0b374ba4fd51e2c7e6f311cc1cf3ba9b99" Sep 16 08:08:51.896280 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:51.896244 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" Sep 16 08:08:51.928131 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:51.928080 2552 generic.go:356] "Generic (PLEG): container finished" podID="859b21e8-3a32-49ae-be7c-a7249d2e46a7" containerID="3cbe1e993c33cf89453646a14533595ec09d316ff819a43d99a536e60ad44dec" exitCode=0 Sep 16 08:08:51.928342 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:51.928132 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" event={"ID":"859b21e8-3a32-49ae-be7c-a7249d2e46a7","Type":"ContainerDied","Data":"3cbe1e993c33cf89453646a14533595ec09d316ff819a43d99a536e60ad44dec"} Sep 16 08:08:51.928342 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:51.928154 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" Sep 16 08:08:51.928342 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:51.928194 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz" event={"ID":"859b21e8-3a32-49ae-be7c-a7249d2e46a7","Type":"ContainerDied","Data":"6108f8e2e9f853ccfdd52ec29a1ed34eecc0e48e6667d7413ce808f46fdf22fe"} Sep 16 08:08:51.928342 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:51.928220 2552 scope.go:117] "RemoveContainer" containerID="3cbe1e993c33cf89453646a14533595ec09d316ff819a43d99a536e60ad44dec" Sep 16 08:08:51.938626 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:51.938598 2552 scope.go:117] "RemoveContainer" containerID="a08dec342f9f7fe0b0bb901e58807ed0c1624722dbab28cd69856222a56dede8" Sep 16 08:08:51.948618 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:51.948593 2552 scope.go:117] "RemoveContainer" containerID="3cbe1e993c33cf89453646a14533595ec09d316ff819a43d99a536e60ad44dec" Sep 16 08:08:51.948952 ip-10-0-142-170 kubenswrapper[2552]: E0916 08:08:51.948935 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cbe1e993c33cf89453646a14533595ec09d316ff819a43d99a536e60ad44dec\": container with ID starting with 3cbe1e993c33cf89453646a14533595ec09d316ff819a43d99a536e60ad44dec not found: ID does not exist" containerID="3cbe1e993c33cf89453646a14533595ec09d316ff819a43d99a536e60ad44dec" Sep 16 08:08:51.949011 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:51.948966 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cbe1e993c33cf89453646a14533595ec09d316ff819a43d99a536e60ad44dec"} err="failed to get container status \"3cbe1e993c33cf89453646a14533595ec09d316ff819a43d99a536e60ad44dec\": rpc error: code = NotFound desc = could not find container \"3cbe1e993c33cf89453646a14533595ec09d316ff819a43d99a536e60ad44dec\": container with ID starting with 3cbe1e993c33cf89453646a14533595ec09d316ff819a43d99a536e60ad44dec not found: ID does not exist" Sep 16 08:08:51.949011 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:51.948985 2552 scope.go:117] "RemoveContainer" containerID="a08dec342f9f7fe0b0bb901e58807ed0c1624722dbab28cd69856222a56dede8" Sep 16 08:08:51.949443 ip-10-0-142-170 kubenswrapper[2552]: E0916 08:08:51.949424 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a08dec342f9f7fe0b0bb901e58807ed0c1624722dbab28cd69856222a56dede8\": container with ID starting with a08dec342f9f7fe0b0bb901e58807ed0c1624722dbab28cd69856222a56dede8 not found: ID does not exist" containerID="a08dec342f9f7fe0b0bb901e58807ed0c1624722dbab28cd69856222a56dede8" Sep 16 08:08:51.949500 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:51.949451 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a08dec342f9f7fe0b0bb901e58807ed0c1624722dbab28cd69856222a56dede8"} err="failed to get container status \"a08dec342f9f7fe0b0bb901e58807ed0c1624722dbab28cd69856222a56dede8\": rpc error: code = NotFound desc = could not find container \"a08dec342f9f7fe0b0bb901e58807ed0c1624722dbab28cd69856222a56dede8\": container with ID starting with a08dec342f9f7fe0b0bb901e58807ed0c1624722dbab28cd69856222a56dede8 not found: ID does not exist" Sep 16 08:08:52.020500 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:52.020469 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/859b21e8-3a32-49ae-be7c-a7249d2e46a7-istio-envoy\") pod \"859b21e8-3a32-49ae-be7c-a7249d2e46a7\" (UID: \"859b21e8-3a32-49ae-be7c-a7249d2e46a7\") " Sep 16 08:08:52.020650 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:52.020522 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/859b21e8-3a32-49ae-be7c-a7249d2e46a7-istio-token\") pod \"859b21e8-3a32-49ae-be7c-a7249d2e46a7\" (UID: \"859b21e8-3a32-49ae-be7c-a7249d2e46a7\") " Sep 16 08:08:52.020650 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:52.020561 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/859b21e8-3a32-49ae-be7c-a7249d2e46a7-credential-socket\") pod \"859b21e8-3a32-49ae-be7c-a7249d2e46a7\" (UID: \"859b21e8-3a32-49ae-be7c-a7249d2e46a7\") " Sep 16 08:08:52.020650 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:52.020601 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/859b21e8-3a32-49ae-be7c-a7249d2e46a7-workload-certs\") pod \"859b21e8-3a32-49ae-be7c-a7249d2e46a7\" (UID: \"859b21e8-3a32-49ae-be7c-a7249d2e46a7\") " Sep 16 08:08:52.020802 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:52.020650 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/859b21e8-3a32-49ae-be7c-a7249d2e46a7-istio-data\") pod \"859b21e8-3a32-49ae-be7c-a7249d2e46a7\" (UID: \"859b21e8-3a32-49ae-be7c-a7249d2e46a7\") " Sep 16 08:08:52.020802 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:52.020680 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rn4ck\" (UniqueName: \"kubernetes.io/projected/859b21e8-3a32-49ae-be7c-a7249d2e46a7-kube-api-access-rn4ck\") pod \"859b21e8-3a32-49ae-be7c-a7249d2e46a7\" (UID: \"859b21e8-3a32-49ae-be7c-a7249d2e46a7\") " Sep 16 08:08:52.020802 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:52.020714 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/859b21e8-3a32-49ae-be7c-a7249d2e46a7-workload-socket\") pod \"859b21e8-3a32-49ae-be7c-a7249d2e46a7\" (UID: \"859b21e8-3a32-49ae-be7c-a7249d2e46a7\") " Sep 16 08:08:52.020802 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:52.020756 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/859b21e8-3a32-49ae-be7c-a7249d2e46a7-istio-podinfo\") pod \"859b21e8-3a32-49ae-be7c-a7249d2e46a7\" (UID: \"859b21e8-3a32-49ae-be7c-a7249d2e46a7\") " Sep 16 08:08:52.020802 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:52.020784 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/859b21e8-3a32-49ae-be7c-a7249d2e46a7-istiod-ca-cert\") pod \"859b21e8-3a32-49ae-be7c-a7249d2e46a7\" (UID: \"859b21e8-3a32-49ae-be7c-a7249d2e46a7\") " Sep 16 08:08:52.021055 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:52.020965 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/859b21e8-3a32-49ae-be7c-a7249d2e46a7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "859b21e8-3a32-49ae-be7c-a7249d2e46a7" (UID: "859b21e8-3a32-49ae-be7c-a7249d2e46a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:08:52.021157 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:52.021067 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/859b21e8-3a32-49ae-be7c-a7249d2e46a7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "859b21e8-3a32-49ae-be7c-a7249d2e46a7" (UID: "859b21e8-3a32-49ae-be7c-a7249d2e46a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:08:52.021224 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:52.021178 2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/859b21e8-3a32-49ae-be7c-a7249d2e46a7-workload-socket\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:08:52.021224 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:52.021197 2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/859b21e8-3a32-49ae-be7c-a7249d2e46a7-workload-certs\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:08:52.021338 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:52.021270 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/859b21e8-3a32-49ae-be7c-a7249d2e46a7-istio-data" (OuterVolumeSpecName: "istio-data") pod "859b21e8-3a32-49ae-be7c-a7249d2e46a7" (UID: "859b21e8-3a32-49ae-be7c-a7249d2e46a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:08:52.021338 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:52.021285 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/859b21e8-3a32-49ae-be7c-a7249d2e46a7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "859b21e8-3a32-49ae-be7c-a7249d2e46a7" (UID: "859b21e8-3a32-49ae-be7c-a7249d2e46a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:08:52.021338 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:52.021318 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/859b21e8-3a32-49ae-be7c-a7249d2e46a7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "859b21e8-3a32-49ae-be7c-a7249d2e46a7" (UID: "859b21e8-3a32-49ae-be7c-a7249d2e46a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 16 08:08:52.022809 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:52.022777 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/859b21e8-3a32-49ae-be7c-a7249d2e46a7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "859b21e8-3a32-49ae-be7c-a7249d2e46a7" (UID: "859b21e8-3a32-49ae-be7c-a7249d2e46a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:08:52.022920 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:52.022819 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/859b21e8-3a32-49ae-be7c-a7249d2e46a7-istio-token" (OuterVolumeSpecName: "istio-token") pod "859b21e8-3a32-49ae-be7c-a7249d2e46a7" (UID: "859b21e8-3a32-49ae-be7c-a7249d2e46a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 08:08:52.023429 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:52.023401 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/859b21e8-3a32-49ae-be7c-a7249d2e46a7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "859b21e8-3a32-49ae-be7c-a7249d2e46a7" (UID: "859b21e8-3a32-49ae-be7c-a7249d2e46a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 16 08:08:52.023497 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:52.023411 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/859b21e8-3a32-49ae-be7c-a7249d2e46a7-kube-api-access-rn4ck" (OuterVolumeSpecName: "kube-api-access-rn4ck") pod "859b21e8-3a32-49ae-be7c-a7249d2e46a7" (UID: "859b21e8-3a32-49ae-be7c-a7249d2e46a7"). InnerVolumeSpecName "kube-api-access-rn4ck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 08:08:52.121732 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:52.121693 2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/859b21e8-3a32-49ae-be7c-a7249d2e46a7-istio-podinfo\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:08:52.121732 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:52.121721 2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/859b21e8-3a32-49ae-be7c-a7249d2e46a7-istiod-ca-cert\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:08:52.121732 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:52.121735 2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/859b21e8-3a32-49ae-be7c-a7249d2e46a7-istio-envoy\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:08:52.121732 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:52.121745 2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/859b21e8-3a32-49ae-be7c-a7249d2e46a7-istio-token\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:08:52.122014 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:52.121756 2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/859b21e8-3a32-49ae-be7c-a7249d2e46a7-credential-socket\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:08:52.122014 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:52.121768 2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/859b21e8-3a32-49ae-be7c-a7249d2e46a7-istio-data\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:08:52.122014 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:52.121781 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn4ck\" (UniqueName: \"kubernetes.io/projected/859b21e8-3a32-49ae-be7c-a7249d2e46a7-kube-api-access-rn4ck\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:08:52.251622 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:52.251582 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz"] Sep 16 08:08:52.254969 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:52.254933 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-e9fc8c-openshift-default-684ffd56bb-wwhcz"] Sep 16 08:08:52.938260 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:08:52.938222 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="859b21e8-3a32-49ae-be7c-a7249d2e46a7" path="/var/lib/kubelet/pods/859b21e8-3a32-49ae-be7c-a7249d2e46a7/volumes" Sep 16 08:09:21.196214 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:09:21.196181 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:09:21.297806 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:09:21.297773 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:10:00.263751 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:10:00.263712 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:10:12.492363 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:10:12.492326 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:10:13.992645 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:10:13.992606 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:11:38.168719 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:11:38.168685 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s7n7g"] Sep 16 08:11:38.169295 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:11:38.168894 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s7n7g" podUID="955c6c96-0681-482f-b967-946e3b2eda9f" containerName="manager" containerID="cri-o://c4471232bfc1dfdde840118a0572e29e1197f3ced49622cfc06905ca367dc6b2" gracePeriod=10 Sep 16 08:11:38.518406 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:11:38.518378 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s7n7g" Sep 16 08:11:38.612275 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:11:38.612233 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/955c6c96-0681-482f-b967-946e3b2eda9f-extensions-socket-volume\") pod \"955c6c96-0681-482f-b967-946e3b2eda9f\" (UID: \"955c6c96-0681-482f-b967-946e3b2eda9f\") " Sep 16 08:11:38.612483 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:11:38.612293 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nbpx\" (UniqueName: \"kubernetes.io/projected/955c6c96-0681-482f-b967-946e3b2eda9f-kube-api-access-6nbpx\") pod \"955c6c96-0681-482f-b967-946e3b2eda9f\" (UID: \"955c6c96-0681-482f-b967-946e3b2eda9f\") " Sep 16 08:11:38.612649 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:11:38.612625 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/955c6c96-0681-482f-b967-946e3b2eda9f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "955c6c96-0681-482f-b967-946e3b2eda9f" (UID: "955c6c96-0681-482f-b967-946e3b2eda9f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:11:38.614436 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:11:38.614406 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/955c6c96-0681-482f-b967-946e3b2eda9f-kube-api-access-6nbpx" (OuterVolumeSpecName: "kube-api-access-6nbpx") pod "955c6c96-0681-482f-b967-946e3b2eda9f" (UID: "955c6c96-0681-482f-b967-946e3b2eda9f"). InnerVolumeSpecName "kube-api-access-6nbpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 08:11:38.713867 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:11:38.713767 2552 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/955c6c96-0681-482f-b967-946e3b2eda9f-extensions-socket-volume\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:11:38.713867 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:11:38.713800 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nbpx\" (UniqueName: \"kubernetes.io/projected/955c6c96-0681-482f-b967-946e3b2eda9f-kube-api-access-6nbpx\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:11:38.731761 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:11:38.731723 2552 generic.go:356] "Generic (PLEG): container finished" podID="955c6c96-0681-482f-b967-946e3b2eda9f" containerID="c4471232bfc1dfdde840118a0572e29e1197f3ced49622cfc06905ca367dc6b2" exitCode=0 Sep 16 08:11:38.731928 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:11:38.731786 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s7n7g" Sep 16 08:11:38.731928 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:11:38.731810 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s7n7g" event={"ID":"955c6c96-0681-482f-b967-946e3b2eda9f","Type":"ContainerDied","Data":"c4471232bfc1dfdde840118a0572e29e1197f3ced49622cfc06905ca367dc6b2"} Sep 16 08:11:38.731928 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:11:38.731866 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s7n7g" event={"ID":"955c6c96-0681-482f-b967-946e3b2eda9f","Type":"ContainerDied","Data":"fd082e052e9a5b32742ea230434a19c121dfb10357127daec5a5b031565de84d"} Sep 16 08:11:38.731928 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:11:38.731893 2552 scope.go:117] "RemoveContainer" containerID="c4471232bfc1dfdde840118a0572e29e1197f3ced49622cfc06905ca367dc6b2" Sep 16 08:11:38.741730 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:11:38.741711 2552 scope.go:117] "RemoveContainer" containerID="dec07385c2e6d7f2a22bfe88fed8e3ec2f4e66935dc3d1fd537a133cee364940" Sep 16 08:11:38.749618 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:11:38.749597 2552 scope.go:117] "RemoveContainer" containerID="c4471232bfc1dfdde840118a0572e29e1197f3ced49622cfc06905ca367dc6b2" Sep 16 08:11:38.749929 ip-10-0-142-170 kubenswrapper[2552]: E0916 08:11:38.749909 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4471232bfc1dfdde840118a0572e29e1197f3ced49622cfc06905ca367dc6b2\": container with ID starting with c4471232bfc1dfdde840118a0572e29e1197f3ced49622cfc06905ca367dc6b2 not found: ID does not exist" containerID="c4471232bfc1dfdde840118a0572e29e1197f3ced49622cfc06905ca367dc6b2" Sep 16 08:11:38.750005 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:11:38.749943 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4471232bfc1dfdde840118a0572e29e1197f3ced49622cfc06905ca367dc6b2"} err="failed to get container status \"c4471232bfc1dfdde840118a0572e29e1197f3ced49622cfc06905ca367dc6b2\": rpc error: code = NotFound desc = could not find container \"c4471232bfc1dfdde840118a0572e29e1197f3ced49622cfc06905ca367dc6b2\": container with ID starting with c4471232bfc1dfdde840118a0572e29e1197f3ced49622cfc06905ca367dc6b2 not found: ID does not exist" Sep 16 08:11:38.750005 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:11:38.749968 2552 scope.go:117] "RemoveContainer" containerID="dec07385c2e6d7f2a22bfe88fed8e3ec2f4e66935dc3d1fd537a133cee364940" Sep 16 08:11:38.750287 ip-10-0-142-170 kubenswrapper[2552]: E0916 08:11:38.750267 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dec07385c2e6d7f2a22bfe88fed8e3ec2f4e66935dc3d1fd537a133cee364940\": container with ID starting with dec07385c2e6d7f2a22bfe88fed8e3ec2f4e66935dc3d1fd537a133cee364940 not found: ID does not exist" containerID="dec07385c2e6d7f2a22bfe88fed8e3ec2f4e66935dc3d1fd537a133cee364940" Sep 16 08:11:38.750354 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:11:38.750294 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dec07385c2e6d7f2a22bfe88fed8e3ec2f4e66935dc3d1fd537a133cee364940"} err="failed to get container status \"dec07385c2e6d7f2a22bfe88fed8e3ec2f4e66935dc3d1fd537a133cee364940\": rpc error: code = NotFound desc = could not find container \"dec07385c2e6d7f2a22bfe88fed8e3ec2f4e66935dc3d1fd537a133cee364940\": container with ID starting with dec07385c2e6d7f2a22bfe88fed8e3ec2f4e66935dc3d1fd537a133cee364940 not found: ID does not exist" Sep 16 08:11:38.752955 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:11:38.752930 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s7n7g"] Sep 16 08:11:38.756731 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:11:38.756706 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-s7n7g"] Sep 16 08:11:38.936315 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:11:38.936272 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="955c6c96-0681-482f-b967-946e3b2eda9f" path="/var/lib/kubelet/pods/955c6c96-0681-482f-b967-946e3b2eda9f/volumes" Sep 16 08:12:14.666944 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:14.666909 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:12:14.715988 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:14.715952 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:12:38.944949 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:38.944915 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vn6jp"] Sep 16 08:12:38.946057 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:38.945254 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8929a09-cb89-4979-b219-e6d28df7cce8" containerName="main" Sep 16 08:12:38.946057 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:38.945265 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8929a09-cb89-4979-b219-e6d28df7cce8" containerName="main" Sep 16 08:12:38.946057 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:38.945279 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="955c6c96-0681-482f-b967-946e3b2eda9f" containerName="manager" Sep 16 08:12:38.946057 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:38.945285 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="955c6c96-0681-482f-b967-946e3b2eda9f" containerName="manager" Sep 16 08:12:38.946057 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:38.945294 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f0e732a-2468-4ae9-a704-12ceb0ae4eff" containerName="istio-proxy" Sep 16 08:12:38.946057 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:38.945300 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f0e732a-2468-4ae9-a704-12ceb0ae4eff" containerName="istio-proxy" Sep 16 08:12:38.946057 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:38.945306 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a81a2c2-a21d-47d4-bfb7-0454e8e1f12b" containerName="payload-processing" Sep 16 08:12:38.946057 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:38.945312 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a81a2c2-a21d-47d4-bfb7-0454e8e1f12b" containerName="payload-processing" Sep 16 08:12:38.946057 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:38.945321 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f4b9791-3964-46e0-8b08-4d66c5d73734" containerName="revoke-keys" Sep 16 08:12:38.946057 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:38.945326 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f4b9791-3964-46e0-8b08-4d66c5d73734" containerName="revoke-keys" Sep 16 08:12:38.946057 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:38.945332 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="859b21e8-3a32-49ae-be7c-a7249d2e46a7" containerName="istio-proxy" Sep 16 08:12:38.946057 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:38.945338 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="859b21e8-3a32-49ae-be7c-a7249d2e46a7" containerName="istio-proxy" Sep 16 08:12:38.946057 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:38.945345 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="859b21e8-3a32-49ae-be7c-a7249d2e46a7" containerName="istio-proxy" Sep 16 08:12:38.946057 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:38.945350 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="859b21e8-3a32-49ae-be7c-a7249d2e46a7" containerName="istio-proxy" Sep 16 08:12:38.946057 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:38.945356 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10611b7d-e979-4b16-b87f-0c0fd083899e" containerName="payload-pre-processing" Sep 16 08:12:38.946057 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:38.945362 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="10611b7d-e979-4b16-b87f-0c0fd083899e" containerName="payload-pre-processing" Sep 16 08:12:38.946057 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:38.945369 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f0e732a-2468-4ae9-a704-12ceb0ae4eff" containerName="istio-proxy" Sep 16 08:12:38.946057 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:38.945375 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f0e732a-2468-4ae9-a704-12ceb0ae4eff" containerName="istio-proxy" Sep 16 08:12:38.946057 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:38.945381 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04195d61-0c03-4757-9612-d9e58e5d19e3" containerName="payload-pre-processing" Sep 16 08:12:38.946057 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:38.945387 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="04195d61-0c03-4757-9612-d9e58e5d19e3" containerName="payload-pre-processing" Sep 16 08:12:38.946057 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:38.945393 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1aff88a-f557-4911-a096-859deb592996" containerName="payload-processing" Sep 16 08:12:38.946057 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:38.945398 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1aff88a-f557-4911-a096-859deb592996" containerName="payload-processing" Sep 16 08:12:38.946057 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:38.945448 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1aff88a-f557-4911-a096-859deb592996" containerName="payload-processing" Sep 16 08:12:38.946057 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:38.945455 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="955c6c96-0681-482f-b967-946e3b2eda9f" containerName="manager" Sep 16 08:12:38.946057 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:38.945461 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="859b21e8-3a32-49ae-be7c-a7249d2e46a7" containerName="istio-proxy" Sep 16 08:12:38.946057 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:38.945468 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f0e732a-2468-4ae9-a704-12ceb0ae4eff" containerName="istio-proxy" Sep 16 08:12:38.946057 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:38.945473 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="04195d61-0c03-4757-9612-d9e58e5d19e3" containerName="payload-pre-processing" Sep 16 08:12:38.946057 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:38.945479 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a81a2c2-a21d-47d4-bfb7-0454e8e1f12b" containerName="payload-processing" Sep 16 08:12:38.946057 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:38.945483 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="859b21e8-3a32-49ae-be7c-a7249d2e46a7" containerName="istio-proxy" Sep 16 08:12:38.946057 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:38.945487 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f0e732a-2468-4ae9-a704-12ceb0ae4eff" containerName="istio-proxy" Sep 16 08:12:38.946057 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:38.945493 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="10611b7d-e979-4b16-b87f-0c0fd083899e" containerName="payload-pre-processing" Sep 16 08:12:38.946057 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:38.945500 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8929a09-cb89-4979-b219-e6d28df7cce8" containerName="main" Sep 16 08:12:38.946057 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:38.945505 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f4b9791-3964-46e0-8b08-4d66c5d73734" containerName="revoke-keys" Sep 16 08:12:38.948786 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:38.948766 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vn6jp" Sep 16 08:12:38.951186 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:38.951156 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-xbvtl\"" Sep 16 08:12:38.963649 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:38.963616 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vn6jp"] Sep 16 08:12:39.048933 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:39.048887 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgwct\" (UniqueName: \"kubernetes.io/projected/f98af655-d206-479c-b49a-887123030af2-kube-api-access-cgwct\") pod \"kuadrant-operator-controller-manager-5769f99d9f-vn6jp\" (UID: \"f98af655-d206-479c-b49a-887123030af2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vn6jp" Sep 16 08:12:39.049170 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:39.048951 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f98af655-d206-479c-b49a-887123030af2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-vn6jp\" (UID: \"f98af655-d206-479c-b49a-887123030af2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vn6jp" Sep 16 08:12:39.150295 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:39.150257 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgwct\" (UniqueName: \"kubernetes.io/projected/f98af655-d206-479c-b49a-887123030af2-kube-api-access-cgwct\") pod \"kuadrant-operator-controller-manager-5769f99d9f-vn6jp\" (UID: \"f98af655-d206-479c-b49a-887123030af2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vn6jp" Sep 16 08:12:39.150468 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:39.150300 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f98af655-d206-479c-b49a-887123030af2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-vn6jp\" (UID: \"f98af655-d206-479c-b49a-887123030af2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vn6jp" Sep 16 08:12:39.150713 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:39.150694 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f98af655-d206-479c-b49a-887123030af2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-vn6jp\" (UID: \"f98af655-d206-479c-b49a-887123030af2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vn6jp" Sep 16 08:12:39.162192 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:39.162151 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgwct\" (UniqueName: \"kubernetes.io/projected/f98af655-d206-479c-b49a-887123030af2-kube-api-access-cgwct\") pod \"kuadrant-operator-controller-manager-5769f99d9f-vn6jp\" (UID: \"f98af655-d206-479c-b49a-887123030af2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vn6jp" Sep 16 08:12:39.259335 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:39.259231 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vn6jp" Sep 16 08:12:39.539771 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:39.539690 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vn6jp"] Sep 16 08:12:39.543834 ip-10-0-142-170 kubenswrapper[2552]: W0916 08:12:39.543803 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf98af655_d206_479c_b49a_887123030af2.slice/crio-a1558e1df44be25b015641cd119688fae64d749c3424fc02429bec5c8b8a9b5b WatchSource:0}: Error finding container a1558e1df44be25b015641cd119688fae64d749c3424fc02429bec5c8b8a9b5b: Status 404 returned error can't find the container with id a1558e1df44be25b015641cd119688fae64d749c3424fc02429bec5c8b8a9b5b Sep 16 08:12:40.031828 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:40.031791 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vn6jp" event={"ID":"f98af655-d206-479c-b49a-887123030af2","Type":"ContainerStarted","Data":"ac48e19a3149dcce10c606aab0e0cf1d73c67eecf0c3178976f2c0887d4d519e"} Sep 16 08:12:40.032268 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:40.031875 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vn6jp" Sep 16 08:12:40.032268 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:40.031893 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vn6jp" event={"ID":"f98af655-d206-479c-b49a-887123030af2","Type":"ContainerStarted","Data":"a1558e1df44be25b015641cd119688fae64d749c3424fc02429bec5c8b8a9b5b"} Sep 16 08:12:40.051721 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:40.051644 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vn6jp" podStartSLOduration=2.051620613 podStartE2EDuration="2.051620613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 08:12:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 08:12:40.050244309 +0000 UTC m=+2331.634022658" watchObservedRunningTime="2026-09-16 08:12:40.051620613 +0000 UTC m=+2331.635398966" Sep 16 08:12:49.404355 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:49.404318 2552 scope.go:117] "RemoveContainer" containerID="f794b17cb75d0f9ef584231378df43b7a3f3210390adf99cfb398c2787f253eb" Sep 16 08:12:51.039962 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:51.039928 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-vn6jp" Sep 16 08:12:55.055672 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:55.055636 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:12:55.109313 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:12:55.109278 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:13:23.968850 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:13:23.968815 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66"] Sep 16 08:13:23.969367 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:13:23.969347 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="955c6c96-0681-482f-b967-946e3b2eda9f" containerName="manager" Sep 16 08:13:23.969412 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:13:23.969372 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="955c6c96-0681-482f-b967-946e3b2eda9f" containerName="manager" Sep 16 08:13:23.969474 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:13:23.969463 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="955c6c96-0681-482f-b967-946e3b2eda9f" containerName="manager" Sep 16 08:13:23.973118 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:13:23.973082 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66" Sep 16 08:13:23.975458 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:13:23.975424 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-c5501a5b-kserve-self-signed-certs\"" Sep 16 08:13:23.982456 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:13:23.982431 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66"] Sep 16 08:13:24.012219 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:13:24.012172 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:13:24.059404 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:13:24.059363 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b88ecd1-2ecc-430b-bae2-e73c800b41a2-model-cache\") pod \"e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66\" (UID: \"0b88ecd1-2ecc-430b-bae2-e73c800b41a2\") " pod="llm/e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66" Sep 16 08:13:24.059404 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:13:24.059411 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b88ecd1-2ecc-430b-bae2-e73c800b41a2-home\") pod \"e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66\" (UID: \"0b88ecd1-2ecc-430b-bae2-e73c800b41a2\") " pod="llm/e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66" Sep 16 08:13:24.059673 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:13:24.059468 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b88ecd1-2ecc-430b-bae2-e73c800b41a2-tls-certs\") pod \"e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66\" (UID: \"0b88ecd1-2ecc-430b-bae2-e73c800b41a2\") " pod="llm/e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66" Sep 16 08:13:24.059673 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:13:24.059511 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh6bz\" (UniqueName: \"kubernetes.io/projected/0b88ecd1-2ecc-430b-bae2-e73c800b41a2-kube-api-access-wh6bz\") pod \"e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66\" (UID: \"0b88ecd1-2ecc-430b-bae2-e73c800b41a2\") " pod="llm/e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66" Sep 16 08:13:24.059673 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:13:24.059531 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b88ecd1-2ecc-430b-bae2-e73c800b41a2-tmp-dir\") pod \"e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66\" (UID: \"0b88ecd1-2ecc-430b-bae2-e73c800b41a2\") " pod="llm/e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66" Sep 16 08:13:24.059673 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:13:24.059557 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b88ecd1-2ecc-430b-bae2-e73c800b41a2-dshm\") pod \"e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66\" (UID: \"0b88ecd1-2ecc-430b-bae2-e73c800b41a2\") " pod="llm/e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66" Sep 16 08:13:24.160587 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:13:24.160544 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b88ecd1-2ecc-430b-bae2-e73c800b41a2-model-cache\") pod \"e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66\" (UID: \"0b88ecd1-2ecc-430b-bae2-e73c800b41a2\") " pod="llm/e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66" Sep 16 08:13:24.160587 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:13:24.160590 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b88ecd1-2ecc-430b-bae2-e73c800b41a2-home\") pod \"e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66\" (UID: \"0b88ecd1-2ecc-430b-bae2-e73c800b41a2\") " pod="llm/e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66" Sep 16 08:13:24.160864 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:13:24.160619 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b88ecd1-2ecc-430b-bae2-e73c800b41a2-tls-certs\") pod \"e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66\" (UID: \"0b88ecd1-2ecc-430b-bae2-e73c800b41a2\") " pod="llm/e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66" Sep 16 08:13:24.160864 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:13:24.160661 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh6bz\" (UniqueName: \"kubernetes.io/projected/0b88ecd1-2ecc-430b-bae2-e73c800b41a2-kube-api-access-wh6bz\") pod \"e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66\" (UID: \"0b88ecd1-2ecc-430b-bae2-e73c800b41a2\") " pod="llm/e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66" Sep 16 08:13:24.160864 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:13:24.160682 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b88ecd1-2ecc-430b-bae2-e73c800b41a2-tmp-dir\") pod \"e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66\" (UID: \"0b88ecd1-2ecc-430b-bae2-e73c800b41a2\") " pod="llm/e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66" Sep 16 08:13:24.160864 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:13:24.160716 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b88ecd1-2ecc-430b-bae2-e73c800b41a2-dshm\") pod \"e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66\" (UID: \"0b88ecd1-2ecc-430b-bae2-e73c800b41a2\") " pod="llm/e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66" Sep 16 08:13:24.161078 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:13:24.161043 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b88ecd1-2ecc-430b-bae2-e73c800b41a2-model-cache\") pod \"e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66\" (UID: \"0b88ecd1-2ecc-430b-bae2-e73c800b41a2\") " pod="llm/e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66" Sep 16 08:13:24.161185 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:13:24.161084 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b88ecd1-2ecc-430b-bae2-e73c800b41a2-home\") pod \"e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66\" (UID: \"0b88ecd1-2ecc-430b-bae2-e73c800b41a2\") " pod="llm/e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66" Sep 16 08:13:24.161185 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:13:24.161177 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b88ecd1-2ecc-430b-bae2-e73c800b41a2-tmp-dir\") pod \"e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66\" (UID: \"0b88ecd1-2ecc-430b-bae2-e73c800b41a2\") " pod="llm/e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66" Sep 16 08:13:24.163158 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:13:24.163128 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b88ecd1-2ecc-430b-bae2-e73c800b41a2-dshm\") pod \"e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66\" (UID: \"0b88ecd1-2ecc-430b-bae2-e73c800b41a2\") " pod="llm/e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66" Sep 16 08:13:24.163332 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:13:24.163316 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b88ecd1-2ecc-430b-bae2-e73c800b41a2-tls-certs\") pod \"e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66\" (UID: \"0b88ecd1-2ecc-430b-bae2-e73c800b41a2\") " pod="llm/e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66" Sep 16 08:13:24.169032 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:13:24.169006 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh6bz\" (UniqueName: \"kubernetes.io/projected/0b88ecd1-2ecc-430b-bae2-e73c800b41a2-kube-api-access-wh6bz\") pod \"e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66\" (UID: \"0b88ecd1-2ecc-430b-bae2-e73c800b41a2\") " pod="llm/e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66" Sep 16 08:13:24.284223 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:13:24.284140 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66" Sep 16 08:13:24.556082 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:13:24.556054 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66"] Sep 16 08:13:24.558319 ip-10-0-142-170 kubenswrapper[2552]: W0916 08:13:24.558290 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b88ecd1_2ecc_430b_bae2_e73c800b41a2.slice/crio-733e08f54ddb2a8643882ecabb5f59cb9b3d4ca26584b5494fefc1da316e157f WatchSource:0}: Error finding container 733e08f54ddb2a8643882ecabb5f59cb9b3d4ca26584b5494fefc1da316e157f: Status 404 returned error can't find the container with id 733e08f54ddb2a8643882ecabb5f59cb9b3d4ca26584b5494fefc1da316e157f Sep 16 08:13:25.248212 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:13:25.248166 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66" event={"ID":"0b88ecd1-2ecc-430b-bae2-e73c800b41a2","Type":"ContainerStarted","Data":"6fc8d88858ee454587839b496774f070ff80f575c12f94d5a50edb98aa4a2e1c"} Sep 16 08:13:25.248212 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:13:25.248218 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66" event={"ID":"0b88ecd1-2ecc-430b-bae2-e73c800b41a2","Type":"ContainerStarted","Data":"733e08f54ddb2a8643882ecabb5f59cb9b3d4ca26584b5494fefc1da316e157f"} Sep 16 08:13:25.264543 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:13:25.264485 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66" podStartSLOduration=2.264464723 podStartE2EDuration="2.264464723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 08:13:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 08:13:25.263361762 +0000 UTC m=+2376.847140102" watchObservedRunningTime="2026-09-16 08:13:25.264464723 +0000 UTC m=+2376.848243062" Sep 16 08:13:34.284974 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:13:34.284939 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66" Sep 16 08:13:34.285494 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:13:34.284993 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66" Sep 16 08:13:34.297778 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:13:34.297747 2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66" Sep 16 08:13:34.309394 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:13:34.309366 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66" Sep 16 08:13:39.696976 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:13:39.696941 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:14:02.368551 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:02.368516 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:14:21.457818 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:21.457741 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66"] Sep 16 08:14:21.477898 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:21.477867 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:14:23.347527 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:23.347491 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:14:36.477656 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:36.477614 2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66" podUID="0b88ecd1-2ecc-430b-bae2-e73c800b41a2" containerName="main" containerID="cri-o://6fc8d88858ee454587839b496774f070ff80f575c12f94d5a50edb98aa4a2e1c" gracePeriod=45 Sep 16 08:14:36.731545 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:36.731477 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66" Sep 16 08:14:36.840223 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:36.840174 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wh6bz\" (UniqueName: \"kubernetes.io/projected/0b88ecd1-2ecc-430b-bae2-e73c800b41a2-kube-api-access-wh6bz\") pod \"0b88ecd1-2ecc-430b-bae2-e73c800b41a2\" (UID: \"0b88ecd1-2ecc-430b-bae2-e73c800b41a2\") " Sep 16 08:14:36.840430 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:36.840247 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b88ecd1-2ecc-430b-bae2-e73c800b41a2-tmp-dir\") pod \"0b88ecd1-2ecc-430b-bae2-e73c800b41a2\" (UID: \"0b88ecd1-2ecc-430b-bae2-e73c800b41a2\") " Sep 16 08:14:36.840430 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:36.840374 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b88ecd1-2ecc-430b-bae2-e73c800b41a2-dshm\") pod \"0b88ecd1-2ecc-430b-bae2-e73c800b41a2\" (UID: \"0b88ecd1-2ecc-430b-bae2-e73c800b41a2\") " Sep 16 08:14:36.840430 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:36.840397 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b88ecd1-2ecc-430b-bae2-e73c800b41a2-home\") pod \"0b88ecd1-2ecc-430b-bae2-e73c800b41a2\" (UID: \"0b88ecd1-2ecc-430b-bae2-e73c800b41a2\") " Sep 16 08:14:36.840430 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:36.840425 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b88ecd1-2ecc-430b-bae2-e73c800b41a2-model-cache\") pod \"0b88ecd1-2ecc-430b-bae2-e73c800b41a2\" (UID: \"0b88ecd1-2ecc-430b-bae2-e73c800b41a2\") " Sep 16 08:14:36.840671 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:36.840526 2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b88ecd1-2ecc-430b-bae2-e73c800b41a2-tls-certs\") pod \"0b88ecd1-2ecc-430b-bae2-e73c800b41a2\" (UID: \"0b88ecd1-2ecc-430b-bae2-e73c800b41a2\") " Sep 16 08:14:36.840751 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:36.840723 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b88ecd1-2ecc-430b-bae2-e73c800b41a2-home" (OuterVolumeSpecName: "home") pod "0b88ecd1-2ecc-430b-bae2-e73c800b41a2" (UID: "0b88ecd1-2ecc-430b-bae2-e73c800b41a2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:14:36.840806 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:36.840736 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b88ecd1-2ecc-430b-bae2-e73c800b41a2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0b88ecd1-2ecc-430b-bae2-e73c800b41a2" (UID: "0b88ecd1-2ecc-430b-bae2-e73c800b41a2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:14:36.840875 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:36.840857 2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b88ecd1-2ecc-430b-bae2-e73c800b41a2-tmp-dir\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:14:36.840939 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:36.840876 2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b88ecd1-2ecc-430b-bae2-e73c800b41a2-home\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:14:36.841018 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:36.840982 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b88ecd1-2ecc-430b-bae2-e73c800b41a2-model-cache" (OuterVolumeSpecName: "model-cache") pod "0b88ecd1-2ecc-430b-bae2-e73c800b41a2" (UID: "0b88ecd1-2ecc-430b-bae2-e73c800b41a2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:14:36.842708 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:36.842670 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b88ecd1-2ecc-430b-bae2-e73c800b41a2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0b88ecd1-2ecc-430b-bae2-e73c800b41a2" (UID: "0b88ecd1-2ecc-430b-bae2-e73c800b41a2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 16 08:14:36.842708 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:36.842679 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b88ecd1-2ecc-430b-bae2-e73c800b41a2-kube-api-access-wh6bz" (OuterVolumeSpecName: "kube-api-access-wh6bz") pod "0b88ecd1-2ecc-430b-bae2-e73c800b41a2" (UID: "0b88ecd1-2ecc-430b-bae2-e73c800b41a2"). InnerVolumeSpecName "kube-api-access-wh6bz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 16 08:14:36.842871 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:36.842738 2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b88ecd1-2ecc-430b-bae2-e73c800b41a2-dshm" (OuterVolumeSpecName: "dshm") pod "0b88ecd1-2ecc-430b-bae2-e73c800b41a2" (UID: "0b88ecd1-2ecc-430b-bae2-e73c800b41a2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 16 08:14:36.941329 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:36.941298 2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b88ecd1-2ecc-430b-bae2-e73c800b41a2-model-cache\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:14:36.941329 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:36.941326 2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b88ecd1-2ecc-430b-bae2-e73c800b41a2-tls-certs\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:14:36.941329 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:36.941336 2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh6bz\" (UniqueName: \"kubernetes.io/projected/0b88ecd1-2ecc-430b-bae2-e73c800b41a2-kube-api-access-wh6bz\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:14:36.941572 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:36.941345 2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b88ecd1-2ecc-430b-bae2-e73c800b41a2-dshm\") on node \"ip-10-0-142-170.ec2.internal\" DevicePath \"\"" Sep 16 08:14:37.592874 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:37.592839 2552 generic.go:356] "Generic (PLEG): container finished" podID="0b88ecd1-2ecc-430b-bae2-e73c800b41a2" containerID="6fc8d88858ee454587839b496774f070ff80f575c12f94d5a50edb98aa4a2e1c" exitCode=0 Sep 16 08:14:37.593328 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:37.592888 2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66" Sep 16 08:14:37.593328 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:37.592921 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66" event={"ID":"0b88ecd1-2ecc-430b-bae2-e73c800b41a2","Type":"ContainerDied","Data":"6fc8d88858ee454587839b496774f070ff80f575c12f94d5a50edb98aa4a2e1c"} Sep 16 08:14:37.593328 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:37.592967 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66" event={"ID":"0b88ecd1-2ecc-430b-bae2-e73c800b41a2","Type":"ContainerDied","Data":"733e08f54ddb2a8643882ecabb5f59cb9b3d4ca26584b5494fefc1da316e157f"} Sep 16 08:14:37.593328 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:37.592984 2552 scope.go:117] "RemoveContainer" containerID="6fc8d88858ee454587839b496774f070ff80f575c12f94d5a50edb98aa4a2e1c" Sep 16 08:14:37.602588 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:37.602559 2552 scope.go:117] "RemoveContainer" containerID="6fc8d88858ee454587839b496774f070ff80f575c12f94d5a50edb98aa4a2e1c" Sep 16 08:14:37.603029 ip-10-0-142-170 kubenswrapper[2552]: E0916 08:14:37.602995 2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fc8d88858ee454587839b496774f070ff80f575c12f94d5a50edb98aa4a2e1c\": container with ID starting with 6fc8d88858ee454587839b496774f070ff80f575c12f94d5a50edb98aa4a2e1c not found: ID does not exist" containerID="6fc8d88858ee454587839b496774f070ff80f575c12f94d5a50edb98aa4a2e1c" Sep 16 08:14:37.603497 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:37.603043 2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fc8d88858ee454587839b496774f070ff80f575c12f94d5a50edb98aa4a2e1c"} err="failed to get container status \"6fc8d88858ee454587839b496774f070ff80f575c12f94d5a50edb98aa4a2e1c\": rpc error: code = NotFound desc = could not find container \"6fc8d88858ee454587839b496774f070ff80f575c12f94d5a50edb98aa4a2e1c\": container with ID starting with 6fc8d88858ee454587839b496774f070ff80f575c12f94d5a50edb98aa4a2e1c not found: ID does not exist" Sep 16 08:14:37.610996 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:37.610959 2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66"] Sep 16 08:14:37.612524 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:37.612502 2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-c5501a5b-kserve-6886c9974c-rzc66"] Sep 16 08:14:38.936452 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:38.936419 2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b88ecd1-2ecc-430b-bae2-e73c800b41a2" path="/var/lib/kubelet/pods/0b88ecd1-2ecc-430b-bae2-e73c800b41a2/volumes" Sep 16 08:14:49.823873 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:49.823788 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-b85d5f948-wswds"] Sep 16 08:14:49.824290 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:49.824189 2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b88ecd1-2ecc-430b-bae2-e73c800b41a2" containerName="main" Sep 16 08:14:49.824290 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:49.824202 2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b88ecd1-2ecc-430b-bae2-e73c800b41a2" containerName="main" Sep 16 08:14:49.824290 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:49.824268 2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b88ecd1-2ecc-430b-bae2-e73c800b41a2" containerName="main" Sep 16 08:14:49.832146 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:49.831185 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-b85d5f948-wswds" Sep 16 08:14:49.834697 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:49.834671 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 16 08:14:49.834875 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:49.834689 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 16 08:14:49.835005 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:49.834689 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fknxb\"" Sep 16 08:14:49.835005 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:49.834703 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 16 08:14:49.835005 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:49.834712 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 16 08:14:49.838284 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:49.838262 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-b85d5f948-wswds"] Sep 16 08:14:49.847924 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:49.847889 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v24wq\" (UniqueName: \"kubernetes.io/projected/d28bba96-4bde-4714-8c92-61eef43e0db8-kube-api-access-v24wq\") pod \"maas-api-b85d5f948-wswds\" (UID: \"d28bba96-4bde-4714-8c92-61eef43e0db8\") " pod="odh-ai-gateway-infra/maas-api-b85d5f948-wswds" Sep 16 08:14:49.848215 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:49.848191 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d28bba96-4bde-4714-8c92-61eef43e0db8-openshift-service-ca\") pod \"maas-api-b85d5f948-wswds\" (UID: \"d28bba96-4bde-4714-8c92-61eef43e0db8\") " pod="odh-ai-gateway-infra/maas-api-b85d5f948-wswds" Sep 16 08:14:49.848403 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:49.848384 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d28bba96-4bde-4714-8c92-61eef43e0db8-metrics-certs\") pod \"maas-api-b85d5f948-wswds\" (UID: \"d28bba96-4bde-4714-8c92-61eef43e0db8\") " pod="odh-ai-gateway-infra/maas-api-b85d5f948-wswds" Sep 16 08:14:49.848580 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:49.848558 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d28bba96-4bde-4714-8c92-61eef43e0db8-maas-api-tls\") pod \"maas-api-b85d5f948-wswds\" (UID: \"d28bba96-4bde-4714-8c92-61eef43e0db8\") " pod="odh-ai-gateway-infra/maas-api-b85d5f948-wswds" Sep 16 08:14:49.949862 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:49.949820 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d28bba96-4bde-4714-8c92-61eef43e0db8-maas-api-tls\") pod \"maas-api-b85d5f948-wswds\" (UID: \"d28bba96-4bde-4714-8c92-61eef43e0db8\") " pod="odh-ai-gateway-infra/maas-api-b85d5f948-wswds" Sep 16 08:14:49.950093 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:49.949909 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v24wq\" (UniqueName: \"kubernetes.io/projected/d28bba96-4bde-4714-8c92-61eef43e0db8-kube-api-access-v24wq\") pod \"maas-api-b85d5f948-wswds\" (UID: \"d28bba96-4bde-4714-8c92-61eef43e0db8\") " pod="odh-ai-gateway-infra/maas-api-b85d5f948-wswds" Sep 16 08:14:49.950093 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:49.949936 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d28bba96-4bde-4714-8c92-61eef43e0db8-openshift-service-ca\") pod \"maas-api-b85d5f948-wswds\" (UID: \"d28bba96-4bde-4714-8c92-61eef43e0db8\") " pod="odh-ai-gateway-infra/maas-api-b85d5f948-wswds" Sep 16 08:14:49.950093 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:49.949997 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d28bba96-4bde-4714-8c92-61eef43e0db8-metrics-certs\") pod \"maas-api-b85d5f948-wswds\" (UID: \"d28bba96-4bde-4714-8c92-61eef43e0db8\") " pod="odh-ai-gateway-infra/maas-api-b85d5f948-wswds" Sep 16 08:14:49.950851 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:49.950819 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d28bba96-4bde-4714-8c92-61eef43e0db8-openshift-service-ca\") pod \"maas-api-b85d5f948-wswds\" (UID: \"d28bba96-4bde-4714-8c92-61eef43e0db8\") " pod="odh-ai-gateway-infra/maas-api-b85d5f948-wswds" Sep 16 08:14:49.952516 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:49.952484 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d28bba96-4bde-4714-8c92-61eef43e0db8-maas-api-tls\") pod \"maas-api-b85d5f948-wswds\" (UID: \"d28bba96-4bde-4714-8c92-61eef43e0db8\") " pod="odh-ai-gateway-infra/maas-api-b85d5f948-wswds" Sep 16 08:14:49.952624 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:49.952556 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d28bba96-4bde-4714-8c92-61eef43e0db8-metrics-certs\") pod \"maas-api-b85d5f948-wswds\" (UID: \"d28bba96-4bde-4714-8c92-61eef43e0db8\") " pod="odh-ai-gateway-infra/maas-api-b85d5f948-wswds" Sep 16 08:14:49.958710 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:49.958685 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v24wq\" (UniqueName: \"kubernetes.io/projected/d28bba96-4bde-4714-8c92-61eef43e0db8-kube-api-access-v24wq\") pod \"maas-api-b85d5f948-wswds\" (UID: \"d28bba96-4bde-4714-8c92-61eef43e0db8\") " pod="odh-ai-gateway-infra/maas-api-b85d5f948-wswds" Sep 16 08:14:50.144629 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:50.144522 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-b85d5f948-wswds" Sep 16 08:14:50.420774 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:50.420623 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-b85d5f948-wswds"] Sep 16 08:14:50.423643 ip-10-0-142-170 kubenswrapper[2552]: W0916 08:14:50.423605 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd28bba96_4bde_4714_8c92_61eef43e0db8.slice/crio-2f2a04ae9052529e1ae4659e2e7407949bf15ed05148557c46c4c0d8ae21f2e3 WatchSource:0}: Error finding container 2f2a04ae9052529e1ae4659e2e7407949bf15ed05148557c46c4c0d8ae21f2e3: Status 404 returned error can't find the container with id 2f2a04ae9052529e1ae4659e2e7407949bf15ed05148557c46c4c0d8ae21f2e3 Sep 16 08:14:50.425149 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:50.425128 2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 16 08:14:50.655517 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:50.655474 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-b85d5f948-wswds" event={"ID":"d28bba96-4bde-4714-8c92-61eef43e0db8","Type":"ContainerStarted","Data":"2f2a04ae9052529e1ae4659e2e7407949bf15ed05148557c46c4c0d8ae21f2e3"} Sep 16 08:14:53.673464 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:53.673429 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-b85d5f948-wswds" event={"ID":"d28bba96-4bde-4714-8c92-61eef43e0db8","Type":"ContainerStarted","Data":"87213bf015d21d2d9edfac31f87948b1b5a53af7726eb3a8f5642de4d4a510ef"} Sep 16 08:14:53.673947 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:53.673495 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-b85d5f948-wswds" Sep 16 08:14:53.695166 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:53.695112 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-b85d5f948-wswds" podStartSLOduration=2.426071265 podStartE2EDuration="4.695078836s" totalImagesPullingTime="2.269007571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 08:14:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 08:14:53.693828269 +0000 UTC m=+2465.277606606" watchObservedRunningTime="2026-09-16 08:14:53.695078836 +0000 UTC m=+2465.278857241" Sep 16 08:14:59.684546 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:14:59.684510 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-b85d5f948-wswds" Sep 16 08:15:02.809416 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:02.809380 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:15:20.980065 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:20.980018 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-6fw9g_962c76dc-bb7c-45bf-b4c1-fc2968fa707c/server/0.log" Sep 16 08:15:21.318365 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:21.318273 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dcb658945-hbqzx_c4734177-10ea-4111-bf3e-840fa34f0a6c/copy-manifests/0.log" Sep 16 08:15:21.324730 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:21.324698 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dcb658945-hbqzx_c4734177-10ea-4111-bf3e-840fa34f0a6c/manager/0.log" Sep 16 08:15:21.908612 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:21.908579 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-b85d5f948-wswds_d28bba96-4bde-4714-8c92-61eef43e0db8/maas-api/0.log" Sep 16 08:15:22.016333 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:22.016301 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29825745-88658_ebb41bac-18f5-4c9f-955c-7ab2b7b896eb/cleanup/0.log" Sep 16 08:15:23.417488 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:23.417462 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-8wgwz_2362570e-47ce-4c9c-bbac-382833e94ae0/kuadrant-console-plugin/0.log" Sep 16 08:15:23.535036 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:23.535003 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-vn6jp_f98af655-d206-479c-b49a-887123030af2/manager/0.log" Sep 16 08:15:24.329396 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:24.329367 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b6965477b-dxhrr_50f94726-d0ba-41b4-9288-cff938220e36/kube-auth-proxy/0.log" Sep 16 08:15:25.318249 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:25.318218 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-5bhzn_feaebf7f-ae1c-4614-b028-ae3f5b168ecb/main/0.log" Sep 16 08:15:25.678111 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:25.678076 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ftqvhw_23c672c9-120c-4de1-aac1-b99ec52f350f/main/0.log" Sep 16 08:15:25.797466 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:25.797434 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-5csgx_154478c5-73f4-4e25-bf30-b75b3d0fe576/main/0.log" Sep 16 08:15:25.922733 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:25.922699 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-5hs67_6216f4c2-b225-43d5-a1a9-3e42ed6d7809/main/0.log" Sep 16 08:15:30.988449 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:30.988406 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:15:31.051721 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:31.051685 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:15:31.530411 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:31.530376 2552 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" limit=1024 Sep 16 08:15:33.958549 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:33.958513 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wbslq_4152b2c2-a19c-44fa-96f5-dc3eb661aa9b/global-pull-secret-syncer/0.log" Sep 16 08:15:34.046867 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:34.046823 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-7zl67_ef5303b5-4dc6-4e4c-9716-bcf88a796cae/konnectivity-agent/0.log" Sep 16 08:15:34.169324 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:34.169290 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-170.ec2.internal_1fcac57a8698f66711e8e05f204f0824/haproxy/0.log" Sep 16 08:15:38.592354 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:38.592266 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-8wgwz_2362570e-47ce-4c9c-bbac-382833e94ae0/kuadrant-console-plugin/0.log" Sep 16 08:15:38.657765 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:38.657729 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-vn6jp_f98af655-d206-479c-b49a-887123030af2/manager/0.log" Sep 16 08:15:40.706931 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:40.706893 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b5g4j_c41268ea-998a-4cb5-8763-9780490c6432/node-exporter/0.log" Sep 16 08:15:40.725486 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:40.725451 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b5g4j_c41268ea-998a-4cb5-8763-9780490c6432/kube-rbac-proxy/0.log" Sep 16 08:15:40.744981 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:40.744953 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-b5g4j_c41268ea-998a-4cb5-8763-9780490c6432/init-textfile/0.log" Sep 16 08:15:40.837959 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:40.837903 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-pjv4h_998d1cf1-cf6f-405e-807d-eb18e33f2e54/kube-rbac-proxy-main/0.log" Sep 16 08:15:40.859588 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:40.859556 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-pjv4h_998d1cf1-cf6f-405e-807d-eb18e33f2e54/kube-rbac-proxy-self/0.log" Sep 16 08:15:40.879702 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:40.879671 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-67dcf8c979-pjv4h_998d1cf1-cf6f-405e-807d-eb18e33f2e54/openshift-state-metrics/0.log" Sep 16 08:15:40.915093 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:40.915058 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c8d25c0d-6cda-4dd3-997d-a9012823ce7a/prometheus/0.log" Sep 16 08:15:40.941807 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:40.941776 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c8d25c0d-6cda-4dd3-997d-a9012823ce7a/config-reloader/0.log" Sep 16 08:15:40.963691 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:40.963666 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c8d25c0d-6cda-4dd3-997d-a9012823ce7a/thanos-sidecar/0.log" Sep 16 08:15:40.988148 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:40.988059 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c8d25c0d-6cda-4dd3-997d-a9012823ce7a/kube-rbac-proxy-web/0.log" Sep 16 08:15:41.008713 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:41.008688 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c8d25c0d-6cda-4dd3-997d-a9012823ce7a/kube-rbac-proxy/0.log" Sep 16 08:15:41.028702 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:41.028680 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c8d25c0d-6cda-4dd3-997d-a9012823ce7a/kube-rbac-proxy-thanos/0.log" Sep 16 08:15:41.047998 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:41.047972 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c8d25c0d-6cda-4dd3-997d-a9012823ce7a/init-config-reloader/0.log" Sep 16 08:15:41.080605 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:41.080504 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-cnsvs_7dbc90e5-a266-40cd-b099-35ec42755ec4/prometheus-operator/0.log" Sep 16 08:15:41.097119 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:41.097075 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-cnsvs_7dbc90e5-a266-40cd-b099-35ec42755ec4/kube-rbac-proxy/0.log" Sep 16 08:15:41.136509 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:41.136482 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-rrcsn_2b09201b-f78c-4de1-91ff-b583cab8316e/prometheus-operator-admission-webhook/0.log" Sep 16 08:15:44.073754 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:44.073719 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-wvs7b_eccfa20f-4a48-4588-a7fa-a82fe43e1ccc/download-server/0.log" Sep 16 08:15:44.558505 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:44.558464 2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zxrzq/perf-node-gather-daemonset-d8fl4"] Sep 16 08:15:44.562078 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:44.562048 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-d8fl4" Sep 16 08:15:44.564384 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:44.564358 2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zxrzq\"/\"default-dockercfg-nr2xl\"" Sep 16 08:15:44.564531 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:44.564408 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zxrzq\"/\"openshift-service-ca.crt\"" Sep 16 08:15:44.565224 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:44.565208 2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zxrzq\"/\"kube-root-ca.crt\"" Sep 16 08:15:44.568588 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:44.568564 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zxrzq/perf-node-gather-daemonset-d8fl4"] Sep 16 08:15:44.639689 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:44.639651 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d5e12f21-22fb-45ba-8290-88d567e23d96-podres\") pod \"perf-node-gather-daemonset-d8fl4\" (UID: \"d5e12f21-22fb-45ba-8290-88d567e23d96\") " pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-d8fl4" Sep 16 08:15:44.639689 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:44.639701 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d5e12f21-22fb-45ba-8290-88d567e23d96-proc\") pod \"perf-node-gather-daemonset-d8fl4\" (UID: \"d5e12f21-22fb-45ba-8290-88d567e23d96\") " pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-d8fl4" Sep 16 08:15:44.639923 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:44.639727 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h8jr\" (UniqueName: \"kubernetes.io/projected/d5e12f21-22fb-45ba-8290-88d567e23d96-kube-api-access-4h8jr\") pod \"perf-node-gather-daemonset-d8fl4\" (UID: \"d5e12f21-22fb-45ba-8290-88d567e23d96\") " pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-d8fl4" Sep 16 08:15:44.639923 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:44.639847 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5e12f21-22fb-45ba-8290-88d567e23d96-sys\") pod \"perf-node-gather-daemonset-d8fl4\" (UID: \"d5e12f21-22fb-45ba-8290-88d567e23d96\") " pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-d8fl4" Sep 16 08:15:44.639923 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:44.639898 2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d5e12f21-22fb-45ba-8290-88d567e23d96-lib-modules\") pod \"perf-node-gather-daemonset-d8fl4\" (UID: \"d5e12f21-22fb-45ba-8290-88d567e23d96\") " pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-d8fl4" Sep 16 08:15:44.741014 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:44.740957 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5e12f21-22fb-45ba-8290-88d567e23d96-sys\") pod \"perf-node-gather-daemonset-d8fl4\" (UID: \"d5e12f21-22fb-45ba-8290-88d567e23d96\") " pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-d8fl4" Sep 16 08:15:44.741242 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:44.741119 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d5e12f21-22fb-45ba-8290-88d567e23d96-sys\") pod \"perf-node-gather-daemonset-d8fl4\" (UID: \"d5e12f21-22fb-45ba-8290-88d567e23d96\") " pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-d8fl4" Sep 16 08:15:44.741242 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:44.741133 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d5e12f21-22fb-45ba-8290-88d567e23d96-lib-modules\") pod \"perf-node-gather-daemonset-d8fl4\" (UID: \"d5e12f21-22fb-45ba-8290-88d567e23d96\") " pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-d8fl4" Sep 16 08:15:44.741242 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:44.741194 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d5e12f21-22fb-45ba-8290-88d567e23d96-podres\") pod \"perf-node-gather-daemonset-d8fl4\" (UID: \"d5e12f21-22fb-45ba-8290-88d567e23d96\") " pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-d8fl4" Sep 16 08:15:44.741242 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:44.741230 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d5e12f21-22fb-45ba-8290-88d567e23d96-proc\") pod \"perf-node-gather-daemonset-d8fl4\" (UID: \"d5e12f21-22fb-45ba-8290-88d567e23d96\") " pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-d8fl4" Sep 16 08:15:44.741435 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:44.741252 2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h8jr\" (UniqueName: \"kubernetes.io/projected/d5e12f21-22fb-45ba-8290-88d567e23d96-kube-api-access-4h8jr\") pod \"perf-node-gather-daemonset-d8fl4\" (UID: \"d5e12f21-22fb-45ba-8290-88d567e23d96\") " pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-d8fl4" Sep 16 08:15:44.741435 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:44.741319 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d5e12f21-22fb-45ba-8290-88d567e23d96-lib-modules\") pod \"perf-node-gather-daemonset-d8fl4\" (UID: \"d5e12f21-22fb-45ba-8290-88d567e23d96\") " pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-d8fl4" Sep 16 08:15:44.741435 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:44.741349 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d5e12f21-22fb-45ba-8290-88d567e23d96-proc\") pod \"perf-node-gather-daemonset-d8fl4\" (UID: \"d5e12f21-22fb-45ba-8290-88d567e23d96\") " pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-d8fl4" Sep 16 08:15:44.741435 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:44.741364 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d5e12f21-22fb-45ba-8290-88d567e23d96-podres\") pod \"perf-node-gather-daemonset-d8fl4\" (UID: \"d5e12f21-22fb-45ba-8290-88d567e23d96\") " pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-d8fl4" Sep 16 08:15:44.749486 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:44.749461 2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h8jr\" (UniqueName: \"kubernetes.io/projected/d5e12f21-22fb-45ba-8290-88d567e23d96-kube-api-access-4h8jr\") pod \"perf-node-gather-daemonset-d8fl4\" (UID: \"d5e12f21-22fb-45ba-8290-88d567e23d96\") " pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-d8fl4" Sep 16 08:15:44.873602 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:44.873502 2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-d8fl4" Sep 16 08:15:45.144962 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:45.144899 2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zxrzq/perf-node-gather-daemonset-d8fl4"] Sep 16 08:15:45.147415 ip-10-0-142-170 kubenswrapper[2552]: W0916 08:15:45.147389 2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd5e12f21_22fb_45ba_8290_88d567e23d96.slice/crio-bdfbe372ee818be30d78444d45b185567414527f6476e5773003e8a056baa12c WatchSource:0}: Error finding container bdfbe372ee818be30d78444d45b185567414527f6476e5773003e8a056baa12c: Status 404 returned error can't find the container with id bdfbe372ee818be30d78444d45b185567414527f6476e5773003e8a056baa12c Sep 16 08:15:45.945458 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:45.945424 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-d8fl4" event={"ID":"d5e12f21-22fb-45ba-8290-88d567e23d96","Type":"ContainerStarted","Data":"4a9a16dc15cf6d86744da77dc782266cdcca7eef8ae025d70036f3a6e8d36889"} Sep 16 08:15:45.945643 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:45.945475 2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-d8fl4" event={"ID":"d5e12f21-22fb-45ba-8290-88d567e23d96","Type":"ContainerStarted","Data":"bdfbe372ee818be30d78444d45b185567414527f6476e5773003e8a056baa12c"} Sep 16 08:15:45.945643 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:45.945526 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-d8fl4" Sep 16 08:15:45.961508 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:45.961450 2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-d8fl4" podStartSLOduration=1.961433269 podStartE2EDuration="1.961433269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 08:15:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 08:15:45.95915605 +0000 UTC m=+2517.542934385" watchObservedRunningTime="2026-09-16 08:15:45.961433269 +0000 UTC m=+2517.545211606" Sep 16 08:15:46.036438 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:46.036404 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qks9b_9852905b-319e-4378-8c40-266974260880/dns/0.log" Sep 16 08:15:46.057122 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:46.057066 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qks9b_9852905b-319e-4378-8c40-266974260880/kube-rbac-proxy/0.log" Sep 16 08:15:46.122712 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:46.122684 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rxbfc_024d7ae0-3a5a-4d5e-bedb-592058d44673/dns-node-resolver/0.log" Sep 16 08:15:46.614316 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:46.614279 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-56676c689d-5tjq7_3c72acfc-adc3-4592-b81e-e841ea01b360/registry/0.log" Sep 16 08:15:46.658767 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:46.658737 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9xkmz_29d2a484-46de-40a9-be4a-b9c6b266fbae/node-ca/0.log" Sep 16 08:15:47.980866 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:47.980836 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b6965477b-dxhrr_50f94726-d0ba-41b4-9288-cff938220e36/kube-auth-proxy/0.log" Sep 16 08:15:49.067110 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:49.067066 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2gfrg_4aad7734-8782-4fac-b802-89803b94da62/serve-healthcheck-canary/0.log" Sep 16 08:15:49.883676 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:49.883647 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rw4xp_3cc58375-e99c-4fd2-8d0c-1a652869ac2d/kube-rbac-proxy/0.log" Sep 16 08:15:49.907939 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:49.907914 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rw4xp_3cc58375-e99c-4fd2-8d0c-1a652869ac2d/exporter/0.log" Sep 16 08:15:49.947522 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:49.947485 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rw4xp_3cc58375-e99c-4fd2-8d0c-1a652869ac2d/extractor/0.log" Sep 16 08:15:51.960785 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:51.960755 2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zxrzq/perf-node-gather-daemonset-d8fl4" Sep 16 08:15:52.406289 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:52.406201 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-6fw9g_962c76dc-bb7c-45bf-b4c1-fc2968fa707c/server/0.log" Sep 16 08:15:52.608230 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:52.608198 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dcb658945-hbqzx_c4734177-10ea-4111-bf3e-840fa34f0a6c/manager/0.log" Sep 16 08:15:52.627461 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:52.627437 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dcb658945-hbqzx_c4734177-10ea-4111-bf3e-840fa34f0a6c/copy-manifests/0.log" Sep 16 08:15:54.088791 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:54.088767 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6f78fc697-hxrsc_5602b217-00e1-4d62-b1e1-982a88eb3f5a/manager/0.log" Sep 16 08:15:55.218654 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:55.218617 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b6965477b-dxhrr_50f94726-d0ba-41b4-9288-cff938220e36/kube-auth-proxy/0.log" Sep 16 08:15:57.275480 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:57.275450 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7b6965477b-dxhrr_50f94726-d0ba-41b4-9288-cff938220e36/kube-auth-proxy/0.log" Sep 16 08:15:59.488357 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:59.488327 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-7r4rg_fd239b36-8f44-43b4-b6d5-cd450071846a/migrator/0.log" Sep 16 08:15:59.507021 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:15:59.506989 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-7r4rg_fd239b36-8f44-43b4-b6d5-cd450071846a/graceful-termination/0.log" Sep 16 08:16:00.896279 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:16:00.896249 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mdzsg_2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7/kube-multus-additional-cni-plugins/0.log" Sep 16 08:16:00.915754 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:16:00.915722 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mdzsg_2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7/egress-router-binary-copy/0.log" Sep 16 08:16:00.933963 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:16:00.933934 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mdzsg_2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7/cni-plugins/0.log" Sep 16 08:16:00.952876 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:16:00.952841 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mdzsg_2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7/bond-cni-plugin/0.log" Sep 16 08:16:00.971543 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:16:00.971514 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mdzsg_2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7/routeoverride-cni/0.log" Sep 16 08:16:00.991113 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:16:00.991072 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mdzsg_2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7/whereabouts-cni-bincopy/0.log" Sep 16 08:16:01.011376 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:16:01.011351 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mdzsg_2ac1499c-a3b2-4e0c-8b6d-533b05d55ce7/whereabouts-cni/0.log" Sep 16 08:16:01.230138 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:16:01.230086 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-tf242_8cdfedd6-88d2-459f-bf39-504bc65e2fed/kube-multus/0.log" Sep 16 08:16:01.328513 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:16:01.328483 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-94n4h_aca15303-6546-4428-bb32-539bb25d2486/network-metrics-daemon/0.log" Sep 16 08:16:01.343634 ip-10-0-142-170 kubenswrapper[2552]: I0916 08:16:01.343606 2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-94n4h_aca15303-6546-4428-bb32-539bb25d2486/kube-rbac-proxy/0.log"